Good afternoon. Today, I will present our paper “Effects of argumentative explanation
types -on the perception of review-based recommendations”. My name is Diana Hernandez.
The focus of my ongoing research is on how to provide better explanations in recommender
systems, in particular argumentative explanations.
The motivation for that is that,
Despite the maturity already achieved by RS, they’re still perceived as black boxes
In the other hand, the perception of transparency and effectiveness by users can be increased
when explanations are provided.
In particular, we aim to go beyond this famous example of amazon’s textual explanations,
Thus, we work on an approach that seeks to provide- more robust statements on diverse
aspects- in order to support the claim that an article can be recommended to a user.
… and we plan to do this by exploiting the rich content that can be found in user reviews.
In this regard, the interest in exploiting online reviews has increased lately, not only
to improve the accuracy of the predictions, but also to provide better explanations.
For example, Zhang and colleagues proposed a matrix factorization model to generate explanations,
based on the alignment of explicit features and the latent representations of items and
user preferences obtained from reviews.
We also have the work of Zheng and colleagues, which is based on a deep learning approach,
to learn simultaneously the item features and users preferences from reviews as well,
in order to predict ratings.This approach was extended by Chen colleagues, who stressed
that not all reviews should contribute the same, but only helpful reviews should be used
in order to improve predictions. Here, according to this approach, helpful reviews could also
be provided as explanations.
Furthermore, we have approaches that are based on the generation of natural language sentences
- that summarize the content found in reviews, as proposed by Costa and colleagues.
Regarding textual explanations, we can see here some examples of - different approaches
proposed so far.
Previous work is highly based on the brief assessment of relevant features, and the use
of templates similar to those we can see here, or abstractive summarizations as the example
of the above-mentioned work by Costa.
But would users actually prefer such concise explanations to those that can provide a higher
level of detail? And if the latter is the case, how can we
use the information found in the reviews to generate better explanations?
In order to answer those questions, we have designed and tested a series of explanation
templates - for the hotels domain, that reflect an argumentative structure and include some
of the elements that are part of the Toulmin model, one of the most widely used conceptual
models of argumentation.
However, a number of studies have criticized the applicability of this model - for argumentation
in real life. In this respect, Habernal and colleagues proposed a variation of this model
- that could represent better the kind of arguments that are - usually provided in web
discourse generated by users. Although this adaptation was mainly designed to support
argument mining tasks, we believe that, by leveraging this scheme, we can generate explanations
by means of structured arguments - that could still be easy to follow by users.
So, this is how we utilized the scheme proposed by Habernal to generate our templates:
Here, a claim or the conclusion that the hotel is a good option is supported by a report
of positive opinions (the backing component). We also provide a report of negative opinions,
as rebuttal, but offering also a refutation, in order to provide possible reasons for such
negative statements.
We aimed to compare the effect of 3 different types of explanations, that reflect such an
argumentative structure, so we have the aggregation condition, where we provided
statistical information, particularly percentages of positive and negative opinions- about certain
features or aspects. then we had the summary condition, that is,
a summary of opinions without showing percentages And the review condition, here we provide
a useful review as example of why the hotel is a good option.
Additionally, we also wanted to test the effect that different levels of justification might
have on users’ perception. Here, in the condition high level of justification,
we have more sentences addressing both general and specific aspects, the latter being fine-grained
aspects, for example “cleanliness of the bathroom” or the “cleanliness of the sheets”,
whereas the low level condition only provides information about main aspects, like the room
or facilities in general, using much less sentences.
We aimed to test the effect of these templates on users’ perception, regarding explanation
quality, transparency, and other explanatory aims.
So, more formally, our research questions were:
first: does the type of explanation influence user’s perception?
second: does the level of justification influence this perception?
and third, we wanted to analyse whether differences in user characteristics - could also contribute
to differences in perception.
So, in particular, we consider the style in which users make decisions, which could be
either rational or intuitive. And we also analysed to what extent the ability
of users to listening to others- and taking into account their opinions, could influence
our variables of interest. This ability is actually referred as social awareness
To test the above, we recruited a total of 152 AMT workers for our user study.
Participants were randomly assigned to one out of six conditions, which were the combination
of the variables type of explanation and level of justification.
Participants were asked to respond a questionnaire on their opinion about the quality of the
explanations, as well as their perception about the system prototype, in terms of transparency,
effectiveness, efficiency and trust A user characteristics questionnaire was also
used to measure decision making style and social awareness.
Participants were presented with a prototype , that provided a list of hotels corresponding
to a hypothetical hotel search. They were instructed to click on the “View
Details” button, so they could read the explanations for each hotel
This screenshot depicts the condition “aggregation” - “high” level of justification.
Here are the results: First, we have that the type of explanation
does influence significantly the perception of explanation quality.
In particular, explanations that provide percentages of positive and negative opinions are perceived
significantly better than summaries without percentages.
Second, we found that the type of explanation influences significantly the perception of
transparency, in particular we found that the aggregation explanations conveyed more
transparency than providing just a helpful review.
Contrary to our expectations, we found no main effects of the level of justification.
We believe that one reason could be that - we only provide a static view of what the system
infers are the most important aspects for users.
However, users might be interested to dig into details beyond their usual preferences.
For example, even if ‘cleanliness’ is not necessarily the most important aspect
to a user, she/he might be interested to read specifics on why a hotel has some negative
reports in this regard.
Now, regarding user characteristics, we found no main effects of decision making styles.
However, we found that social awareness influences both transparency and trust. That is, users
with a higher disposition to listen to others, tend to perceive the system as more transparent
and trustworthy,- when the proposed explanations are provided, independent of their type or
justification level.
We also observed a tendency to prefer explanations with opinion percentages for users with higher
social awareness scores, compared to explanations that provide no percentages. That is, the
more they listen to others and take into account their opinions, the better they perceived
the aggregated view that provides percentages of positive and negative aspects.
Finally, it is important to acknowledge some of the limitations of our study, which we
are currently working on, for extended work in this matter.
First, we have used a prototype, were users actual preferences could not be requested
or detected. Although we tried to mitigate this matter by presenting participants with
a cover story, in which we told the users to pretend that their most important aspects
were the ones we indicated, the use of a real system, may have an impact on the perception
of the explanations.
Second, the Use of AMT platform, where choices are hard to motivate.
Regarding our approach, as pointed out by some of the participants, explanations based
on templates may be perceived as too repetitive, while implementations based on natural language
generation may be perceived as more flexible.
As conclusion, our results suggest that the type of explanation influence the perception
of review-based recommender systems. Particularly, an aggregated view of positive and negative
opinions using percentages seems to be a meaningful way to present explanations. Our results also
indicate that social awareness seems to play a meaningful role in the overall perception
of both explanations and the overall RS.
Thank you very much