This presentation aims to subjectively evaluate our proposed feature-based CF style explanation
that exploits multiple data sources compared to conventional CF and CB style explanations,
that mostly rely on single source of data.
So let’s get started.
There are many existing approaches to generate explainable recommendations.
For example in case of CF approach which only requires availability of explicit or implicit
feedback from users and try to predict the ratings for the items to recommend to the
user.
Many attempts have been made to explain these predictions to users but is still a challenging
task
Common examples of CF style explanations are the Amazon “ Customers who bought this also
bought…”, explanations describing why an item has been recommended in a rather simple
manner –independent of the complexity of the underlying model.
On the other hand, CB rely on, for example, predefined metadata or tags generated by other
But this approach requires availability of content data which is not always directly
users or extracted from unstructured texts to provide explanations.
available.
systems.
Both CF and CB style explanations can be considered effective in most standard recommendation
Examples of such content-based explanations can be seen in Tagsplanations and News Dude
scenarios.
But in reality, recommendation process is much more complex especially in high risk
Digital Cameras.
domains , e.g.
In such domains, the user’s decision becomes more difficult, where deciding to buy camera
is much complex than choosing a movie to watch.
So in such cases, being able to fall back on multiple data sources could generate more
transparent and argumentative explanations that could help users in their decision.
Hybrid approaches exist that have already combined CF and CB data.
But these approaches mostly focus on improving the system accuracy rather than generating
Even in cases where the explanations are provided, they do not explain how the recommended items
explainable recommendations.
are relevant to the current user in terms of user’s preferences for item-features.
In this line of work, we in the previous work proposed an approach that combines CB with
CF in a hybrid manner but with a purpose of providing explainable recommendations in terms
of exploiting content-features.
So how our approach works.
In the first step, the user-item-ratings matrix and the item-features matrix are exploited
in an entropy-based feature-weighting technique , that compute feature-weightings for each
user.
This creates a feature-based profiles of all available users in our dataset.
This process is done offline.
In the cold-start, we elicits the active user’s preferences w.r.t to features of cameras by
explicitly asking the user to rate number of features.
In the next step, the active user’s as well as other users’ feature-based profiles are
exploited in a Gower’s similarity approach to compute users who are similar to the active
user in terms of shared feature-based preferences.
The system then generates item and feature recommendations as well as their corresponding
explanations based on these similar users’ preferences.
The idea of our approach is to explain the recommender’s output via item features.
Yet, an in-depth empirical evaluation of our proposed approach is missing from user perspective,
which we aim to address in this work.
We then implemented a prototype system on top of this approach, which provides different
levels of explanations which I will briefly explain here.
As the recommendations are generated from similar users’ feature-based profiles, therefore,
in figure (A) we first explained how these users are similar to the current user in terms
Then in figure (B), we explained each recommended item based on similar users’ and the active
of their shared feature preferences .
user’s feature preferences and how this item matched those preferences.
Our approach not only recommended items but also features of the items where each recommended
feature is further explained in terms of how similar users’ have selected this feature
So in this work, to subjectively evaluate our approach as compared to established baselines
along with a specific feature-value.
i.e., item-based CF and CB, we formulated following hypotheses.
Our feature-based CF improves subjective assessment and user experience as compared to:
recommendations and explanations compared to conventional item-based CF
recommendations and explanations compared to conventional CB.
To test our hypotheses, we conducted two user studies.
The first user study was conducted in a within-subject design to test our hypothesis 1, where each
participant was presented with both Feature-based CF and Item-based CF system.
And the second user study was conducted in a between subject design to test our hypothesis
2, where each participant was presented with either Feature-based CF or Content-based system.
In both user studies, we first explicitly elicited users’ preferences in terms of
either items in cased of CF, or in terms of features in case of our feature-based CF and
CB approach.
Additionally, in case of our approach, they were also required to rate the selected features.
Then the system generates corresponding recommendations and its explanations.
Participants were required to explore the system and its functionalities, then select
a camera that matches their preferences and then subjectively evaluate the system by answering
series of questions.
The questionnaire items were taken from existing recommender evaluation frameworks.
Additionally, we used self-created items for decision support inspired from the existing
SUS and UEQ questionnaires were used for overall user experience.
work.
In the first user study, 20 university students participated.
In this study, we compared our feature-based CF approach with item-based CF . And all participants
experienced both systems in a counter-balance manner.
In the item-based CF system as can be seen in the figure, the explanations were provided
in relation to the user’s previously rated items.
To test the first part of our first hypothesis, that is the “Feature-based CF improves subjective
assessment of recommendations and explanations compared to conventional item-based CF”.We
conducted a one-way repeated measures MANOVA and found statistically significant differences
between two systems for aggregated dependent variables.
Univariate test results further revealed that our approach significantly performed better
in terms of Perceived Recommendation Quality, Explanation Quality, Transparency, Satisfaction,
and Decision Support.
Yet, results overall indicate that we can generally accept H1.
Then we tested second part of our hypothesis 1, that is the” Feature-based CF improves
user experience compared to conventional item-based CF.
But found no significant differences in terms of interface adequacy and SUS, however, our
feature-based CF showed slightly better results.
In general we can not accept this hypothesis
However, in our qualitative feedback, majority of participants preferred our system.
The second user study, was conducted to test our second hypothesis where we compared our
We recruited 100 amazon mechanical Turkers and used the data of 73 after excluding outliers.
feature-based CF approach with content-based approach but in a between subject design.
In CB system as you can see in the figure, the recommendations and explanations were
generated by simply matching the user’s selected features with the item-features.
To test first part of our second hypothesis that is the “Feature-based CF improves subjective
assessment of recommendations and explanations compared to conventional CB”.We again conducted
a one-way repeated measures MANOVA and found statistically significant differences between
two systems for aggregated dependent variables.
However, the differences were only significant in terms of two factors where the CB performed
better than our feature-based CF in terms of decision support and overall satisfaction.
Yet, results overall indicate that we can not accept this hypothesis
Then we tested second part of our hypothesis 2, that is the” Feature-based CF improves
user experience compared to conventional CB.
But found no significant differences in terms of interface adequacy and SUS, and also in
terms of UEQ
Overall, this hypothesis can also not be accepted
So to summarize our results, our approach performed significantly better than the item-based
CF, however in second user study, the CB performed significantly better.
But in case of both user studies there were no significant differences between systems
in terms of user experience.
The reason that why CB condition was positively perceived by participants might be explained
under the assumption, that participants were still provided with everything needed in terms
of content-features that might be sufficient for them to make their decision.
The insignificant differences in user experience can further be explained in terms of the qualitative
feedback that we collected.
When we asked the participants about what they like and dislike about our feature-based
CF system, even though many liked the amount of information, explanation as well as the
ability to rate features.
But for many of them the information was quite a lot and bit overwhelming . For some it was
also confusing to figure out how to use the system and they also suggested to make the
interface simplified and easier.
And this might explain why our approach did not perform significantly better in terms
of user experience as compared to item-based CF and CB approach
The gain through the integration of additional content-features with CF approach becomes
more prominent when compared with item-based CF that does not take features into account.
And less prominent when compared with content-based approach that already exploits content-data.
The insignificant differences in user experience might be explained in terms of complex interface
design as indicated by participants in their qualitative feedback .
On the other hand, this might also indicate that when our system compared with either
item-based CF or CB approach, participants perceived the systems of equal quality, which
Our future work would focus on further investigating these factors.
still draws a promising picture for our approach .
Thank you for watching this presentation