- [Woman] Jacob Solomon, from the University of Michigan,
he's also going to be talking about customization
in terms of recommender systems, thank you.
- Okay, so I'm going to try and kind of,
go through this pretty fast, because,
I'm from the eastern time zone and it's
really far past my lunch time, so.
(laughing)
And besides that, I have pretty much,
one very simple point to make for this talk.
But before I kind of get to that,
I want to set a few basic concepts.
Concepts that I imagine that many people
who had come to CHI are fairly familiar with,
but it's good to sort of, put them out there.
Recommender systems are systems that
kind of, fundamentally try to solve a tailoring problem.
And this means that if there's some items out
in the world, so for example,
movies that people might want to watch,
or music to listen to,
or maybe certain types of health insurance to buy.
And there's different types of users.
And different users have different preferences
for different types of items.
The goal of a recommender system is to use some
type of algorithm to match, match items to users.
To find the items that match users' preferences
and then communicate that to them effectively.
And there's a lot of different technical
and sociotechnical approaches to doing this.
Recently a lot of the HCI work on recommender systems
has looked at this idea of controllability,
or customizability.
Finding ways to give users a fine,
or a very sort of fine grain control over how
recommender systems, how tailoring algorithms,
and recommender algorithms work.
So some examples might be,
users might be able to give specified
weightings for different attributes.
So if I was a user looking for a movie,
and I might say, I want action movies starring Matt Damon,
but I care more about getting an action movie,
than getting a Matt Damon movie.
So I want the algorithm to put more emphasis on that.
So that might be one sort of approach for controllability
or customizability.
But some other work has looked at different
ways of like filtering the social network,
that recommendations come from.
Or providing sort of iterative critiquing of
recommendations.
And most of the work that's looked at these different
design approaches for customization,
are pretty consistent in finding that users like this
that giving control and giving customization
let's users improve the system and improve
the recommendations.
And for certain improve the overall user experience.
So the design heuristic that I want to argue for,
is that in a recommender system that is highly
customizable, the algorithm, or the underlying logic,
or system that produces recommendations needs
to be highly flexible,
such that for any particular recommendation,
there are lot's of, there are multiple different ways,
that the system can be configured,
that ultimately produce the same recommendation.
So let me show what I mean a little bit.
Imagine I have a recommender system that's
recommending colored shapes,
and if I configure it one way,
it will give me a blue triangle,
I change the settings and it would give me
a red square or change it again and I get
a green circle.
What I want to argue for is simply that it's
important, and a good design,
to make systems that have many different ways that
that can be configured that ultimately go a single
type of recommendation.
And the reason I make this argument is based on
some work that I've done in customizable
recommender systems.
My kind of broader research gen has more to do with
the decision making that users make after seeing
recommendations.
This is kind of something that I just kind of
found along the way.
But what I found is that there's an important
distinction between the heterogeneity that there
are among users in terms of their preferences.
So people are different,
people want different types of items, that's,
we know that,
that's why we have recommender systems in the first place.
But they're also different from each other
in terms of how they will navigate the system,
and how they will try and configure and control
the system to get what they want.
And what I found is that those,
those are sort of two different dimensions
in which users are different from each other.
And they don't always line up very well.
So what I have, I have a number of user experiments,
that I've done,
these are experiments that I've done on Mechanical Turk,
with some sort of toy systems that I,
that I've developed to kind of study decision making,
and with recommender systems.
One of the key parts of these experiments is that,
I don't have users come and make,
and use recommender systems for themselves,
I assign them their preference,
I tell them what kind of recommendations
they're trying to get from the system.
And that way,
the sample of users to participate in these studies,
all want the same thing.
So these are all users trying to get the exact
same type of recommendation from a system.
So I have this kind of, homogeneous user set.
And what I find is, in spite of that,
all these users kind of wanting to get the same
type of recommendations, in terms of like,
the same specific items,
that users will go about configuring it in
a lot of different ways.
So two systems that I'll kind of talk about
really briefly.
A system that makes recommendations for travel
destinations.
And another one for making recommendations about
physical exercise,
what kind of activity would be a good exercise
for a given situation.
In the study with the travel recommender,
I gave users a persona,
a very detailed explanation of a person
they were supposed to act like they were a travel agent
for a person, to find him a good, to plan a good vacation,
and they were given a lot of details about this
persona's preferences.
And they could configure the system,
there are a number of different attributes
that these, they can move sliders to kind of fine tune,
how the algorithm would work to produce the recommendations.
And, but they were all, all these users again were,
were working for the same persona.
So they really were trying to get the same,
essentially, recommendation.
But out of 375 people who did this,
I did a cluster analysis,
and I kind of looked at,
kind of tried to cluster different combinations
of where people put the sliders,
and the options that they chose.
And it was pretty easy to make a lot of clusters.
Basically, so this chart shows how much,
within cluster variance,
I guess to do these each time you add a new cluster,
and you can kind of see it doesn't flatten out
until about 30.
Meaning you can keep kind of dividing the groups up
even more because there's more variance to
to be reduced.
And 30 is a lot, given that again,
these sort of in theory,
were supposed to all be the same user.
Right,
they were all supposed to be looking for the same thing.
This is just kind of some distributions of where
left or right people moved the sliders.
And some other analysis.
There was also a large number of different combinations,
for different, for the different sliders.
So for the exercise study,
did this a little bit differently,
where I gave them much more explicit preferences,
they were kind of incentivized to find exercises
that had specific features,
like they were told, if you learn a,
if you find exercises that are classified as high cardio,
or can be done individual, or without a partner,
then you'll get more points,
and they were kind of incentivized to find these
specific kind of exercises.
And again, I cluster,
and I should say in this study,
each, I had about a hundred users,
and each user did this 10 different times,
with sort of different profiles.
And I looked at, for each of these different profiles,
how users configured the system.
And this is an example of, kind of what I found.
This is a sort of prototypical example of
you can look and see, there's sort of one large cluster,
of these users all configured the system
approximately the same way.
And then there's you know, two, two or three,
depending on the specific profile,
there's usually two or three additional sort of
clusters.
And in all these clusters, you know,
the large cluster never accounted for more than
about 40 or 50% of, of all the users.
Which means even sort of the most popular way,
of configuring a system,
is only, isn't even gonna get half of the users,
who all want the same thing.
Right.
And so that's just to kind of,
well, actually, I'm just gonna go through that.
And that's essentially the basis for,
for why I'm arguing for this heuristic.
That users, even when they want the same thing,
they're gonna try to go about it in different ways.
And that two users who maybe configure a
recommender system the same way,
it doesn't actually mean they want,
that they are looking for the same item.
So there's a danger in,
in looking at a large set of users who have
behaved the same way,
when using kind of an intelligent system,
like a recommender system,
and saying these users are trying to do the same thing.
And conversely, users who behave differently,
may actually have the same goal.
And that's why I think it's important to look
for sort of ways to make these systems flexible,
so that you can find lots of different ways to,
so users who sort of bring their own unique
mental models, unique experiences with these systems,
can kind of navigate and find their own way to,
to tell the recommender system.
And I'll just kind of finish by acknowledging
some colleagues who've provided a lot of feedback on this,
and I'll stop there.
(applause)
- [Woman] Thank you.
I know we probably should break for lunch,
but I think it'd be good to have a couple of
questions.
So we have some time, always back on time,
that's good.
Thank you, just say who you are and what your question is.
- [Man] Hi, Alvin from Ericsson Research.
Just a question about the research in general,
you call this a recommender system,
but wouldn't it be more accurate to call it
a faceted search system?
- No, so that is a good question,
so well, the travel agent one,
had kind of accommodation,
so a faceted search would be something that,
yeah, I mean, there's sort of a know attribute
of an item and you can basically, sort of,
you know, bullion logic and filter it down.
And these systems, so some of those attributes
were a little bit like that.
But for example, they were,
there were some that were like,
weight the importance of this.
Weight the importance of the cost of the trip.
Do you care?
So they would say how much they wanted it to cost,
but how important is it that the recommendation
matches that preference?
- [Woman] Okay, on that note,
then I'd like to thank Jacob and the rest of the speakers,
thank you very much for the interesting lectures
and interesting talks.
(applause)