- Hello, my name is Lukas Eberhard
from Graz University of Technology.
Welcome to my today's talk about
narrative-driven movie recommendations on Reddit.
Before I'm going to start,
I want you all to imagine that you are currently
in the mood for watching a movie.
You do not exactly know which movie you wanna watch
but you have some kind of an idea,
which characteristics the movie should have.
Let's assume that you are in the mood
for watching a thriller
that is similar to the movie The Dark Knight Rises,
and it should have some kind of mindblending storyline.
The first thing you're probably gonna do is
to check out one of your favorite streaming providers
and see which recommendations you get there.
But they are just based on the specific movie,
The Dark Knight Rises,
or on your user profile,
so they do not need to be thriller and
they also do not need to have a mindblending storyline.
So what else could you do?
You could go to the Internet Movie Database, IMDb,
which is a large online platform
with a huge amount of information about movies
and also user interactions like ratings and user reviews.
If you go to the page of The Dark Knight Rises,
you will find 12 movie recommendations,
but here again they do not need to be thrillers
and they do not necessarily have mindblending storylines.
So what else could you do?
You could go to Reddit, which is an online platform
with lots of online communities,
and one of those communities is
the subreddit called MovieSuggestions,
where users can request for movie recommendations
in a very specific form where they have free text requests
and rated community answers with movie suggestions.
In our case, it can look this.
We go to reddit.com into the subreddit MovieSuggestions,
submit a new post that says it's a request,
we want thrillers similar to the Dark Knight Rises,
but with a mindfucking plot.
So we submit this post, then it's there,
and the next thing we're gonna do is wait.
We have to wait for the community answers
which can take minutes or even hours,
except we are in the mood for watching a movie right now.
So it's time dependent depending on which time of day it is.
Also we can't be sure about
the quantity of suggestions we get there.
It can be the case that we just get
two movie recommendations from the community
that we have both already seen.
Also we can't be sure about
the quality of the recommendations
since they can just be a very personal flavor.
So those four points, those issues,
were the main idea and the main motivation for us
to build a narrative-driven reccomender system
that acts as a service for recommending movies
on all platforms like Reddit
by computing recommendations based on the user requests.
To evaluate our recommender systems in the end
we use existing Reddit submissions as the input,
compute our recommendations and in the end
we compare our recommendations
with the one from the Reddit community
and check the overlap.
So the target is to kind of imitate the community taste
of the Reddit community
and with this evaluation we can see how far
we can imitate this community taste.
So the first part of our recommender system
is to have a structured input.
But as you have seen before,
the Reddit submissions are a free-form narrative
as our example which is this short text here.
So we needed to bring this into a structured form.
We did this by performing a crowdsourcing experiment
where we took existing Reddit submissions
and asked crowd workers to tag those submissions,
to tag movie titles into submissions,
to tag genres or key words,
to bring them into a structured form
and for our example this would look like this.
We would have a genre: thriller.
We would have a movie: The Dark Knight Rises.
And we would have the key words: mindfucking plot.
At this point I also want to tell you that
we have published this dataset to enable researchers
to conduct independent analysis in the context
of narrative-driven movie recommendations.
You can find this dataset on www.rbz.io/datasets.
So now this structured input is the first part
of our recommender system and this is the input.
And the second part are then the recommender algorithms.
We implemented five state of the art recommender algorithms
which are an item-based collaborative filtering
based on user ratings from IMDb,
a Matrix factorization approach
also based on user ratings from IMDb,
a content-based filtering approach with TF-IDF
where we used movie descriptions
and user reviews from IMDb,
an embeddings approach with doc2vec,
also using movie descriptions and reviews from IMDb,
and a network-based approach where we created
the bipartite graph connecting actors with movies
and computed recommendations based on this network.
So with those five algorithms we computed
so-called candidate recommendations
since they are not final,
and in the third part of our recommender system
we are going to refine and re-rank
those candidate recommendations.
So the second part are the recommender algorithms,
and then the third part are
the post-filtering and re-ranking.
This part we have split up into two different experiments,
the first one is we just use data from IMDb
from the post-filtering and re-ranking.
Especially we use ratings, genres, the year,
plot keywords, predecessors, and successors
for the re-ranking which means
the more genres, for example,
a movie in this candidate recommendation list
has in common with the input movies,
in our case The Dark Knight Rises,
the higher it gets ranked.
For the second experiment we,
additionally to the IMDb data,
used information from the target platform,
in our case Reddit, in form of the narratives.
We use genres, years, keywords and actors.
From our example, this would mean that
we use the genre thriller,
and the keywords mindfucking plot.
Meaning that movies in the candidate list
that have the keywords mindfucking plot
in the movie description or the user reviews
get ranked higher in this candidate recommendation list.
So for all those characteristics we compute scores
for each movie in the candidate recommendation list
and we also have different weights because
not all of those scores or all of those characteristics
are equally important.
And we compute the optimal weights by performing a
quick search experiment.
So this is now the third part
of our recommender system
with two different experiments,
one's just using data from IMDb,
and one's using data from IMDb and the Reddit narratives.
So the second and third part
form our recommender framework
and for the fourth step
we can see the final recommendations.
So this picture should give you a short overview
over our recommender system.
For evaluation we use a training dataset
where we train our recommender algorithms
on data from IMDb.
For evaluation we use 1,480 Reddit submissions
brought into structured form by the crowd workers
where we use 80% for validation,
meaning that we perform grid search experiments
by optimizing our hyperparameters
and a 20% holdout test set for performance evaluation.
And the 80-20 split was done chronologically.
So for time reasons I just show you
the F1 score results at list length 10.
This line here is our base line,
our IMDb base line,
meaning that this is the performance of the very simple
and default IMDb recommendations,
as we have seen in the example before,
the 12 recommendations for The Dark Knight Rises movie.
So for our first experiment we obtained those F1 scores
and as you can see for all our five approaches
we can outperform the base line.
Those are the results when we just use IMDb data
for the post-filtering and re-ranking task,
and we additionally add the Reddit narratives
for the post-filtering and re-ranking,
we can gain a performance boost
for all of our five approaches.
The best performance were obtained with the doc2vec approach
and the Matrix factorization approach.
As you can see although the F1 scores are not that high,
we are more than two times better than the base line
with four of our approaches.
So the results show that adding information
from the target platform,
in our case the information of the narratives from Reddit
can gain that performance boost.
So to sum up our project,
we performed a crowdsourcing experiment
and also published this dataset.
We analyzed and evaluated
five different recommender algorithms
by calculating narrative-driven recommendations
and re-ranking the recommendations
with post-filtering techniques.
The results show that all approaches
struggled to match the human recommendations
but with all approaches we could
outperform the baseline,
and also for all approaches we could
improve the performance when
incorporating the narratives from Reddit.
So for future work we've planned to add
and analyze additional post-filters,
and also to extend existing algorithms
by incorporating the narratives from Reddit
already in the training phase
by, for example, additional regularization terms.
Also we have planned to investigate further domains,
for example book recommendations,
board game recommendations,
or video game recommendations.
Also we want to perform a qualitative evaluation.
You've seen our F1 scores were not that high
meaning that many of our recommendations
did not match the ones from the community.
But we do not know if they are really bad recommendations
or if they are good,
but just were not mentioned by the community.
But with a qualitative evaluation,
we could gain insights of how good
our recommendations really are.
Further we want to implement a recommender bot
to conduct further experiments on Reddit.
This bot should take the Reddit submissions as input,
compute recommendations,
and post its recommendations on Reddit.
This could look something like this
where we are on Reddit,
call our bot by its name, it's rbzio
We used this bot specific syntax where
we have G for genre: thriller,
M for movie: The Dark Knight Rises,
and K for keyword: mindfuck.
We post our request on Reddit
and if we then refresh the page
we can see that the bot already answered
with 10 movie recommendations.
So this bot is still under construction
but stay tuned because it will come soon.
This brings me to an end of my presentation.
I want to thank you all
and again give a reminder on our published dataset
which you can find on our website.
Thank you.
(crowd applauds)
- Okay, great, we have time for some questions.
We have about five minutes for questions,
so hopefully everyone had coffee
so you're able to ask some.
Okay, I'll ask a question
and give people a chance to think of good questions.
So I think back on slide eight or nine
where you showed a framework for your recommender system
there were two modes,
so you have the IMDb combined with Reddit
and then IMDb alone.
And these things are kind of in parallels.
So this to me looks like the architecture
for a hybrid recommender system
because you have two different methods
but I didn't see much detail
on how they might be combined
and what the benefits and drawbacks
of that combination might be.
- So this is just a very abstract and very simple
picture of our architecture,
and those two different experiments means
that they are not working parallel but they are separate.
So for the first experiment we have
our structured input, computer recommendations,
use just IMDb data for the re-ranking task,
and then compute the final recommendation.
The second experiment,
which is separated from the first one,
we also have the structured input,
again computer recommendations,
candidate recommendations,
and then we use both of those data
for the re-ranking task
and the final recommendations in the end.
I think what your question is regarding
is the combination of those two different data.
- [Host] Whether you think there could be benefit
and then maybe ways that it could be done.
- Okay. In our first experiment we have seen
that just using IMDb data,
without using data from the target platform,
we can outperform the baseline.
So we can gain already good results,
but incorporating additionally the data from Reddit,
from the target platform,
we can boost the performance.
How we did this, we have different scores
for each of the characteristics
and we have different weights.
And the optimal weights were computed
with a grid search experiment
so we have our optimal weights
for each of the characteristics
and sum them up.
And so each of the candidate recommendation
in the candidate recommendations list
has the final score which then brings us to
the final ranking of the recommendations.
- [Host] Thank you.
So a question from Peter here.
- [Peter] So I'm just trying to figure out
what is missing with IMDb?
(muffled speaking)
So it's exactly the same thing.
(muffled speaking)
I am puzzled why on IMDb,
is that we were told that you were looking at the reviews.
Why are reviews on IMDb
not performing as well as Reddit?
What is missing here?
The same data is more there,
and it doesn't work.
- Okay, so we think that the Reddit community
is some kind of different than normal people.
(crowd laughing)
Because if it wouldn't be the case,
it would just go to IMDb,
check out the recommendations, and take it.
They wouldn't go to Reddit
and ask for their specific requests,
but they do.
So there are some kind of special.
So they have their special requests
which are much more detailed than just going to IMDb
where you have 12 recommendations for one movie,
but as I said,
they do not necessarily be thrillers
and they do not have necessarily
this mindblending storyline.
So on Reddit you have this very specific request.
So people go there and ask for very specific requests.
And as our results have shown,
incorporating Reddit narratives,
so incorporating this information
with thriller and mindfucking plot as keywords,
in our re-ranking part could improve our results.
Meaning that if we take one special request
we have more data available
when incorporating data from the target platform
so we gain better results.
- Okay, thanks.
We can thank the speaker one last time.
(crowd applauds)
- Thank you.