Hi everyone, welcome to our presentation for the paper: Beyond Optimizing for Clicks: Incorporating
Editorial Values in News Recommendation.
My name is Feng, we were from the FD Mediagroep AI Team.
We worked with a bunch of data scientists and interns on the main focus of bridging
the AI and traditional media world.
And the news recommender system we are talking about today is one of our main projects we
have done for our organization's newspaper: Het Financieele Dagblad.
So, FD is a daily dutch newspaper, which focuses mainly on business & financial news.
It is a strong brand in the Netherlands and we offer paid subscriptions to users.
So, our readers are relatively very targeted.
In this presentation, I will show you how we give our readers personalized reading experience,
and at the same time, how we study and incorporate provider's editorial values in our system,
which makes our news recommendations benefit both providers and readers.
To achieve that goal, we have two research questions to be answered.
The first one is, we want to see whether our current recommender system can effectively
steer users to useful recommendations instead of just accurate recommendations.
And we'll define what ‘usefulness’ is later.
The next one is, noticing that dynamism is one of our main editorial values, we want
to answer the question whether we can effectively adjust our news recommender to steer our readers
towards more dynamic reading behavior while maintaining accuracy at the same time?
So, with these two questions in mind, we started to design our system.
The basic problem in our context is as simple as this, given a user profile and an article,
we want to predict whether the user will click on this article or not.
Since we are in the news domain, by nature, we always have the item cold start problem,
as we would like to recommend new articles as soon as they’re published.
So, to be able to effectively give our recommendations for continuously newly published articles,
we designed our system in a content-based way.
We utilize the point-wise learning to rank method and the recommendations are mainly
dependent on the content features from both users and articles.
So, in the training process, we use users' clicks as the positive samples and shown but
non-clicked items as the negative samples.
Since we do not want overfitting on the negative samples, we perform a sampling step to balance
out the data.
Each of these user-article pairs is then used to perform feature extraction.
The output feature vectors are then used for training our model.
We use a GBDT architecture here, which is trained nightly, using the user interaction
data from the previous 7 days.
The optimal number of 7 days for training were tuned in offline experiments.
As I mentioned, since we always have the item cold start problem.
The choice of our features are quite important to be able to perform meaningful predictions.
We use three kinds of features in our system, the article features, the user features, and
the user-article interaction features.
For the article features, we use both the metadata and the enrichments of the article
content.
The metadata features are like authors and sections, which are provided by the journalists.
The enrichments of the text are features like sentiment scores, and word embeddings.
For the user features, we don't use demographic features considering user privacy as well
as model bias.
Instead, we focus mainly on aggregated user reading behavior, such as the most read authors
and tags for each user.
Lastly, the user-article features contain both overlap features as well as the comparison
features, such as whether the author of the article is among the top-10 most read for
the user.
So finally, we combined all these three kinds of features into one big feature vector for
all the pairs.
The final vector contains around 14k columns, while most of them are categorical features
like tags and authors.
The full list of the features can be checked in our supplemental material.
Okay, so that's all about the system and feature design from the perspective of data scientists.
Since we were in a news organization that has a clear responsibility towards society
and also upholds our own editorial values.
So, we don't want to just build our system to be as accurate as possible for users, but
we also want to consider the providers as the stakeholders.
We want to study whether a news organization’s journalistic values can also be considered
of importance in the context of algorithmic personalization.
So, last year, our organization participated in a study by Bastian and Helberger in which
journalistic values emerged that are considered important in the context of news recommendation.
We followed this study up with our own interviews and meetings between FD’s developers, data
scientists, and journalists.
As a result, we defined the editorial values as the following four aspects, (i) the ability
to surprise readers, (ii) providing timely and fresh news, (iii) yielding more diverse
reading behavior, and (iv) increasing item coverage.
We conclude these four aspects as four editorial values, which are serendipity, dynamism, diversity
and the coverage.
We set out to explore how our news recommender can effectively incorporate these values.
And we do so by performing two user studies.
First, we aim to answer the first research question: “Does our current recommender
system effectively steer users to useful recommendations?”
We answer this question by measuring several aspects of recommendation ‘usefulness,’
and their effects on reading behavior.
We selected 115 power users as our target users in the first study.
We conducted the experiment in August, 2019 and collected the user interactions for around one
month.
As a baseline for our study, we consider a manually updated non-personalized top 5 of
highlighted articles.
These articles appear on the website in a grid at the very top of the frontpage.
And we present our personalized recommendations from three different sections on the web.
We first have a my news widget, which shows the top 5 recommended articles published in
the past 24 hours.
Then we have a section called MissedLastWeek, this section shows the recommended articles
published in the last seven days but not last 24 hours.
And we also have a separate page to show users all our recommended articles, but they are
not on the front page.
During this test, we measure and compare our readers’ reading behavior on manually curated
and recommended article lists.
To get a sense of general system performance, we also measure the system's accuracy in NDCG,
precision and recall.
For the diversity measurement, we employ the commonly used intra-list diversity, we measure
similarity between articles using different article attributes like author(s), tags, sections,
and word embeddings.
For dynamism, we measure the inter-list diversity by checking how much an article lists changes
between two updates.
We model the serendipity as an article’s dissimilarity to a reader’s historical reading
behavior.
And lastly, for coverage, we model it as the percentage of daily published articles that
are shown to users.
Here are some of the results for our first study.
I won't go over all the numbers here, but I will give you the insights of our findings.
So, for accuracy, we find that users' average clicks are relatively high in our recommendations.
Recall scores show that over half of the clicked articles rank among the top 10 recommendations.
So, we consider our rough recommendations are not bad.
For the diversity, we find that our top 5 recommendations are more diverse in terms
of ‘section’ and ‘word embeddings’, which means they are more diverse on the topic
and content.
For the tags, they are actually both quite diverse.
The reason is that, there are many hundreds of unique tags in our organization, and they
actually usually have a low overlap between articles.
That's why they are both quite high on diversity here.
In terms of ‘authors,’ we see that the highlighted articles show higher diversity
than our recommendations.
This might be explained by our finding that ‘authors’ and ‘tags’ are two of the
most important user-article features in our models, which means that recommendations will
tend to be personalized more strongly towards ‘authors’ and ‘tags’ that are similar
to our users’ reading history.
For dynamism, we find that our recommendations might not change frequently, but once they
change, they introduce more new items into our lists.
For serendipity, we find that our manually selected lists are actually more serendipitous
than our recommendations in 'tags' and 'authors', while our recommendations are a bit more serendipitous
on the content.
Again, as I mentioned, our model is more personalized on 'authors' and 'tags', and that's why they
show less serendipity here.
For coverage, we see that our recommendations actually provide a narrow set of articles for each
user, which makes sense since we want our system to be personalized.
While at the same time, if we consider the recommendations for all users, we actually
find that the overall coverage of recommended articles is much higher than the manual selection,
which is exactly what we expect.
Having noticed dynamism as an important editorial value for our news personalization, we set
out to answer our second research question: “Can we effectively adjust our news recommender
to steer our readers towards more dynamic reading behavior?”
In order to do this we perform an intervention study, where we bias our recommender system
towards higher dynamism.
For our second user study, it is pity that we only applied a single usefulness treatment
out of the four that we have studied.
One reason for this is some real world constraints like page loading time and some other technical
requirements.
Another reason is regarding the concerns about exposing users to too many treatments might
reduce the quality of our recommendations.
So, for our dynamism online test, we set our original recommender system as the baseline
and another system which steers towards more dynamic recommendations as the variant.
The test involved over 1000 highly active users.
And our recommendations are shown in the same 3 sections as I mentioned before.
We utilize a linear re-ranking method to incorporate the article recency to our final ranking scores.
So the basic idea is that we mix recency with relevance, such that more recently published
articles will get higher rankings.
As the findings for our second user study, for accuracy, we find none of our sections
show statistically significant differences between the treatment and the baseline, which
means that there is actually no big difference in accuracy.
While at the same time, we do see that the variant recommender system is able to effectively
increase dynamism to users' recommendations without losing accuracy.
We also observed there is some discrepancy in accuracy between the offline results from
User Study 1 and the online results in User study 2, which is in line with previous work.
Since this work was done in the context of a real website with live users, it is not
possible for us to release user data and pre-trained models.
We provide as many system information as possible in our supplemental materials.
Besides that, we also have a few limitations both in system and experiment design.
One of the limitations is that we are learning from clicks instead of from users’ actual
preference, and we also did not consider the presentation bias in our system design.
While for this study we consider these issues out of scope, which we find are common in
the news domain.
Instead of directly using the user interactions in our first user study, we actually used
simulated offline evaluation.
And our second user study is also a part of a bigger online test for our website.
It was not feasible to claim our test users exclusively for recommender system testing.
For instance, when we were running our second user study, there were also some other A/B
tests running at the same time like the front page layout changing, which can have some
interference to our study.
So, as our conclusion, throughout all our user studies, we find that news recommendation
can benefit both news providers and readers.
From the provider’s perspective an increased overall item coverage means that content will
be served to the intended target audiences, which may keep readers more engaged and overall
provide economic benefits.
From the news reader’s side, benefits include being served content readers may not have
found on a non-personalized front page by themselves, and an increased diversity of
news consumption.
Finally, our study also shows that algorithm design with multiple stakeholders might not
need to be a tradeoff, but they can be fruitful for each, as multiple goals can be achieved
at the same time.
That's all.
Thank you.