Andrew Kuznetsov (CMU): Hello, my name is Andrew Kuznetsov, and today I’ll be presenting Templates and Trust-o-meters. Before I begin, I’d like to thank my collaborators at Wikimedia.
Andrew Kuznetsov (CMU): Since its creation, Wikipedia has become one of the most visited websites for information on the internet, and has been compared in quality to Encyclopedia Britannica.
Andrew Kuznetsov (CMU): Since Wikipedia is free to edit by anyone it's important their readers understand how trustworthy the content they are consuming us.
Andrew Kuznetsov (CMU): Some skeptical readers find it hard to believe that good content could be produced from the combined efforts of many, often anonymous editors.
Andrew Kuznetsov (CMU): On the other hand, some Wikipedia readers are overly optimistic, and keep a high opinion of Wikipedia without understanding how it is created.
Andrew Kuznetsov (CMU): In fact, showing these users the actual process lowers their trust of Wikipedia.
Andrew Kuznetsov (CMU): Jimmy Wales, a founder of Wikipedia, frames this problem in a lighter analogy: Wikipedia is ‘like a sausage: you might like the taste of it, but you don’t necessarily want to see how it’s made”.
Andrew Kuznetsov (CMU): This is less than ideal, as we want users to learn from these signals, and maybe even learn to trust content if it’s good.
Andrew Kuznetsov (CMU): Previous researchers have explored a different interventions to help readers make trust decisions, such as indicators, but several crucial barriers remain to designing trust indicators to help readers today.
Andrew Kuznetsov (CMU): We believe these barriers translate into several research questions there's a lot here so i'll walk through them.
Andrew Kuznetsov (CMU): First, anyone designing an indicator for trust would need to pick what information to include in their indicator. They’ll want to know which metrics are influential and which users won’t care about.
Andrew Kuznetsov (CMU): Unfortunately, prior work has not directly measured and compared the effect of showing different article quality metrics. It is unclear what metrics should be prioritized.
Andrew Kuznetsov (CMU): Second, in order to be effective, an indicator would need to be noticeable by users when it’s placed within the article. Any future indicator would need to be created in a way that’s compact and stands out.
Andrew Kuznetsov (CMU): Unfortunately, previous work has involved participants trained to notice the indicator. As a result, it’s not clear how to best place compact indicators that could otherwise be missed by users.
Andrew Kuznetsov (CMU): In our work, we study untrained users to understand how to best integrate compact indicators alongside a Wikipedia page.
Andrew Kuznetsov (CMU): Lastly, in order for an indicator to be effective across an entire site like Wikipedia, it would need to be influential across many diverse articles and readers. It’s possible that two different Wikipedia readers could interpret the same score on an article very differently.
Andrew Kuznetsov (CMU): Prior work regarding editor discussions has suggested that users have diverse, and often negative reactions to seeing the behind-the-scenes work of Wikipedia.
Andrew Kuznetsov (CMU): This raises concerns that metrics showing this type of information may not be able to raise user trust.
Andrew Kuznetsov (CMU): Bringing it all together, this work has three main focuses as we try and understand the space of trust indicators.
Andrew Kuznetsov (CMU): What signals are most impactful for reader trust?
Andrew Kuznetsov (CMU): As previously mentioned, in experiment one we set out to find impactful signals of article quality
Andrew Kuznetsov (CMU): Experiment 1 consisted of several parts. First, we created positive and negative notices that represented metrics we wanted to test.
Andrew Kuznetsov (CMU): Second, we sampled Wikipedia articles at different levels of quality and controversy.
Andrew Kuznetsov (CMU): Lastly, we conducted a MTurk study with a user trust prompt. In this slide I’ll briefly go over how the notices were generated.
Andrew Kuznetsov (CMU): To create our notices, we started by clustering article trust related information from several relevant sources.
Andrew Kuznetsov (CMU): In our results, we found significant impacts from many notices, but none from any positive notices.
Andrew Kuznetsov (CMU): Among the negative notices, metrics related to references, neutrality, and incompleteness significantly decreased user trust.
Andrew Kuznetsov (CMU): Taken as a whole, this suggests there are a number of metrics we can use in a trust indicator to decrease user trust, but don’t surface any metrics that could increase reader trust perceptions.
Andrew Kuznetsov (CMU): In experiment one, we found impactful signals by comparing them directly.
Andrew Kuznetsov (CMU): However, while doing this we noticed another factor that had likely affected user trust - salience. Within Wikipedia, it is well known that notices compete for reader attention. However, from our last experiment we saw that if we were to shrink these notices down to just the influential information
Andrew Kuznetsov (CMU): users would miss the intervention nearly 50% of the time.We needed to find configurations that would enable compact indicators to overcome this issue.
Andrew Kuznetsov (CMU): This directly motivated our second experiment, which I will now describe. First, we created several forms based on an impactful notice from a previous experiment.
Andrew Kuznetsov (CMU): Next, we placed these indicators on an article. Finally, we measured if users had noticed the intervention.
Andrew Kuznetsov (CMU): The indicator forms that we created and tested can be seen here, categorized by high and low footprint.
Andrew Kuznetsov (CMU): We created them to test a number of modifications to the current way Wikipedia banners are placed, such as doubling the height of the notice, adding a slow blink, or using small icons with the notice as a pop-up.
Andrew Kuznetsov (CMU): Our results demonstrated that the most effective modifications were those involving position and motion.
Andrew Kuznetsov (CMU): In experiment 2, we discovered several modifications that could help compact indicators stand out.
Andrew Kuznetsov (CMU): However, while doing so we noticed that there is some natural variety to how users responded.
Andrew Kuznetsov (CMU): First, together with our product design collaborators at the Wikimedia foundation, we created this indicator using existing guidelines on trustworthiness aids
Andrew Kuznetsov (CMU): as well as some learnings from our prior experiments. It is above, with details in the paper.
Andrew Kuznetsov (CMU): Like the previous experiments, experiment 3 consisted of several parts. First, we created six different conditions (one for each value) and one without the indicator at all.
Andrew Kuznetsov (CMU): Then, we gathered a new set of articles at different levels of quality and controversy
Andrew Kuznetsov (CMU): Lastly, we used MTurk participants and a prompt for change in user trust from prior literature. Our results demonstrate that a trust indicator can both consistently raise and lower article trust in the appropriate direction across pages of different quality and controversies.
Andrew Kuznetsov (CMU): Our results also showed that trust indicators can significantly raise a user’s perception of Wikipedia overall, even when the readers look at a single article.
Andrew Kuznetsov (CMU): Since most popular Wikipedia articles have positive metrics that would put them into the green side of the indicator, this suggests that implementing trust indicators across the website would result in a significant lift of Wikipedia platform trust.
Andrew Kuznetsov (CMU): Finding that trust indicators could raise the trustworthiness of the entire platform was an exciting result, and we decided to dive further into the details. Like before, we noticed that their reactions to the trust indicator seemed to be quite varied.
Andrew Kuznetsov (CMU): This made us wonder if there was anything specific that caused this divide in opinions, so we looked at multiple factors.
Andrew Kuznetsov (CMU): This led us to conduct additional regression analysis. Doing so surfaced a significant interaction with talk page experience, leading us to believe that the indicator effect is stronger for users not familiar with how Wikipedia works.
Andrew Kuznetsov (CMU): More generally, our results regarding trust indicators point to several future considerations. First, we highlight that community feedback is important to determining which indicator values should be shown on which pages.
Andrew Kuznetsov (CMU): Second, we show new, impactful metrics that may require addition processes within Wikipedia to reliably compute.
Andrew Kuznetsov (CMU): Lastly, we see indicators as an opportunity for creating new editor experiences.
Andrew Kuznetsov (CMU): In conclusion, our big key takeaway is that trust indicators can raise Wikipedia trust. Specifically, they can do so up or down, appropriate to the situation at an article-level
Andrew Kuznetsov (CMU): They can also be configured to help them stand out to readers.Lastly, we find that trust indicators drop Wikipedia trust significantly more for those without Wikipedia discussion experience. Thank you