In this video, I'll present the paper titled "Mesh: scaffolding comparison tables for
online decision making." My name is Joseph Chang. This paper is joint work with Nathan
Han and Aniket Kittur. Exploring formation online is how many of us make sense of the
world today. We browse the internet to support all kinds of decision making from things like
Who should I vote for? What neighborhood should I live in? To which desk lamp Should I
purchase this work explore ways to support online decision making in the domain of
product comparison research. So that's imagine you go on Amazon tried to find the right test
line for you. Immediately, you get these hundreds of different options in your search
results. Each of them have hundreds to thousands of reviews, and many of them are
highly rated. So you might start by clicking on this first option.
Now you're on the product page.
And as you read the descriptions and reviews, you start to pick out interesting criteria
that fit your personal context.
So things like some of them support glare reduction,
or whether it has USB outlets for charging your devices. Some of the criterias might
require you to read many reviews to get a better sense of how well this option perform.
So here you see there's a bunch of reviews mentioning that this one can flicker, which
seems pretty bad. At this point, you picked up a set of criteria that you care about, you
have a pretty good sense of how well this first desk lamp fits your goals. But when you
go back to the search result, you still have all these different options that you need to
figure out. Ideally, you want something like this a comparison table with all the options
compared using all the criterias that you picked up through your exploration. This
allows us to see clear trade off and make purchase decisions. But in reality, the
process looks more like this. to characterize each of these options, you need to click into
each of them with a bunch of reviews and descriptions to figure out how well they fit
your personal context. Now there are many top down approaches that help with this process.
So things like making a direct product recommendation, review, ranking and
summarizations, or interfaces that allow you to navigate across different types of options
more efficiently. But fundamentally, user still need to take a bottom up approach in
many cases, figuring out what criterias are useful for them. And reading reviews and
information is across different sources to make up their own minds about these different
options. So what tools do people currently use to support these online comparison research,
we did a survey with 89 participants, and we found that around half of them built
spreadsheets in the past to help them keep track of their online decision making. So
here's one example of what these type of spreadsheet might look like. This one is a
spreadsheets for comparing different espresso machines.
There's, you can see basic product
information on the left like price and model names. And then there's some objective facts
about these different machines like specifications. And then you have these more
subjective notes on the right for things like the design or whether it is easy to adjust and
use. So about half of the participants that that they've created spreadsheets like this in
the past, we interviewed 30 of them each for 60 minutes, and then we came up with three
high level design goals. Firstly, we found that people want to compare multiple options
using the same set of criteria. But they also pointed to how switching back and forth
between browser tabs and spreadsheets to compare their options is a pretty frustrating
process.
Secondly, they emphasize on the importance of seeing raw evidence instead of just
aggregation. So instead of seeing just the average review ratings, they want to read
reviews, so that they can make up their own mind and make their own judgments on these
individual pieces of evidence.
And thirdly, they want to work in a more organized fashion, describing their current
process as chaotic, are often falling into rabbit holes, pointing to experiences where it
can be hard to keep track of their overall progress and focus on the important criteria
that they set out to investigate. So based on this design goal, we came up with a Chrome
extension called Mesh. When you open Mesh, it recognizes all the Amazon product pages you
have open in your browser tabs. This allows users to import them into a project. And it
also allows the system to fetch product informations and reviews from Amazon. After
importing options are shown as columns in the table, then Users can also add additional
online sources about each product by pasting in URLs. So to minimize the effort for
comparing different options using the same criteria, user can create criterias in their
tables as rows. The system will in turn fetch reviews that mentioned the criteria name for
each option, and show the average
rating as the default values for the cells.
Users can then click on a criterion to see relevant evidence side by side for each
option. This included reviews, product descriptions and sentences from the additional
sources that contain the criteria name. Based on examining this evidence, users can then
override the default ratings with their own and also take short notes to record their
opinions from investigating and reading these evidence. As the user explore more evidence
They gradually built up a comparison table that contained their own options and criteria,
and also how they interpreted evidence to characterize each options in this more
organized way. To help them better compare their different options, they can now then
reorder or sort their options based on the ratings of each criterion. They can also
assign weights to their criteria, and the system will calculate a weighted average
rating in real time.
So that was a high level walkthrough of the system. And there's more details in the paper.
But does it actually work? So for evaluation, we conducted three studies focusing on three
different aspects. Study one focused on interaction, where we wanted to figure out if
users can be more efficient using the tool study to focused on learning, we wanted to
figure out if users can learn more insights from data using the tool. And finally, for
study three, we conducted a field deployment where a participant installed Mesh on their
own computers and conducted their own real world tasks for a prolonged period of time.
So starting with study one, we wanted to investigate how quickly and accurately users
can figure out can find answers using Mesh. To measure this, we gave them a blank table with
fixed options and these objective criteria that had gold standard answers.
So the task we used was to fill out these criterias for espresso machines, which
included things like the dimensions of the machines, and the water tank sizes of the
machines.
We use that with a subject baseline condition, where we turn off the feature that
automatically gathered evidence across different sources. So the baseline condition
is similar to users current process where they have to browse the Amazon product pages or
search directly on Google or other search engines to find answers for these product
specifications.
And in the system condition, they see relevant evidence that the system pulled in from
different sources, but they also have to learn this new and unfamiliar interface during the
study. So here are the results of our study one, the red bars were from the baseline
condition, and the blue ones were from Mesh. At a high level, participants found
significantly more correct answers while using Mesh. It also took them significantly less
time to find those answers. More surprisingly, based on the NASA TLX questionnaire, they
actually proceed using Mesh as having a lowered workload, even though they had to
learn this whole new interface that they're not familiar with during the study. So this
first study sort of tested the usability of the interface and found that participants can
work more efficiently for these objective criteria that had correct answers. For the
second study, we look at criteria that were more subjective and experience based, so this
task can be more challenging for the first one, because participants have to look at
multiple reviews to better characterize each criteria for each options. To control for task
complexity, we used a fixed persona for buying a robot vacuum. The persona has three
criterias that were subjective and experience based included that the model should not get
stuck too often, you should not be too loud, it also should be effective when cleaning up
dog hair. So we also gave each participant five fixed options that were all highly rated
on Amazon with hundreds of reviews. For the baseline condition, we'll compare Mesh to
Google Spreadsheet as a between subject condition. Google spreadsheet is a tool that
people commonly use to support this type of research based on our preliminary survey. So
we have participants each conduct 20 minutes of research using their assigned tools, and
then write a short summary of what they've learned during that 20 minute period. So this
is a common practice for evaluating these research support tools to try and capture how
well people have learned in a fixed amount of time. We then rated each of the learning
summaries on these four aspects. The first one was how relevant it was to the persona. This
is based on how participants in the preliminary interview said that they sometimes
get lost in the research process, and it can be difficult for them to focus on the
important criteria that they set out to investigate. The second one was how insightful
the summaries were this aimed to measure the amount of information that they've learned in
that 20 minute time period. The third one was confidence, which we used as a proxy for
measuring how well it can support decision making. And then the fourth one was usefulness
which we use as a general quality control measure. Each of these were, rated using a
seven point Likert scale for agreement by five crowd workers. And then we took the average as
the final score for each aspect. We also showed the coworkers the same persona that we
use in the main task when they were writing the summaries, but they were blind to which
conditions generated each summary. So here are the results for study to the red bars again,
start from the baseline condition, in this case, Google Spreadsheet Overall, participants
who used Mesh generated summaries that were rated significantly higher on these four
aspects. This suggests that the approach of scaffolding these comparison tables allowed
participants to focus on the important criteria in the persona that we gave them, and
allow them to gain deeper insights from data during that 20 minutes of research. So here
are two example summaries that were representative of the two conditions, the one
on top us from the baseline condition. You can see here, the summary focus more on how all of
the options were very similar and highly rated and it can be difficult to tell them apart.
Whereas the second one from a participants will use Mesh, this person was able to point
out meaningful differences that he or she saw in data. The first two studies were lab
studies where we control for tax complexity using fixed options and criteria to test match
in a more realistic settings with deployed the system for two weeks to five participants in
study three to collect more on qualitative insights. You can see in the table that
they've conducted a wide variety of shopping tasks, ranging from clothing, appliances, cat
food, furnitures, and more. Each of them were interviewed for one hour initially, and then
one hour after the deployment we then encoded the transcripts in three passes using a
grounded theory approach. So before I get to the themes, from the interviews, we also
collected usage statistics on the details are in the paper. But the main thing here is that
all of them spent a significant amount of time in the systems and were all actively engaged
using all the different features.
So there are many interesting quotes in the paper on their strategies and how they
benefited from the system. In the interest of time, I will go through two main themes. The
first one all participants described how Mesh allowed them to take a more organized and
structured approach, which was one of the three design goals we set out to support. They
also mentioned a lower interaction cost when managing multiple information sources and
consuming evidence from them. Two of the participants specifically mentioned how the
table structure is, is more suitable for this type of online research when compared to the
linear structure of browser tabs. Four of the participants made actual purchases within the
two week study period, showing that these tests that they were conducting were
realistic, and people can use Mesh to support making that final decision. Many participants
also compared Mesh directly to using a spreadsheet. Here's one of those quotes. Pause
the video if you want to read it. The second most common theme was that participants talked
about using the default star ratings to prioritize their research efforts. So these
this is the yellow stars that show up immediately after they create a new criteria
row in the system. And they describe using it to focus on criteria focus their investigation
on criteria that had large discrepancies between their options. This potentially
allowed them to rule out more options early on the process and shortcut their research
process. They also describe how these default ratings allow them to see meaningful
differences between all these options, that can be very difficult to differentiate on
Amazon because they all had highly, very high average ratings. Finally, they also talked
about using their existing criteria and the system can allow them to quickly evaluate
newly added options that they were considering. Here's one example project from
one of the participants, you can see that there were multiple options and criteria as
some of the ratings were replaced by the user's own personal ratings. There was also
notes in different cells. So to conclude, online decision making can be challenging. One
reason is that evidence are often scattered across different browser tabs. And they can be
also be biased, subjective and conflicting. In these scenarios, user often need to compare
multiple information sources and make their own judgments about data. So for this, we
introduce Mesh, a system where users iteratively build up a comparison table by
discovering options and criteria as they explore evidence. The system was based on
three design goals that we developed with an exploratory interview, we conducted three
studies showing that Mesh can allow users to work faster, more accurately and with a lower
perceived interaction costs when figuring out objective criteria. We also showed that showed
that when they were working with subjective criteria, they were more insightful and more
confidence during a 20 minute study. And finally, we conducted a field deployment study
showing that people can use Mesh to conduct real world tasks. And showed the ways they
benefited from using from building up these comparison tables and making that final
purchase decision. So finally, we show that providing better scaffolding support. In this
case, creating these comparison tables can benefit users not only by lowering interaction
costs but also allow them to gain deeper insights from data to support online decision making.