hi my name is Napol Rachatasumrit today
I'll be presenting highlights from our
work on ForSense
a browser extension for improving
people's online research
in the past couple of decades the
internet has become an essential tool
for people to access information and
conduct research
in 2018 the data from Pew Research
center said
the 96 percents of adults in the u.s rely on
online searches to make big decisions
and we can still see similar stat today
sometimes we only need to find an answer
to a particular questions like
what is the address of the restaurant
but many times there is no one clear
single answers
like which car to buy or how to build a
computer and the focus of this work will
be on the latter
unlike the question answering tasks this
kind of exploratory research it
is very complex and requires a
significant amount of cognitive effort
it usually involves going back and forth
between different tasks multiple times
even though there are different tools
that help people in gathering or
organizing information
the process usually involves moving
information across many fragmented tools
that cause distraction and information
loss our goal is to improve the
efficiency of the web browser-based
online research experience
by leveragingg sense-making theory and
modern machine reading capabilities
we see the opportunities to improve
online research tasks by framing them as
an integrated sense making experience
we can separate online research into two
interconnected activity loops
for rating and sense making during the
foraging people search and collect
information from various sources
while in the sense making people form a
schema of the information they collected
during the research process people
iterate through these loops multiple
terms and usually go back and forth
between them repeatedly
this framework highlights the importance
of an integrated tool
that let people seamlessly alternate
between two stages when doing research
vehicle that such integration leads to
less fragmented experiences
with less distraction arising from
context or complication switching
the framework also brings attention to
potential machine supports for both for
etching and sense making across
different stages in the process
in this project we use BERT which is a
transformer machine learning model
in order to encode text input into a
numerical vector representation called
embedding
then we can compare semantic similarity
between two pieces of text
by looking at the distance between the
embeddings with a sense making theory as
a design framework
our design goal is to support the
iterative nature of online research
including the frequent activity
switching between information and sense
making
and incorporating uninstrusive machine
support to the process
we presented the following four design
principles to guide our system design
first our system will use a smaller unit
of analysis
such as a paragraph of text instead of
the whole webpage
we believe these smaller chunks are more
thematically coherent
that would be more useful for
schematization and machine support
second we aim to explore opportunities
to leverage motor machine reading
capabilities
for the system to be able to support
online research
third we want to eliminate the cognitive
loads from fragmented tools by providing
an integrated experiences
that make it easy to navigate between
forwarding and sense making activities
lastly the matching supports can be
unhelpful and incorrect at times
so it is important for our system to
give people abilities to recover from
those
areas easily in order to access the
effectiveness of our approach
we build a design probe to explore and
gain insights into
the opportunities to improve online
research using current advances in
machine reading
the benefits of integrating multiple
stages of online research in the same
tool
and the effective strategies for
handling imprecise machine suggestions
the goal of a design probe is not to
argue that our approach is superior
or quantitatively measure the
effectiveness of the users
but it focuses on the insights that we
can learn through observing user
interacting with the system
we have built ForSense a browser extension
that supports
and improve the online research
experience it consists of
the browser extension and server
components we focus on the browser
extension in this presentation but more
information
about the server components can be found
in the paper
the first component of the extension is
the clip sidebar the support foraging
activities
it allows users to highlight interesting
parts of the web pages
and collect them as clips the sidebar
will act as a shoebox and keeps a
collection of clips that user collected
users can also assign clips to a
different group once
groups are created with clips in them
the system will use the embeddings of
clips in the group to calculate and
suggest relevant clips in the web page
then users can either accept or ignore
those suggestions
by clicking on the icon next to the
saturation users then can switch to the
sense making canvas
by clicking on the canvas button in the
right corner
the sense making canvas is where users
can organize clips into different groups
and spatially arrange them to make sense
of the topic the groups that were
created here
will also be represented in the sidebar
using the group color on the left of the
clip
on the other hand when you add a clip to
a group in the sidebar the clip will
also automatically put in the group on
the canvas
similar to the paragraph suggestion in
the sidebar versions will use embeddings
of clips in the group to provide other
group capability
users can hover on the other group
button in order to see the suggestion of
a group
that the clip might belong to the clip
will be automatically put in the
suggested group when users click on the
button
and they can also undo it if they change
their mind
we conduct a user study with
ForSense as a design probe
to gain insight into our research
questions
in our user study we recruited 10
participants with a diverse background
ranging from designers pms engineers and
researchers
our study sessions start with a quick
video tutorial of the system
and then we let the participant practice
using the system for 10 minutes
after that we asked the participant to
conduct online research
in order to prepare an outline for an
introductory presentation about a topic
the participant can choose a topic from
the list that we provided
but they also can come up with their own
topic the topics are very diverse in
order to make sure that the participant
does not have prior knowledge
after 30 minutes of research we asked
the participant to create the
presentation
outline for 10 minutes finally we
conduct a subway and say my structured
interview about the experience with the
system
in the next session we will discuss the
result and main insights from our study
overall the participants found ForSense
to be easy to use
with a mean sus score of 72.3
on average there are five groups created
with the around three clips per group
our first research question is to
explore how do people benefit from the
machine support we found that this clip
suggestions while sometimes not accurate
can still be beneficial for users
usually participants will skim quickly
through a website and could miss some of
the relevant sections
these suggestions would bring their
focus to those sections
even when the suggestions were imperfect
the participants would try to recognize
the connections and use them as
inspiration to explore deeper
and also seeing this suggestion are easy
to disacquire or undo
it does become less critical even they
are not totally accurate
our second research question is how
useful is the integration
of operating and sense making activities
in a single system
the findings show that the integration
just in fact helps user to be more
efficient
and reduce distractions from moving
between apps and allow them to stay in
the flow during their research
for example participants like the
ability to assign clips to groups
without having to pause their foraging
and switch to the cameras with all the
other apps
they also like the canvas as a way to
externalize that current mental model
test it and refine it
our last research question is to explore
the effective strategies for handling
imprecise machine support we found that
participants often
form a mental model about how the system
works and how they can affect it through
the interaction with the system
when we let participants initiate the
machine support by first recreating
group themselves
they become more forgiving to the
incorrect situations sometimes they even
rationalize the suggestions that they do
not agree with
and try to think about how they can
improve it
many participants also express the
desire to have more options to fix the
incorrect suggestions
for example the ability to highlight is
part of the clips that is
relevant or provide related keywords for
our group
based on the findings we identify a
couple of key design recommendations to
incorporate machine support in online
research system
first imprecise machine support is more
useful than we expected
especially when it's not critical it can
help users re-evaluate their
perspectives
on the topic and explore more diverse
information
next is to use chart clips as a unit of
information instead of a whole web page
they allow participants to be more
expressive in the way they build their
understanding about the topic
these clips are also more manageable and
semantically coherent compared to a
webpage
the third recommendation is to identify
and design for these teaching moments in
this intelligent system
more expressive ways to teach the system
or correct the mistakes
are welcomed by the users and can help
increase personalization
finally we think it will be useful to
think about complementing before
thinking about collaborating in the
human ai team
this perspective helps relax our
expectation on the system
and focus on how a person's capabilities
can be augmented instead of replaced by
an ai
to recap in this work we aim to improve
an experience for online research
which is a crucial task but also quite
difficult because of its complexity
and fragmentation in the current two
sets inspired by sense making theory
we have designed ForSense a browser
extension that supports the experience
by integrating for writing and sense
making activities
and increasing machine agency through
machine reading
we use ForSense as a design probe to get
insights
into how people will interact with such
system
main insights from our study include
machine support can help accelerate the
process
and seamless integration between two
stages reduce distraction and help users
stay in the flow
we also found that even imperfect
machine interventions could be useful as
participants sometimes welcome them as
inspiration to explore deeper
finally this work would not have been
possible with all my co-authors
my mentor Gonzalo and my collaborators
Jina Rachel and Chris
thank you for listening to our talk