Hello everyone, my name is Mingming Fan, and I'm an assistant professor at the Hong Kong University
of Science and Technology in Guangzhou. Today I'm going to present our research on human-AI
collaboration for UX evaluation: effects of explanations and synchronization. This work was
done with my co-authors Xianyou Yang, Dennis Yu, Vera Liao, and Jian Zhao. Usability testing with
think-aloud protocols is widely used in industry to detect usability problems. During the test,
participants will work on the tasks while at the same time verbalizing their thought processes.
During the analysis, evaluators will watch the recordings to analyze the think-cloud
verbalizations among other behaviors to identify usability problems. However, manually analyzing
usability test videos is arduous. Reviewing these video recordings is time consuming, and the
evaluators may face the "evaluator effect", which means that they may miss certain UX problems due
to their personal biases. Therefore, UX evaluators are encouraged to collaborate with each other.
However, in practice, they often analyze test sessions alone. Very few had a chance to gain
a different perspective from their co-workers. In light of the advancement of AI, researchers
have investigated the human-AI collaboration to improve the UX evaluation analysis efficiency.
One representative work of this line of research is called Vista. Vista is a visual analytical
tool that visualizes an AI's predicted usability problems as a sequence of spikes on a timeline.
It also visualizes the AI's input features as the explanations of its behavior to the evaluator.
Prior Studies have suggested that the human-AI collaboration allows the evaluators to identify
more UX problems than they work alone. However, the prior human-AI collaborative tools suffer
from important limitations. For example, such tools often show the AI's predictions all at once
upfront before the evaluators even start their own analysis. This might bias their analysis.
This type of collaboration between the AI and the evaluator is asynchronous.
Moreover, evaluators wanted to know more about how the
AI makes decisions in addition to seeing its input features.
To improve the human-AI collaboration, we draw Inspiration from human-human collaboration,
which has two dimensions the first dimension is called "work context", which means "how
do evaluators collaborate?" In addition to asynchronous collaboration, there is also
synchronous collaboration. The other dimension is "agency", which means "how much do users feel
they're in control?" The agency can be moderated by the amount of explanation provided to them.
In order to investigate the "work context," in addition to any synchronous collaboration as
investigated in prior studies, we also proposed a new design for "synchronous collaboration", in
which the AI shows its predictions progressively to the point where the video is being played.
We also revise the explanations to provide more information that the evaluators wanted
from prior studies. For example, we included the UX design guideline that is violated, and we
reorganized the input features as a set of icons and scannable text to facilitate the consumption
of such information. With these new designs, we investigated two research questions. The first one
is "how do the synchronization of AI and evaluator affect the evaluator's performance and experience
in their analysis?" The second one is "how would explanations of AI affect the evaluator's
performance and experiences in their analysis?" To answer research questions, we designed
experimental designs with four conditions. Next, I'm going to explain these four conditions.
Asynchronous AI without explanation is similar to some of the prior studies designed in the
sense that the AI presents all its predictions at the very beginning of the analysis so that
when the evaluator starts their analysis they already see all the AI's predictions.
Asynchronous AI with explanation is similar to the prior interface except that it also shows the AI's
explanations of this behavior. Synchronous AI without explanation, on the other hand,
reviews the AI's predictions progressively as the video is being played. Here is a short video demo.
As you can see that as the video is being played
the AI's prediction is revealed to the evaluator progressively
Synchronous AI with explanation is similar to the prior interface
except that it also shows the AI's explanations.
With these four designs, we conducted a two-by-two mixed-design study with these two factors. We
recruited 24 UX evaluators with various prior UX experience to work on tasks in which they identify
the UX problems of two test videos using these four interfaces .we've performed the mixed-effects
regression model on quantitative data, such as the number of UX problems evaluators identified,
the number of video jump forwards and rewinds, including also the subject ratings.
We performed the thematic analysis of qualitative data such as interview data.
Due to the time limit, I only highlight some of the key findings here. This figure shows the
number of UX problems the evaluators identified. Here is the asynchronous AI without explanation.
As you can see that with synchronization evaluators identified significantly more
problems. Similarly, with explanation evaluators also identified more problems.
We evaluated the engagement in the analysis by looking at evaluators' video playing behaviors,
such as how many times they forward jumped the video or they rewinded the video, which were
shown in these two figures. As you can see that with synchronization evaluators jump forwarded and
rewinded the videos more often, which means they engaged in analysis more. Similarly,
with explanation evaluators also engaged in the analysis more. With synchronous AI, participants
felt as if they were working with a "co-worker" at the same time. With explanations, they felt
that they understood how the AI works better. So the key takeaway is that both "synchronous AI" and
"AI with explanations" could improve evaluators' productivity of and engagement in their analysis.
For future research directions, one interesting direction is to look at other synchronous
collaboration modes, for example, on-demand and synchronous collaboration, in which AI reviews
its decisions and rationales upon requests. It will be also interesting to look at human-AI
collaboration for identifying common usability problems across multiple usability test videos.
With that, I'll conclude my talk. I'd like to take any questions. Thank you for watching.