Hi, I am Leon and I am presenting our study
which compared the use of voice versus screen-based interactions for agent-assisted tasks.
In the past 5-10 years conversational agents have become commonplace
taking on various roles to help us complete simple everyday tasks or provide information or basic support.
When we interact with conversational agents there are mainly two types of modalities:
voice-based ones such as smart speakers, or text-based or screen-based ones such as chatbots.
Most of these conversational agents are still rather passive, waiting for the user to make a request.
However, with technological advances they will become more and more capable
to also proactively intervene by themselves, which leads us to the question of
how best to design such proactive assistants, including which modality they should use.
[Simulated Voice Assistant:] Leon, maybe you would like to go to the next slide?
Sure, thanks for this proactive suggestion.
Obviously, in certain situations, one modality may be more desirable or effective than the other
which depends on various factors.
For example: Is it a work or a home environment? Is there one person present or multiple?
All of these aspects are important when deciding which modality to use.
Here we are specifically interested in which modality is more appropriate or effective
in a collaborative situation where people perform a task together, being co-located.
The scenario that we consider is collaborative exploratory data analysis,
which is a common task performed by knowledge workers across various industries.
Think of, for example, marketing managers looking at recent sales data together to identify trends and patterns.
It is also an interesting scenario for a proactive agent because many people have difficulties
reading data visualizations properly and discovering interesting trends and patterns in them.
In our study task we used 22 visualizations from the Global Burden of Disease Study 2013,
one of which is shown below.
Our research questions were:
When you have a proactive agent that prompts users by asking questions about the task at hand,
how does the modality affect the discussion and sensemaking between the pair,
and how does the modality affect interactions with the system and the exploration of the data visualizations?
Here is an overview of what the conditions involved:
In both conditions the visualisations were presented in the same way on a large display on the wall.
How the conditions differed was in terms of how the system and the participants interacted with each other:
Participants could provide their requests through speech in the voice condition
and in the screen condition they did so through a screen interface.
The system, on the other hand, provided its outputs/prompts in the voice condition through synthesised speech
and in the screen condition as text messages next to the data visualizations.
What were our hypotheses?
Firstly, regarding the human-computer interactions, we hypothesized that the voice condition would encourage
more interaction with the system and more of the available data visualizations would be looked at by participants.
Secondly, regarding the human-human interactions, we hypothesized that the voice condition
would encourage more turn-taking and question-asking between participants.
The reason for focusing on question asking is because it is an important aspect of any exploratory task.
We used a Wizard of Oz setup for our study where the Wizard in an adjacent room observed the pairs and decided
based on two simple rules at which point to trigger which prompt, which he selected from the interface here.
The rules were:
Have the participants already discussed the trend or pattern which a prompt is referring to,
if not the prompt would be triggered - but only once there is a longer silence in the pairs' discussion.
So, what exactly happened in the conditions?
In the voice condition one of the participants speaks out a request to generate a visualization,
for example "Show women and men."
And then, after the pair have discussed the visualisation for a while,
the system may ask a question about an aspect in the visualization they have not yet considered or discovered.
And in the screen condition, on the other hand, the same interactions occur
but in this case just through the screen interface and as a message appearing on the large display.
Now let me show you a quick example of two participants requesting a visualization in the voice condition.
[Participant 1:] There doesn't seem to be any clear pattern.
[Participant 2:] Show developed and developing countries.
So, they request a new visualization here, which is then displayed.
[Participant 1:] Oh, look at that.
[Participant 2:] Yeah, OK, that's a lot more interesting.
Then, after having discussed the visualization for a while, the agent may have provided a prompt like this one,
System: Maybe you want to consider how the shape of the line compares to the one of adults?
Participant 1: Yeah, it's a lot kind of wobblier.
After that, the discussion of the agent's question then continued further
before the pair then switched to another visualization, which they discussed.
So, what were our main quantitative findings?
Both of our hypotheses could be confirmed.
In terms of our first hypothesis, there were more interactions with the system
and more data visualizations being looked at in the voice condition than in the screen condition.
And in terms of our second hypothesis -
participants asked each other significantly more exploratory questions in the voice condition;
more specifically, there was a question approximately every minute in the voice condition
and in the screen condition only about every two minutes.
There was also more turn-taking/speaker changes in the voice condition than in the screen condition,
indicating a more rapid back and forth between participants while they were discussing.
Furthermore, we found that in the voice condition interactions between participants were more balanced
in terms of how much they contributed to the conversation as well as how much they interacted with the system.
For more details on these and other findings please take a look at our paper.
To give you an idea of what the exploratory questions of participants looked like, here is an example.
We can see that participants first discuss how the time series graphs have changed
to then move on to speculate how they will evolve further.
Like will it keep rising slowly? Will it jump up? Will it stabilize? And so on.
This example shows nicely how participants are making sense of, question and hypothesize
about the data visualizations following a prompt by the conversational agent.
Now to our main qualitative findings. One key finding was that in the screen condition
there was a tendency to spend more time making sense of the agent's question and what exactly it means
before participants start discussing possible answers.
In the voice condition, on the other hand, participants more directly started discussing
possible answers to the agent's question, as if they are thinking out loud.
One main reason for this could be that if the system speaks with participants
as it was the case in the voice condition, they just feel more compelled to answer directly,
like they would do when another human conversation partner asks them a question.
In the screen condition, users may feel less compelled, since the system does not actively intervene in the conversation.
In both conditions, the proactive prompts seemed to get participants to discover things they would have otherwise missed.
So, to conclude
The voice modality seems particularly suitable for open-ended tasks,
enabling faster-flowing conversations and helping users generate more exploratory questions.
Independent of the modality probing human cognition through proactive conversational agent prompts seemed to be effective.
It got users to slow down, reflect on, and explore other aspects of the data visualizations,
which they would have otherwise missed.
What will the future hold for proactive agents and which tasks will they support us with? We are already excited to see!