Hello I'm Sruthi and I'm presenting our DIS 2022 paper on "Addressing hiccups in conversations
with recommender systems," on behalf of my co-authors Fabien, Anto, and Jean-Michel from
NAVER LABS Europe in France and Minsuk from NAVER AI LAB in South Korea.
Let me start with the definition.
We define hiccups in conversations with recommender systems as minor or temporary problems that
drive the user to quit the conversation.
Conversation ends after a hiccup occurs, because the user does not wish to go through with
the conversation after this hiccup.
So we wanted to identify and address these conversation killers.
For this we conducted two user studies.
In our first user study we observed hiccups when users interact with Google Assistant
requesting for restaurant recommendations.
Then we built and prototyped a dialogue model to mitigate the observed hiccups which we
then tested with the Wizard of Oz experience prototyping in Study 2
to understand the state of persisting hiccups.
Finally we derive implications for addressing hiccups in conversational recommender systems.
To dive into the details, our first research question was to identify hiccups when users
are seeking recommendations with a current conversational agent like Alexa or Google
Assistant.
So we tasked nine participants to try "Hey Google, recommend me or restaurant" to a mobile
assistant and interviewed them after this task.
Our analysis identified two main reasons for hiccups.
The first one was because current users were clinging on to the graphical user interfaces.
also known as GUIs, which they have been using for a few decades in web search engines.
As you can see in the image here, P2 was asking the system to "go back" to the previous results,
which was not supported and other participants also asked to save the result or see their
history, also our participants' spoken dialogue was similar to a (*web) query.
As you can see here P3 succeeded after three tries, by finally asking "show me a good restaurant
near me right now," All the correct keywords.
In addition to expecting the conversational agent to sustain the features of GUIs.
Surprisingly, our second main reason for hiccups was because participants expected the Agent
to engage explain and help like a friend, especially after providing results they expected
the agent to say what to do next very proactively.
And they also needed explanations, for example when recommending a restaurant they expected
the agent to say "I'm recommending this because they have incredible sauces."
So overall, we observed hiccups pretty much everywhere participants struggled to start
a conversation and to explore further they also needed guidance after results and errors
So we used the two main reasons for a hiccups which is users' expectation of GUIs features
and human like characteristics to design a simple dialogue model
mimicking both humans and GUIs.
This is a generalised dialogue model which can be used for any
conversational recommender system.
Our dialogue model employed human mimicry to guide and explain to the users during the
conversation, and it also had GUI mimicry, for example, to filter, save, or go back and
forward in the results.
to deploy this dialogue of model we built a Wizard of Oz prototype where the user interacted
with a mobile assistant using spoken dialogue and touch.
And the wizard used synthetic voice and visual stimuli to interact with the user, while managing
the dialogue in the back end using the dialogue model we designed.
So, going back to the overview we just saw the hiccups we identified in our first study
and the dialogue model we built to mitigate them.
Now we will see the state of hiccups after using this dialogue model in our second study,
which is the wizard of Oz experience prototyping and the implications we designed for addressing
hiccups in conversation recommender systems.
So after we built the prototype our research question was to understand the state of hiccups
which persisted in the conversation when the conversation recommender system mimicked GUIs
and human characteristics of proactivity and explanations.
So we ran a Wizard of Oz experience prototyping with 11 participants who participated in our
experiment remotely via Zoom, we interviewed them after the interaction and we also coded
and visualised their interaction with the wizard.
Participants use both visual and voice interactions.
They alternated pretty comfortably between them and they were quick to adopt GUI mimicry,
for example, they asked the wizard to filter or sort their results using spoken dialogue.
And they also continued the conversation after receiving results, after errors, and pauses.
So overall participants benefited by adopting GUI mimicry and they cooperated with the wizard
to overcome errors
And commented positively regarding the explanations provided by the Wizard.
However, it was very difficult for the Wizard to understand whether a participant is stuck
or whether they are thinking when they are pausing.
And as the conversations became longer, it was very difficult to navigate back and forth
in the conversation.
So overall, we observed improvements in the hiccups observed in Study 1 but also added
two more hiccups in Study 2.
Based on our studies we derived four implications for addressing hiccups in conversational recommender
systems, the first one is to equip the user with features to control and explore the conversation.
The agent also has to be proactive in helping the user to discover new features and the
possibilities to interact with the system.
The second one is to design system actions post recommendations, for example, to proactively
help the user to explore further or to elicit feedback from them.
And the third is to design interactions for users, after errors, for example, to recover
from a miss-heard query or to understand the scope of the system.
And the last one is to create references for long conversations, so that both the system
and the user can use these reference tags to easily access different parts of the conversation.
So that is addressing hiccups in conversations with recommender systems.
Thanks for listening and please feel free to reach out to us to discuss.