Hi, everyone. This is Minha Lee,
and I'm first year Master student in ID KAIST.
This work is about exploring better responses of conversation agents with insufficient information.
Despite the increasing presence of conversational agents (CAs) in our daily lives,
the lack of information and technology behind them prevents CAs from answering many questions.
One of the most typical problems facing conversational user interfaces today
is that they often disappoint people by giving the same answer
such as “I don’t know”.
In this work, we focused on situations in which CAs do not provide
a proper answer because of this lack of information.
And we aimed to present a better alternative answer
than “I don’t know” in these situations.
For that, we attempted to understand users’ perceptions
and feelings by creating different CAs.
We set up four agents that give simple answers in different ways,
and followed the Wizard of Oz method using text-to-speech.
And we set 2 questions, which include objective and subjective facts.
Also, We set four variations of CA responses:
Answer 1 (A1) was “I don’t know.”
Answer 2 (A2) was unfounded inference.
Answer 3 (A3) was inference based on a general fact.
Answer 4 (A4) was inference based on some reasonable related information.
We planned to observe the usability perspective through SUS survey
and user experience perspective through a debriefing interview.
These two tables show participants’ tendency
to evaluate the usability of each agent with different response methods.
From the usability aspect,
inference based on a reasonable related information
is significantly more usable than other answers.
And when they heard the answer, “I don’t know,”
participants were embarrassed that it was all the CA could provide.
Some said they were annoyed,
and some felt the CAs were irresponsive and insincere.
Overall, participants thought they did not get what they expected from the CAs
because they did not look for other information
and only answered, “I don’t know.”
In the A2 case, which provided unfounded inference,
most of participants said CAs have no responsibility,
and they preferred a clear answer such as A1,
to a long sentence with meaningless guesses, such as A2 and A3.
In contrast to its usability,
when asked about the aspect of closeness,
some participants said they felt the A3 answer was the best.
They felt that this answer was generally like a friend’s answer,
and it was not only about the general facts,
but also about their personal preference.
A4 got significantly high score in SUS survey.
and that means, presenting appropriately related information
may be a good alternative to enhancing the usability of a CA.
And they said,
CA tried to provide a meaningful response to their question,
and they tended not to pay much attention to its accuracy
even if the guess might have been wrong.
Also, what ‘s interesting is
some participants did not realize that CA did not give them
a complete answer, even though it started with “I don’t know.”
As a result, when a CA fails to respond fully to a user’s question,
a user will evaluate the responsibility, sincerity,
and level of cooperation the CA tried to provide.
We also learned that providing reliable information
could significantly improve usability and satisfaction
compared to CAs that answer with no side information
and only change a kind of speech style.
CAs not only need to look for information to answer questions directly,
but also need to build a system to rank information that is relevant
and discover other information a user
who asks a question might require.
Thank you for listening and your attention.