Hello everyone I'm Toby Li from CMU It's my great pleasure to talk about our work in multi-modal
repairs of conversational breakdowns in task oriented dialogues This work is done with my
colleagues from CMU and UC San Diego There are different types of breakdowns in human
agent conversations In this paper we specifically focus on the natural language understanding errors
in task oriented frame based agents In frame based agents when the user says a command the
agent detects an intent which is a type of the task from the command such as search for flights each
intent requires a set of slots such as the airline the source the destination The agent will try to
extract the values for these slots from the user's command if any of the slot values is unknown
the agent will ask about them in the conversation This paper discusses three common types of errors
in this process: Fail to detect the intent Error in intent detection where a system detects the
wrong intent and error in slot extraction for example here it misrecognized the word
Singapore from Singapore Airlines as a departure city We use a grounding theory in communication
to understand human agent interaction where the user and the agent each has some knowledge beliefs
and assumptions The goal of the communication is to establish a common ground of mutual knowledge
beliefs and assumptions A key challenge is that the user needs to understand what the agent knows
through evidence For example if the agent's response to user command is Sorry i don't
understand It's not very useful and offers little insight into what the agent already knows
The second challenge is after the user understands what the agent knows there
might be a misunderstanding The user needs to find an effective strategy to fix the misunderstanding
Prior studies have shown that when the agent acts on misunderstanding and performs a wrong
action the user is not often not able to find an effective conversational repair strategy
because of their lack of lack of understanding of the source of the error One effective way
to address these two challenges is to have domain specific fallback conversations like these However
it requires manual design which is not scalable and therefore not widely used in current agents We
present SOVITE a new interface for conversational breakdown repair that uses a multi-modal approach
that leverages the GUI screenshots of existing apps It builds upon our open
source SUGILITE agent and can help users discover identify and repair conversational breakdowns
in various task domains without requiring manually created conversation flows for error handling
The first key step is to show the user the agent's state of understanding SOVITE shows screenshots of
a relevant app as that's what i would do if i were to use this app to perform the task
This approach provides good closeness of mapping to how users perform the tasks themselves allowing
them to easily discover errors if SOVITE shows a wrong screen or a wrong app that indicates an
intent detection error When the app is wrong the user can simply tell the agent which app to use
When the screen is wrong the user can navigate to the right screen
in the app and demonstrate the screen It's worth noting that SOVITE not only searches for intents
that use exactly the same apps and screens but also those that are semantically similar If the
highlight slot values are incorrect there is a slot value extraction error as you can see here if
the slot value is from a menu in the app the user can simply drag the highlight overlay to choose
the right slot value In text entry type slots the user can fix the value using gestures or speech
Finally if the slot values are mismatched the user can just drag to swap their values
There are three key points in our design rationale First we try to bridge the interdependent gulf of
evaluation and gulf of execution by making how users identify breakdowns consistent with how
users repair breakdowns Second we use the mutual disambiguation pattern in multi-modal interaction
by having the user direct manipulate GUIs to disambiguate their spoken natural language inputs
Third we use the apps and app screenshots as a medium to help grounding the user's unrestricted
natural language inputs to something that both the user and the system are familiar with
We conducted a usability evaluation where 10 users tried to use SOVITE in seven example scenarios the
study was conducted through zoom screen share due to the COVID 19 pandemic
The results are positive our participants successfully identified all the no error scenarios
and fixed 95 percent of error scenarios They also reported that SOVITE was helpful and easy to use
in a post-study questionnaire You can refer to the paper for details on the result
That brings us to the end of my talk today This work is a part of our SUGILITE project
on programmable intelligence agent you can find the other papers here and also on my website
By the way i'm on a job market right now so hit me up if you knows a position Thanks