Hi, my name is James Berzuk, I'm a computer science undergraduate student. I've been working
on a summer research project with Dr. James Young at the HRI Lab, at the University of Manitoba,
Canada. I have a background in linguistics, and was interested in how humans and robots
communicate. So we started by investigating the field and started to notice some patterns,
such as the prevalence of a lot of command-and-response-style interactions.
We were interested in these patterns, but we weren't sure, so we decided to do a formal
survey to find out more. We wanted to find out what systems in the field are actually doing.
So we collected 75 papers from major venues and contacted the authors where
necessary for more information. Thank you to everyone who assisted us in this process.
With so many systems collected, we found it difficult to organize and compare them, and
wanted a better way to describe the commonalities, so we formalized the patterns into a descriptive
framework. And then to illustrate the application of that framework, we selected systems from our
survey to use as case studies to demonstrate the framework's use with real systems. Altogether,
we offer a way to understand and compare systems as well as an overview of work in the field.
We developed the framework based on the survey in order to provide a vocabulary to understand
and compare disparate systems. The dimensions of the framework divide into two sections: design,
and implementation. And within design, we have discourse genres, interlocutor composition,
and dialogue structure. In addition to enabling the understanding of existing systems,
these dimensions offer an anchor for consideration when designing new ones.
So the first dimension is discourse genres, which is a tool inspired by work in linguistics,
and is used to classify the interaction according to its basic function.
In our survey we found four basic functions: Commanding, the issuing of commands or
offering of instructions, Questioning, asking questions or receiving answers in a back-and-forth
format, Informing, supplying information in a more unilateral way,
and Entertaining, where one party is providing entertainment to the other.
Note that each of these actions can be performed in either direction,
by the robot to the human or vice versa, and also that
one system may incorporate multiple genres at different points in its interaction. Discourse
genres provide a means of understanding and comparing the goals and purposes between systems.
Now the next is interlocutor composition, which is similar to the H-R ratios you may
be familiar with, but we've generalized the numbers into "one", "a few", and "a crowd".
For example, an interaction may be one robot to one human, one robot to a human crowd, or a few
robots to a few humans. These groupings highlight key distinctions in the designs, for example
when interacting with a few, there may still be possibility for one-to-one interactions,
while this is less feasible with a crowd. The last design aspect
is dialogue structure, which describes the high-level flow of the interaction.
For example, you might have a linear interaction, that is pre-scripted and follows a set path,
even if it has short deviations or loops. Or, you may have a branching interaction that,
while also scripted, is more responsive to user input, and has potential for
deeper and more permanent deviations. In contrast to these, some interactions are unstructured:
they don't really have a clear flow, instead they're made up of isolated utterances,
and they're more reactive to their immediate circumstances. These dialogue structures
describe the flexibility and structural complexity of an interaction. Now we don't
have enough time to get into everything, but we did the same level of analysis when considering
implementation details that impact dialogue, and we found three more dimensions:
whether the system operates independently of a human operator,
the range of modalities that are employed in the interaction,
and select algorithmic techniques that were relevant to the dialogue.
More details about all of these dimensions are available in the paper.
Altogether, the framework allows for the understanding and comparison of disparate systems,
as well as it offers a lens to consider new designs. And as an added benefit of the survey,
by applying our framework to the 75 systems we collected, we have produced an overview of what
kind of systems are in the field. We display the results of this in a diagram in the paper,
but I'll highlight a few interesting results here. Some results are unsurprising,
the vast majority of systems being one robot to one human for example. In comparison,
robot crowds were completely unattested, which might be expected given the cost.
We found that over half of systems were fully autonomous,
which demonstrates a substantial effort being made toward developing practical systems.
But some results are more expected, for example that the robot to human discourse genres were
much more common than the human-to-robot equivalents. This is intuitive in some cases,
it does seem more plausible for a robot-to-be entertaining a human than the reverse,
but it is striking that we found twice as many cases of robots commanding or questioning
people as we did of the reverse. And we also include further data in the appendices in order to
support future analysis, including the full classification of all 75 systems,
and a table of frequencies of each combination of the features in the framework.
So I've introduced the framework, and I've described how it can be used to understand
dialogue interactions, but I'd like to demonstrate how it applies to real systems, so we've selected
a few very different interactions from our survey and I'm going to present them as case studies.
So, for example, Morimoto et al. developed a robot for receiving and addressing customer complaints,
and, I will describe this using the framework. The system involved one robot addressing one human and
it followed a linear structure, first listening to, and then responding to the complaint.
The system went through three discourse genres: the customer informs the robot of the complaint,
the robot questions the customer for more information, and the robot informs
the customer of the company's response. The system had a wizard-of-oz operator,
both parties used voice, and the robot generated its using speech synthesis.
So in this way, the framework provides a concise vocabulary to summarize the robot.
For another example, Vilk & Fitter developed a robot that delivers a stand-up comedy routine.
And I'll again describe this using the framework. It delivers a solo routine
in front of a human crowd, it follows a mostly linear script, autonomously,
and uses synthesized voice and motions to entertain the audience.
The audience in return had limited ability to communicate back using non-speech sounds
like cheers and applause, which the robot analyzes for volume to adjust its delivery.
As a final example, Mizumaru et al. developed a security guard robot that tries to correct the
behaviour of passers by. It tries to stop them from using their phone while they're
walking. So in the language of the framework, the one robot approached lone pedestrians,
and in comparison to the previous interactions, this one was more unstructured: it involved
isolated utterances to each individual, rather than a broader conversation.
The robot commanded the pedestrians using synthesized voice as well as its motion
to make itself more convincing, and it required a human operator,
but its dialogue delivery itself was autonomous, so we classify it as a mixed implementation.
Now, in addition to providing the vocabulary to describe the systems, our framework offers insight
into the nature of the interactions. For example, with Morimoto et al.'s customer service robot,
framing it this way emphasizes the bidirectional nature.
It featured both robot-to-human and human-to-robot discourse genres,
which demonstrates that the robot guides the interaction,
but both parties are active participants. In comparison with Vilk & Fitter's comedian robot,
the discourse genres and the modalities show that the interaction is dominated by the robot.
And further, with Mizumaru et al.'s security guard robot, it had no human-to-robot modalities
whatsoever, which demonstrates a completely unilateral interaction.
This is one way that our framework highlights a key distinction between the systems.
So in conclusion, our survey offers a wide overview of the types of systems in the field,
the framework offers a vocabulary and lens to examine new and existing designs,
and our case studies demonstrate application to real world systems.
We hope this will prove useful in informing future robot dialogue designs. Thank you for listening.