I'm David Porfirio and today I'm and to talk about the work
that I did with my colleagues Evan Fisher, Allison Sauppe,
Aws Albarghouthi, and Bilge Mutlu which
we titled Bodystorming human robot interactions.
I'd like to begin by giving an overview
of the culmination of our work, or the end product,
if you will.
So we've created a design tool called
Synthe that records designers as they act out interactions
between a human and a robot and results
in a full interaction program that
can be run on a real robot.
In discussing this project in more detail,
I'll start by talking a bit about the background
of the problem that we're trying to solve
and our research question.
Followed by talking in more depth about the design
interface that we've created, which again is called Synthe.
And then discussing what's under the hood of the interface
or how specifically we go from designer demonstrations
to a fully articulated executable program.
And finally, I'll end the talk with a discussion
about our evaluation of Synthe.
So I will begin with the background.
So what is an interaction program?
For our purposes it is a program that tells a robot
how to interact with a human.
So for instance, we can visualize an example
of such a program, as a state machine,
in which the robot might wait for the human
to acknowledge it.
Followed by greeting the human, making a, delivery
and then saying goodbye.
And then answering any questions that the human
has during the process.
Now this program must be implemented
on the robot in some way and regardless
of how implementation occurs this process
traditionally occurs through some programmer, or designer,
coding in front of a screen.
Using this method the designer will face
a number of challenges though.
The designer must come up with ideas
about how the robot should behave all the while being
focused on how to implement these behaviors
onto an actual robot.
For instance the designer has to imagine what social cues would
be best to use on the robot, which
could include verbal cues and gestural cues,
and they must figure out how the robot should repair breakdowns
in the interaction.
Such as if the end user becomes frustrated with the robot.
All of this must occur while the designer ensures
that the program is implemented correctly
and while being concerned with the syntax of whatever
programming language they are using.
In the extreme case, the designer
might not even know how to code.
So these implementation challenges
can be a distraction from pure ideation which, in other words,
is focusing purely on what the robot should do.
Proper ideation uninhibited by implementation
is our goal here.
And I want to talk a little bit more about ideation.
There exists many methods to ideate designs
for both experts and novices that all get at the same idea.
We can know more than we can tell.
Or, in our case, we have this intuitive or tacit knowledge
of social interactions that is difficult to express in words
and difficult to express in code.
One such design process that embodies this idea
is called bodystorming.
And bodystorming traditionally involves role playing
which, in our case, could mean that one designer acts
as a robot and another designer acts as a human and user.
Bodystorming also involves environmental similarity,
which refers to brainstorming in an environment that
is similar to the environment being designed for
and is often achieved through the use of props.
With bodystorming we can unlock our intuitive knowledge
of social interactions to form ideas
about how we should program our robots uninhibited
by implementation.
However, we still need to reconcile ideation
with implementation.
So our specific research question
is, how can we enable design teams
to leverage their intuitive knowledge
of social interactions to prototype human robot
interaction programs.
And our proposed solution is twofold.
Firstly, we propose to capture a designer intent
as design team's bodystorm a human robot interaction
program.
And secondly, using observations of designer demonstrations,
we propose to synthesize a full human robot interaction
program.
What this solution will allow us to achieve
is enhancing designer ideation while removing the distractions
that come from implementation.
So now I'll talk in more depth about the design interface
that we created, which again is called Synthe,
as a way to achieve this solution.
This is our tool, Synthe.
At its core, Synthe directly enables designers
to perform three tasks.
The first, acting out interactions while Synthe
records them.
Synth will then generalize the recordings
to a full blown interaction program.
The second, is editing the recordings
using a graphical interface and finally the third,
is simulating the full blown interaction
program on a real robot.
Let's first go over the acting components.
So in this video two designers demonstrate, or act out,
examples of how an interaction should play out
between a human and a robot.
Synthe is recording them in the background using microphones
to capture of speech and arm bands
to capture designer gestures.
The speech and the gesture streams
are labeled and overlaid with each other
in order to create what we call a trace, which
is short for execution trace.
A trace simply contains the sequence
of events within a demonstration the designers have
provided to Synthe.
The states are annotated with the robot actor speech
and also the gestures that the robot actor performed.
The arrows are annotated with the end user actor speech.
After performing a demonstration,
the trace will be presented to designers
to either accept or reject.
And the trace will be used as input
into our synthesis algorithm, which
generalizes all traces previously seen
into a full interaction program.
And we represent these programs as finite automata.
Zooming into part of the program on the screen,
we can see how this automata works.
The automata is actually considered
to be what is called a Mealy automata, in which branching
only occurs at unmarked states which
are represented by circles.
These circles represent the decision
nodes that the robot must take.
Arrows leading away from the decision nodes
represent actions that the end user takes.
And finally boxes represent actions
performed by the robot in response
to an action taken by the user.
So back to our three functions.
I'm going to now briefly give some details about the editing
component of Synthe.
Again, after performing a demonstration,
the trace will be presented to designers
to either accept or reject.
Designers can edit the speech associated with this trace,
also they can edit the speech categories, and the gestures
that have been captured through their demonstration.
And once designers are happy with the set of traces
that they've provided to Synthe, they can activate their program
and simulate it on a real robot.
So those are the three basic functions that Synthe directly
allows designers to perform.
However, recall that brainstorming is also
a large part of bodystorming.
So the idea is that Synthe should also
encourage designers to brainstorm
throughout the process of their design session.
All right, so we've thus far discussed the interface
presented to designers and now I'm
going to talk about our program synthesis algorithm
or in other words, what is under the hood of our interface.
Our technical challenge is as follows.
Given multiple input demonstrations
from designers, that vary in length and content,
how can we synthesize or construct
from the ground up a full automata such
that all input traces are accepted or included
within the automata?
So we setup synthesis as a satisfiability modulo theories,
or SMT problem in first order logic.
I'm not going to go into the math.
I encourage you to read the full paper for that.
But as I just said we take a set of traces
and we generalize them to a full automata
by maximizing the acceptance of each trace within the automata.
In effect, the problem we're trying to solve
is an optimization problem expressed as a max SMT problem.
So we formulate this problem within three constraints,
three main constraints.
The first is that for a trace with the sequence
of human actions there must be a corresponding path
within the automata with the same sequence of human actions.
The second constraint is that for a sequence of robot actions
within the same trace the same path within the full automata
must also contain that sequence of human actions,
sorry, robot actions.
The main reason for splitting up these two constraints
is that it allows us to treat circular decision nodes
and rectangular robot actions differently.
If a sequence of human actions loops back to the same decision
node, the sequence of robot actions
does not necessarily need to loop back
to the same robot action.
In effect, splitting the two constraints
makes our synthesis problem easier.
Secondarily, splitting the constraints
allows us to handle error differently
within each constraint.
In our approach violating constraint
one is not allowed at all, while violating constraint two
is allowed with some error.
And finally, the third constraint
is that we want there to be only one initial state
and we want all traces to lead to the same sink state.
This sink constraint ensures that the end of a trace
corresponds to the end of a program.
So all of this is performed in real time.
The video is not playing.
So all of this is performed in real time
and what you're seeing here is a sped up video of Synthe
in which each new interaction is computed seconds
after designers stop recording new traces
and after they stop editing existing traces.
All right so finally I'm going to discuss
how we evaluated Synthe.
So our goal in our evaluation was
to investigate designer experience using our approach
to bodystorming human robot interactions.
And secondarily, our goal was to determine the effectiveness
of applying bodystorming to human robot interaction design.
Our study design is such that we recruited 12 participant pairs
and trained them on how to use Synthe.
Each pair was instructed to bodystorm either a delivery,
or an information desk interaction,
and they bodystorm this interaction
as though it's with a robot.
So one designer would role play as the robot,
and another designer would role play as the human.
Each pair underwent a 20 to 25 minute design session
and they were allowed to switch off who played the robot
and who played the human within this design session.
And after the design session concluded,
pairs underwent a retrospective think aloud
in which designers watched the video of themselves
bodystorming and retrospectively commented
on their thought processes.
So our results come in two parts.
First, we present the results of the think
aloud as an analysis of designer's
subjective experience.
Second, we present our analysis on designer workflows,
in which we look at the order of designer activities
within their design sessions.
So we ask questions like how often did they
act after brainstorming or how often did they edit
after simulating, et cetera.
So I'll talk about the results from the think aloud first,
which exists within five themes, but for brevity I'm
only going to talk about two of these themes.
In the first theme we found that designers often
brought external ideas to their design sessions.
For instance, many participants based their designs
off of preconceived expectations of how
a robot would perform in the scenario being designed for.
The bodystorming process implemented through Synthe,
however, provided ample feedback to designers on their design.
For instance designers often receive feedback
from their partner or from simulating interactions
on the robot.
In another theme we found that designers had mixed experiences
with role playing.
Some designers thought that acting was helpful
and that it felt natural too.
Some designers also thought that it added naturalness
to their designs.
Other designers felt that the process of acting
was artificial and felt unnatural,
but designers generally did not think that acting added
artificiality to their designs.
So now to discuss the results from looking at the design
workflows that designers adhered to,
we were interested in looking at how much designers
incorporated the bodystorming workflow within their design
sessions.
In doing so we were interested in how much people
used acting or role playing.
We were also interested in how much did they brainstorm.
Additionally simulation is somewhat
interesting in that it kind of gets
at the idea of environmental similarity
because by simulating, designers are effectively
using a problem.
So to compute these results we coded participants
design sessions for brainstorming, editing, acting,
and simulating.
The figures shown on the screen effectively
summarizes these results.
Looking at the graph on the left,
we can see that over the course of the design session, when
participants were actively designing, they spent about 50%
of their time brainstorming, 30% of the time editing
traces, 10% of the time acting out demonstrations,
and about 10% of the time simulating their interactions.
This is interesting because it shows that a major component
of bodystorming-- acting--
does not constitute a large part of designers' time.
We did see however that acting is still a significant part
of designers' workflows.
The graph on the right depicts which
design activities, brainstorming, acting, editing,
and simulating, were on average most likely to occur
after other design activities.
So for instance, the blue arrow from s
to b indicates that on average, after simulating
an interaction, designers went back
to brainstorming 6.5% of the time.
So starting from brainstorming, we
can see that designers most frequently
began an acting session.
And then after acting they most frequently went back
to brainstorming.
Therefore although acting does not
take up a large portion of total design time,
it is done quite frequently.
All right so to end my talk I'm going
to go over a few important takeaways.
First of all our work is the first to apply
program synthesis to the bodystorming design method.
Our work culminates on a tool that requires no programming
experience whatsoever to create full blown programs that
can run on a real robot.
All the while encouraging designers
to brainstorm in novel ways.
Synth is limited, however, in that
only high-level general programs can be created.
Low-level editing of these programs is limited.
Additionally, usage of Synthe requires
that pre-existing natural language
models be used for the classification of designer
speech.
Lastly we believe that there is much left
to explore in creating more detailed programs.
For instance, future work involves
investigating the incorporation of real time
components such as timing between responses.
Additionally, design methods other than bodystorming
can be explored in this space as well.
We would like to explore some design methods that incorporate
more improvisational methods.
And with that, thank you for your time.
I'll take any questions you have now
and that is the open source GitHub repository for Synthe
if you're interested.
[APPLAUSE]
All right, so we have a couple of minutes for questions.
Please raise your hand and we'll get you a microphone.
Thanks.
I wonder if you could talk about what kind of conditionals,
branches, your system supports and how it knows which
way to go down the branches.
Sure, yeah, so transitions in the state
machines are labeled with human speech responses.
And so at the end of the robot performing
an action, or a behavior, it will
wait for a speech response from the human
and that is how it decides to take
a path through the interaction.
The interaction is completely deterministic as well.
All right, while we're getting the next bigger
set up I do have another question.
So it seems like this is great for interactions like the ones
you showed where I'm physically in the same space as the robot,
but often I might want to automate tasks that
are dangerous for a person.
So I might be at a distance working with the robot.
Can you say a little bit more about how
that might make this more complicated
in terms of the design process or how realistic it
is to interact and program this way?
Well, I think that as long as we can
capture interactions in which safety is critical like that.
I don't think it would affect the approach
that Synthe takes to synthesizing
these interactions.
I would say that for safety-critical interactions
like that, I think it would be more important to ensure
that the designs that are created
adhere to certain model-checking properties.
In situations where the human might
be in a dangerous situation like that,
we probably want to make sure with 100%
certainty that the interactions that Synthe produces
would adhere to certain guarantees.
That would be the main change I think.
All right, thanks.