Hello, my name is David Porfirio, and I’m going to present the work that I did with
my colleagues Allison Sauppe, Aws Albarghouthi, and Bilge Mutlu, which we call Transforming
Robot Rrograms Based on Social Context.
So what do I mean by robot program, and why would we want to transform them?
We refer to the robot program as a program that guides the robot in having a social interaction
with a user, represented here by a state machine.
We can see one of these programs in action here, in an example constructed specifically
for this presentation.
So, the first thing that the robot does is query the user for a response.
The user responds “events.”
If the robot doesn’t understand the user, it will prompt them again, as shown here.
If ignored, the robot will apologize for not being able to understand the user.
And finally, if ignored on-loop, the robot will apologize on-loop.
The robot program guiding this interaction is suboptimal for various reasons, some of
which are related to the flow of the interaction, or the structure of the program itself – the
robot repeats itself and doesn’t give the user a chance to respond.
So how could an interaction like this get to be so suboptimal?
It could just be that the designer is a non-expert in designing robot programs, or didn’t test
the program enough.
The interaction could just be buggy.
Our focus is on another category of causes, however, that are not necessarily related
to the design process taken by the designer when creating the interaction.
This category is related to the context within which the interaction is deployed.
In failing to design for the appropriate social context, the interaction might not account
for various environmental constraints, such as how noisy is the environment?
How well will the robot be able to hear and speak to users?
There are also various user constraints that the interaction may not take account for,
such as how comfortable is the user talking to a robot, orh ow patient are they, or happy
are they to be interacting with the robot?
And lastly, what are the preferences of the users interacting with the robot?
Do they want the robot to be funny, or get straight to the point?
Do they want it to talk less or more?
So back to our suboptimal program.
How might we transform its structure to make it fit the interaction context better all
the while making sure that the general flow of the interaction doesn’t break – namely,
the robot answers questions when asked?
So here’s our small program again.
We’ll say that this small program is part of a larger program.
The larger interaction program might have looked something like this, in which the states
correspond to robot behaviors, and the transitions are labelled with human actions that the robot
can recognize.
Recognizable human actions include hearing the human emit uninterpretable speech, no
speech at all, or a question.
In the ideal case, the robot would greet the user, query them to ask it a question, answer
their question, and then say goodbye.
In order to ensure that the core of the program – that the robot must answer the user – doesn’t
break, let’s say that we want the following property to hold: every path in the program
must at some point result in the robot answering a question.
There are multiple possible ways in which we can transform the interaction and have
this property hold, even just focusing on this problematic section of the program.
Perhaps the user finds the robot repeatedly apologizing to be annoying.
We can change the apologize behavior to a wait behavior, in which the robot simply waits
for user input rather than repeatedly prompting for it.
It might also be annoying for the robot to repeatedly query the user for a question if
it doesn’t understand what the user said.
And maybe the user would instead prefer to be apologized to if the robot doesn’t understand
them.
So, we can remove this looping behavior and set the robot to apologize if it didn't understand
the user.
And after apologizing, the robot can be set to wait for further input.
If the user says something the robot doesn’t recognize while waiting, the robot can apologize
for not understanding.
As a result, we’ve have modified the structure of the program according to our hypothetical
interaction context, while maintaining the correctness guarantee that the robot will
always answer questions posed by the user, assuming that the interaction does not loop
infinitely between wait and apologize.
Our research question revolves around this repair process.
Namely, how can robots automatically repair the flow of their interaction programs to
suit a specific interaction context, while adhering to baseline correctness guarantees?
Our technical solution to this question is three-fold.
First, we propose that the robot records its interaction with users, and collects input
about their experience.
Essentially, as each person interacts with the robot, their execution trace through the
robot program is recorded.
Users are then prompted to rate the quality of their execution traces.
Secondly, given a set of user-rated execution traces, we search for a new robot program
that accepts only a subset of these traces in order to maximize user experience.
Thirdly, we repeat this cycle of collecting user input and searching for new interactions.
To illustrate the approach, we’ll go through a toy example.
Suppose we have a robot that is programmed to provide information to passersby.
The robot is programmed to first greet users.
Depending on whether acknowledged by the user, the robot can provide information to the user,
or prompt the user to acknowledge it.
After providing information, continuous acknowledgement of the user will lead to more information.
Otherwise, the robot will say farewell.
And after prompting the user to acknowledge it, if the user does acknowledge it, the robot
will provide information.
Otherwise, if ignored, the robot will bid farewell and end the interaction prematurely.
In step one, we deploy this interaction, and record execution traces that have been rated
by users.
Here’s the interaction program.
This is an example execution trace, in which the robot greets the user and the user acknowledges
it.
The robot provides information and the user acknowledges it again.
The robot provides more information and the user ignores it, possibly meaning that the
user decided to leave the interaction.
And finally the robot bids farewell.
And finally, at the end of interacting with the robot, the user is prompted to score the
interaction based on their user experience, on a scale of bad, neutral, or good.
In this particular trace, suppose the user had a good experience, and thus assigned the
trace a plus one.
Now suppose a different user experiences a different execution trace, in which the robot
greets them, does not hear the user and assumes that the user ignored it, prompts the user
to acknowledge it, and then prematurely terminates the interaction when the user is not heard
a second time.
It is possible that the user was not actually ignoring the robot, and due to contextual
constraints such as the noise in the environment, the robot simply could not hear the user.
Therefore, this user may assign their execution trace a negative one, for a bad experience.
The net score of the interaction is the sum of the scores of the execution traces, and
is zero in this case because we have a positive trace, and a negative trace that are both
accepted by the current interaction.
So our job is to search for an edited, or repaired, interaction that maximizes its score
by accepting only a subset of traces.
Shown on the right, we can see a third execution trace, in which the user is always ignoring
the robot.
In contrast to the second trace, after greeting and prompting the user for acknowledgement,
the robot will provide information even without being acknowledged.
Finally, the robot will way farewell.
And, we’ll say this trace is scored positively.
This new trace is not accepted by the robot program, though.
The robot may have collected it in previously-deployed versions of the program, or by stochastically
exploring novel execution traces with a random subset of users.
Now, I'll talk about the specifics of our searching approach.
Consider the original interaction again, which accepts both the first and second traces.
In making edits to the structure of the program, we can change which traces are accepted by
the program.
Here is one possible edit that only accepts the first trace.
And another edit that only accepts the first trace.
And another.
And a fourth edit that accepts both the first and third traces.
We can use correctness properties to narrow this search space quite a bit.
Suppose we wanted to guarantee that no matter what, the robot will only ever prompt a particular
user once.
In temporal logic, this translates to ”whenever a prompt occurs, there can be no future state
in the interaction in which prompt occurs again.”
So with this property, we can eliminate two of our candidate interactions.
Here is the original interaction again.
And here is one edit that satisfies our correctness property, and only accepts the first trace,
making its net score plus one.
And here is the other edit that satisfies our correctness property, and accepts both
the first and third traces, but not the second, making its net score plus two.
Thus, we return this robot program, since it satisfies our correctness property and
has the highest net score.
In the third step of our approach, we cycle on collecting rated execution traces and modifying
the interaction based on all execution traces collected thus far.
Now onto our evaluation of our repair approach.
For our evaluation, we considered a robot whose purpose it is to assist visitors at
the front desk of a university campus building.
We also created an initial robot program based on this scenario.
Details about the initial program can be found in our paper.
Our evaluation included two parts – first, a validation of our approach under highly
controlled conditions on Amazon Mechanical Turk, in which we utilized what we call “interaction
vignettes” to simulate an interaction experience through text, and second, we did a field experiment.
Our validation study gave us promising results regarding the effectiveness of our approach,
but for brevity I won’t discuss it in this presentation.
I would encourage viewers to check out the paper for a detailed description of the validation
study.
For our field experiment, we sought to evaluate our approach under a realistic setting, and
deployed a robot with the initial interaction program at the welcome desk of a campus building.
Here is a top-down view of the setting of our study.
Two robots were placed on either side of the desk.
One robot was designated as an “adapted” robot, in which its interaction would be modified
based on user input over time.
The other robot was designated as a "nonadapted" robot, in which its program would stay the
same throughout the study.
Over the course of twelve days, we collected execution traces from users of each robot.
In addition to collecting scores, we measured user experience and users’ perceived contextual
fit of the interaction.
In the first and second days, two traces were provided by users who consented to interact
with the nonadapted robot.
Three traces were provided by users who used the adapted robot.
We used all of these traces to modify the interaction on the adapted robot.
The modified program was deployed and used to collect an additional four traces – three
of which were collected on the nonadapted robot, while one was collected on the adapted
robot.
We used all traces collected thus far to modify the interaction on the adapted robot again.
We repeated this process on the fourth day, in which three traces were collected on each
robot.
And on the fifth and sixth days, in which five traces were collected on the nonadapted
robot and four traces collected on the adapted robot.
And finally, from days seven through twelve, the modified program was deployed and used
to collect a final fifteen traces – six from the nonadapted robot and nine from the
adapted robot.
About a quarter of all traces collected were randomly generated in order to provide variance
to the set of traces used to transform the interaction.
Each user who interacted with a robot was presented with a questionnaire that collected
the user’s interaction score of their execution trace, and measured their user experience
and perceived contextual fit of the interaction.
Here are the results for the field experiment during the fifth and final cycle, in which
no further changes were made to the interaction.
We include data from only thirteen of the fifteen participants, since the other two
participants experienced randomly generated traces intended to provide variance to the
set of execution traces collected during the study.
Our results show an increase in user scores within the adapted interaction.
Since users were prompted to rate their interaction experience when providing their scores, we
treat this result as a weaker indication of our approach improving user experience over
time.
However, using interaction scores is a very coarse way to measure user experience, due
to the single item used to measure scores, the small size of the scale used for measuring
them, and the low number of participants in the field experiment.
To conclude my talk, I would like to emphasize the need for collecting more data over a longer
field experiment in order to further validate our approach as effective in the wild.
With the limited amount of data that we collected over a fairly short time frame, it is difficult
to gauge when and how our approach to transforming robot programs will converge on an optimal
program.
Additionally, our current approach of assigning single scores to whole execution traces fails
to utilize all the available data that a single trace might hold.
Future work should investigate ways in which parts of an execution trace can be assigned
a rating Thank you for your time, and I'll take any
questions that anyone has over email.
And I encourage you to check out our source code at the link shown on the screen.