Hello everyone, my name is Emmanuel Senft.
I am a research associate at the University of Wisconsin-Madison,
and today I will present to you Situated Live Programming for Human-Robot collaboration,
a method that allows end-users to create interactive robot behaviors for collaboration with humans.
Our work is contextualized in light manufacturing
and aims at providing new tools for end-users such as factory workers
to design behaviors for a robotic assistant.
Consider the following assembly task.
The task requires fetching two pieces from outside the human workspace,
joining the pieces with a bolt and nut,
and placing the assembled piece in a storage area outside of the human's reach.
To create a collaborative robot program for this task,
the end-user needs to allocate actions between the robot and the human,
for example creating a program for the robot to fetch the pieces and also to store them.
Specifying robot behaviors for interaction involves two challenges.
First, specifying what the robot should do: for example which piece to move and where.
This specification needs to be general enough to allow the action to be repeated and robust to changes in the workspace.
Second, the user needs to specify when action should take place, for example,
when the robot should wait or after which human action the robot should act.
In this work, we present Situated Live Programming, a method addressing these two challenges.
Our method uses Trigger-Action Programming to create reactive robot programs
and we designed an interface which allows users to easily refer to the environment
and layer multiple trigger-action pairs to progressively create complex behaviors.
Trigger-action programming in short TAP is a well known end-user programming paradigm
typically used in home automation and app synchronization.
Traditionally, TAP defines a trigger, for example the temperature being below 65 degrees,
and then associates an action to this trigger, for example start heating.
The system will then monitor sensors and start the action when the trigger is true.
To make TAP applicable to human-robot collaboration we extended it in two ways.
First, we situate TAP in the state of the environment.
We use a tablet to display the video from a robot-mounted camera,
and the user can annotate and refer to this live feed to create trigger-action programs.
Second, we propose that robot behaviors can be decomposed into a set of atomic trigger-action rules
whose interactions define an interactive behavior.
This construction allows the robot program to be created iteratively,
adding and refining rules when needed.
Users can use live programming to edit the robot behavior at runtime,
allowing them to define elements of behaviors only when required.
As an example, we can use situated live programming to program the task mentioned earlier.
First, the user labels relevant regions of the environment.
The user can then create a trigger-action pair specifying when the robot should perform actions.
Here, the trigger specifies that when there is no black piece in the yellow zone representing the human workspace,
the robot should stack a white piece in the green zone onto a black piece in the same zone
and then move the black piece to the yellow zone.
Once this pair is created, any time the condition is true, the robot will execute the associated actions
and then reset to a high vantage point to determine whether any additional actions are ready to be executed.
Once the robot has brought over and stacked the pieces,
the human can perform the manual assembly.
When the human removes the piece from the workspace, the robot will again execute the
trigger to fetch another set of pieces.
When the user places the assembled piece in the workspace, the system recognizes that
the state of the object has changed to assembled.
The user can program a second trigger-action pair which detects an assembled object in
the human workspace and moves the object to the final assembly region within the blue grid.
With these two trigger-action pairs, the user can continue to build more assemblies:
with the robot fetching pieces and storing completed assemblies in the grid.
If any adjustments need to be made to the program or if new steps are needed,
the user can easily add or remove existing trigger action-pairs in the robot program.
Our implemented system focuses on pick-and-place behaviors.
It supports the following triggers: the presence of an object in a zone,
the absence of an object from a zone, and the state of objects.
The main action is a move action that can be parameterized in multiple ways:
specifying what type of objects should be moved, from which zone, to which zone and with which placement pattern,
in a container, in the middle of the zone, or in a grid.
Users could combine multiple conditions in a trigger and specify multiple actions,
creating a wide range of possible behaviors.
As trigger evaluations are periodic, in some cases, two triggers can be true at the same time,
resulting in a priority conflict.
A pop-up appears on the interface and asks the user to arbitrate and decide whether this
arbitration should be remembered in future conflicts.
We performed an exploratory study of our system where ten participants completed four tasks
of increasing complexity: a sorting task, a kitting task, and an assembly task which
had to be completed twice: once with live programming and once without live programming.
In the sorting task, participants programmed the robot to move 3 different types of hardware
into three corresponding bins.
One way to program this task was to have a trigger-action pair for each class of object
to move the object to the corresponding bin if it was detected on the robot table.
In the kitting task, participants programmed the robot to fetch three pieces of hardware
needed for each kit.
The human was then responsible for putting the final kit together.
The kits contained 1 bolt, 2 connectors, and 1 fastener,
and participants had to construct 3 kits.
One way to program this task was to create a button which had attached actions to grab
each object in the kit and place them in the human workspace.
The assembly task was the task shown in the earlier example and consisted of fetching,
assembling, and storing 3 total assemblies.
This task required at least two trigger-action pairs: one for bringing the parts from the
robot table, and one to move the assemblies.
Participants were generally able to complete all of the tasks using our method.
We found that participants used a variety of strategies to complete these tasks.
For example, some participants performed assembly tasks in parallel
as the robot moved or stacked objects.
Other participants executed the task sequentially, where they would only assemble after the robot
had completed its tasks.
Multiple participants reported that the live, iterative programming
allowed them to approach the task from an exploratory stance
when they were unsure what specific actions the task would require.
Overall, the interface was rated as having a good usability
and as being easy to learn and use.
In summary, we presented and evaluated a method to allow end-users with little programming
experience to program collaborative robots.
In the future, such methods could enable novice end-users, such as factory workers, to create
robot programs and help promote the adoption of human-robot collaboration
across a broader range of industries and tasks.
Thank you very much for your attention, and I would like to extend my thanks to my colleagues
and to NASA for supporting this work.