Situated Live Programming is a method that allows end-users to create robot behaviors for collaboration.
The task requires fetching two pieces from outside the human workspace,
joining the pieces with a nut and bolt,
and placing the assembled piece in a storage area outside of the human's reach.
To create a collaborative robot program for this task,
the end-user needs to allocate actions between the robot and the human
and specify how and when robot actions should take place,
for example how to facilitate a handover of the assembled piece to the robot for storage.
In this work, we present a method for programming human-robot collaboration.
Our method allows users to create reactive robot programs by iteratively specifying trigger-action pairs
based on a live view of the environment.
We use a tablet to display the video from a robot-mounted camera,
and the user can annotate and refer to this live feed to
create trigger-action programs defining what the robot should do and when.
This interface allows users to layer multiple trigger-action pairs
to progressively create complex behaviors.
We believe that the resulting situated live programming tool
enables end-users without significant programming expertise
to program collaborative behaviors.
As an example, we can use situated live programming to program the task mentioned earlier.
First, the user labels relevant regions of the environment,
for example the location of unassembled pieces and the human workspace.
The user can then create a trigger-action pair specifying what the robot should do and when.
In this example, the trigger specifies that when there is no black piece in the yellow zone representing the human workspace,
the robot should stack a white piece in the green zone onto a black piece in the same zone.
After this, a second action specifies that the robot should move the black piece to the yellow zone.
Once this pair is created, any time the condition is true, the robot will execute the associated action
and then reset to a high vantage point to determine whether any additional actions are ready to be executed.
Once the robot has brought over and stacked the pieces, the human can perform the manual assembly.
When the human removes the piece from the workspace,
the robot will again execute the trigger to fetch another set of pieces.
When the user places the assembled piece in the workspace,
the system recognizes that the state of the object has changed to assembled.
The user can program a second trigger-action pair which detects an assembled object in the
human workspace and moves the object to the final assembly region within the blue grid.
With these two trigger-action pairs, the user can continue to build more assemblies:
with the robot fetching pieces and storing completed assemblies outside of the workspace.
If any adjustments need to be made to the program or if new steps are needed,
the user can easily add or remove existing trigger action-pairs in the robot program.
We performed an exploratory study of our system where ten participants completed four
tasks of increasing complexity: a sorting task, a kitting task, and an assembly task
which had to be completed twice: once with live programming and once without live programming.
In the sorting task, participants programmed the robot to move 3 different types of hardware
into three corresponding bins.
One way to program this task was to have a trigger-action pair for each class of object
to move the object to the corresponding bin if it was detected on the robot table.
In the kitting task, participants programmed the robot to fetch three pieces of hardware needed for each kit.
The human was then responsible for putting the final kit together.
The kits contained 1 bolt, 2 connectors, and 1 fastener, and participants had to construct 3 kits.
One way to program this task was to create a button which had attached actions to grab each object in the kit
and place them in the human workspace.
The assembly task was the task shown in the earlier example
and consisted of fetching, assembling, and storing 3 total assemblies.
This task required at least two trigger-action pairs:
one for bringing the parts from the robot table, and one to move the assemblies.
whether it was empty, full, or assembled in defining the triggers.
Participants were generally able to complete all of the tasks using our method.
We found that participants used a variety of strategies to complete these tasks.
For example, some participants performed assembly tasks in parallel as the robot moved or stacked objects.
Other participants executed the task sequentially, where they would only
assemble after the robot had completed its tasks.
Multiple participants reported that the live, iterative programming
allowed them to approach the task from an exploratory stance
when they were unsure what specific actions the task would require.
Overall, the interface was rated as having a good usability and as being easy to learn and use.
In summary, we presented and evaluated a method to allow end-users with little programming
experience to program collaborative robots.
In the future, such methods could enable novice end-users,
such as factory workers, to create robot programs
and help promote the adoption of human-robot collaboration across a broader range of industries and tasks.