Hello, I am Andrew Schoen.
Along with my fellow co-authors Nathan White, Curt Henrichs, Amanda Siebert-Evenstone, David
Shaffer, and Bilge Mutlu, I am pleased to present our work on CoFrame: A System for
Training Novice Cobot Programmers, a joint training/programming interface for teaching
programmers how to construct collaborative robot, or cobot, programs in manufacturing
contexts.
This work aims to address a growing skills gap in this domain that currently prevents
cobots from being effectively and widely used.
More specifically, research has shown that prospective cobot programmers frequently lack
the skill and knowledge to adequately utilize these technologies and manage their integration
with human workers, since these interaction level and robotics-focused skills aren’t
taught as effectively as they could be.
Simultaneously, research has highlighted the effective nature of apprenticeships and hands-on
training where prospective cobot programmers may learn these concepts from a more experienced
expert.
In this vein, we set out to create a system that attempts to provide feedback to novice
cobot programmers in line with what an expert mentor might provide.
We do this by starting with the results of an analysis of experts by Siebert-Evenstone
and colleagues that developed an expert model based on interview data to better understand
the methodology and reasoning they use while programming cobots by analyzing their discourse.
They developed the Safety First Expert Model, which contains a set of related frames such
as safety, application, trajectory motion, performance factors, and operator involvement.
Now that we had an expert model, it needed to be reorganized and translated into a form
conducive for learning.
To do this, we constructed 4 higher-level interconnected expert frames from this model
that incorporated different aspects from the Safety First expert model, which serve as
lenses with which to evaluate cobot programs.
These new frames consisted of Safety Concerns, Program Quality, Robot Performance, and Business
Objectives.
We created a set of issues that might be addressed within each frame, and how they relate to
or are dependent on other frames or issues.
For example, only after considering the reachability of a certain pose by the robot would the programmer
consider the end effector pose of the robot as it traveled around it.
Likewise, summary measures like cycle time only make sense when the program has error-free
usage of external machines, given that the cycle time will be dependent on how long the
robot needs to wait on them to finish various processes.
We took these Expert Frames and incorporated them into the CoFrame interface.
The interface has 4 main sections.
The first section on the far left is the Review Tile.
To the right of this are the Simulator and Info Tiles, and to the right of them is the
Program Editor Tile.
The Review Tile is the main gateway to the content of the Expert Model.
At the top of this section is a navigation bar that allows designers to select the different
frames with which to view the interface.
Within this tile, changing the frame causes different sets of issue categories to appear.
These are arranged according to the dependencies discussed previously, and when dependencies
are across different frames, prompts are provided to switch frames and address those issues
there before continuing.
This allows designers to progress through the different issue types and unlock them
as they make changes.
To the right of the Review tile are the Simulator and Info Tiles.
Like the review tile, these tiles also change according to the active expert frame.
When relevant, visualizations of certain issues appear in the simulator when those expert
frame issues are selected.
Below the simulator, the feedback provided in the info tile is customized for different
frames, issues, and selected items.
To the right of the interface is the program editor.
This blockly-like visual-programming tile allows the programmers to construct their
programs and initialize visualizations of relevant blocks of code.
This visual programming language has visual representations for the different actions,
as well as data like locations, waypoints, things, and trajectories.
General information on these types can be seen in the info tile, as well as a detail
sidebar.
The Safety Concerns Frame includes five different issue types, ordered based on dependencies
and the progression of the programming process.
The first issue type is “End Effector Poses,” which identifies cases where the gripper is
moving quickly in the direction of the gripper, which could mean a potential for the robot
to push against the human in an unsafe way.
For example, this trajectory elicits a warning that shows the robot making a short jabbing
motion while moving along the trajectory.
Slowing the trajectory, adding waypoints, or changing the trajectory type could resolve
this warning.
Warnings, as opposed to errors, can be dismissed by the designer after inspecting them.
The remaining issue types include “Thing Movement,” which tracks the interaction
with unsafe objects, and “Collisions,” which track collisions between the robot and
itself and the environment.
Next is “Pinch Points”, which visualizes the places on the robot where the spacing
between parts is narrow enough to be a pinch opportunity for hands or head.
Finally, “Occupancy” tracks overlap between trajectories and designated human zones.
The Program Quality Frame addresses issues with the program construction and parameterization.
The first issue, “Missing Blocks,” identifies cases where block inputs, such as trajectories,
are missing from actions.
The next issue is “Missing Parameters,” which highlight cases where actions or skill-calls
lack necessary parameterization, for example machines, locations, and waypoints.
“Machine Logic” highlights cases where there are errors in the timings of machine
usage, or usage of their inputs and outputs.
“Unused Skills” identify cases where a skill is never used in the program, and "Unused
Features” identify cases where things, machines, locations, and waypoints are never used.
“Empty Block” issues appear when there are hierarchical blocks or skills that do
not contain any actions and can therefore be pruned for clarity.
In all these cases, the relevant block is highlighted in the editor for each issue that
is presented.
The Robot Performance Frame is next, starting with “Reachability” issues, which identify
locations and waypoints that are unreachable by the robot.
“Joint Speed” and “End Effector Speed” identify trajectories where the robot moves
its joints or end effector too quickly.
For these speed-based issues, the speed of the trajectory is visualized as lines in the
3D space according to the position of the corresponding joint or end effector.
These issues can be addressed by modifying the trajectories with waypoints, adjusting
the locations and waypoints themselves, adjusting the speed of the motion, or swapping between
interpolation styles based on joints or inverse kinematics.
After addressing these issues, the system identifies “Payload Issues,” where the
robot picks up things that are too heavy to carry.
Finally, “Space Usage” considers the space used in each trajectory, aiming to minimize
the total amount, and visualizes the space as a convex hull bounded by the locations
of all joints on the robot.
The final frame is the Business Objectives Frame, which features 3 feedback values that
calculate how the Idle Time, Cycle Time, and Return on Investment have changed over the
course of the program’s creation and modification.
These values are updated each time the feedback is refreshed, and are visualized in a graph
in the info tile.
These frames are meant to guide the user as they construct a program meant to activate
the various issues in our Expert Frames.
This takes the form of an assembly task where the robot creates a knife from a set of parts
from a variety of sources, including conveyor output and a 3D printer.
The unsheathed knife is an unsafe object, so there is a plastic case that allows the
robot to carry the knife in a safe manner while the edge is covered, an example of the
system allowing both an elicitation of an issue, in this case the thing movement safety
issue, as well as a mechanism with which to solve that issue with a bit of creativity.
In summary, CoFrame is a new interface designed to communicate the concepts of an Expert Model
of cobot programming to novices through a set of constructive issues that are tracked
during development, and rich visualizations to allow these novice programmers to better
see where violations of the expert model appear.
The CoFrame interface has not yet been evaluated, but we plan to perform a two-phase evaluation,
first with experts to elicit their feedback on the construction of the expert frames and
feedback, and then with novices to measure the effectiveness of the system itself.
We see CoFrame as an exciting platform for future work and research, both as extensions
and evaluation targets.
The other authors and myself thank you for your interest and time, and look forward to
fruitful discussions with you all.