Hi, my name is Bryce Ikeda and I work with Dan Szafir at the University of North Carolina at
Chapel Hill. And I'll be presenting my work focused on building better tools for robot
programmers. We've all had moments like this where you're programming a robot, think it's gonna work,
and then you get frustrated because it does the complete opposite of what you expected.
Why are programming robots so difficult?
For one robots operate in variable environmental factors, and can have irregular sensor reliability
that can often lead to unexpected behavior. Also difficult is building the algorithms that run on
a robot. To help with this process. Perhaps you use one of a multitude of applications developed
to help roboticists. However, while each may be beneficial in a specific way, programmers still
need to download and learn how to use each one. Furthermore, these tools have been developed in
an ad hoc manner, and availability of each tool is dependent on the robot platform being used.
So how should we design a robot debugging tools? That is what I explore in my work. While we
commonly design robot systems with a user-centered design approach, I propose we take a similar step
for building tools for ourselves. First, we need to understand who are the users? In this case,
roboticists. Next, we need to understand what is shown. Fundamentally, we know that visualizations
are helpful and intuitive. On the left, it's difficult to understand the robot joint positions
as text. However, on the right when depicted as its visual representation in a 3D visualizer like
RViz, it's easy to understand. Still, this type of interface displays 3D information on a 2D screen.
That's why in my research, I'm interested in using augmented reality technology, which displays 3D
information in 3D. In the past few years, there has been an increased interest in applying this
technology to communicate robot motion intent, display robot hardware fault visualizations,
and visualizations of raw data for general robot use cases.
While there are numerous examples of how AR might be used to display robot data to users,
what is useful for robot debugging? One way to figure this out is by
viewing this from the perspective of the visualization community,
so we should understand what are the underlying data types we are displaying.
Data types are key characteristics of robot data. By identifying intuitive representations for each
type, we can create a standard for all roboticist to adhere to, thereby making it easier to
understand the displayed information across groups. So we analyze prior work to understand
what may be relevant for robot debugging. For example, researchers have initially categorized
robot data into four groups geometric vector, scalar, and abstract. We then expanded abstract
data into sub components based on data commonly displayed in robotics applications. As an example
of what all this means, vector data has both a magnitude and a direction, such as a robust
trajectory. Raster data is a grid of scalar values that can also have multiple bands of information,
an example being an occupancy grid. For more information on these categories, please see
our paper. Next, we need to think about how these categories are shown. What is his visual encoding.
While some visualizations have an inherent representation within the scene,
others may need to be transformed into a clear spatial representation such as scalar data.
For example a robot's battery level could be represented using a volume
that varies between points zero and 100% battery.
geometric data has an intrinsic mathematical representation that can be displayed using points,
lines and shapes, such as the robots digital twin.
Vector data has a magnitude and an orientation and should be displayed from its point of origin
along its axis of measurement. An example being the future path trajectory of a robot.
Text Data is information expressed verbally and
will typically be displayed situated to the information that correlates to
raster data is a grid or scalar values that can include multiple information bands,
one example being the occupancy grid.
Lastly, categorical or boolean data have distinct attributes that belong to a specific
concept. For example, robots current state can be differentiated by using different colors.
While we just went over potential visual encodings. As for any data type,
there are multiple ways to visualize it. To decide which visual encoding we should use.
We need to understand why is the user looking at it? What's the data desk?
In the context of programming and debugging a robot. Some of the data tasks might include
analyze multiple streams of data, find unexpected patterns in data,
or make predictions about future states.
Following these steps, we designed and developed our own augmented reality visual debugging tool.
So, the last step would be to test the system. We ran a study to understand
how different visualization paradigms affect a roboticist debugging practice
To evaluate
on how well current and emerging visualization tools promote the success of these data tasks,
we investigated three data visualization environments,
a 2D GUI, which displays solely 2D information RViz, which provides 3D data on a 2D screen,
and our own developed augmented reality debugging tool, which provides immersive 3D visualizations.
We ran a study on 24 participants randomly assigning eight participants to each debugging
tool. Each participant had experienced creating at least one robotics project using ROS .
Users were given two programming tasks to complete a 30 minute detection task where the
robot needed to detect an object in front of it and drive up to it and stop a short distance away.
And a 1 hour navigation task where the robot needed to search a room for an object
placed in an unreal location, drive up to it and stop a short distance away.
After the participants completed the tasks, we provided them with a survey fill out. Following
the survey, we collected open ended verbal feedback regarding their assigned debugging tool.
After analyzing our subjective feedback using a thematic analysis
we came up with these codes will only go over a few of them due to time.
First is knowledge of past and future states. Participants wanted a simple way to visualize
future behavior and compare it to past runs. For example, this participant wanted a standard
video player that could easily stop, play or reverse behavior recordings of a robot.
This is actually a feature already provided by tool called rqt. However, when asked only three
participants reported frequently using rqt as part of their regular debugging practice. So
it might be worth investigating how to properly integrate this feature into a single tool.
Next is mitigate occluding visualizations.
occluding visualizations hindered the participants
ability to develop conclusions from data. This ability is also known as sensemaking.
For example, one user didn't know that the future path projectory was being displayed because the
robot or the previous path was occluding it can you find the next waypoint in the picture.
Probably not because it was under the robot.
Lastly, we'll go over perspective taking.
perspective taking is a user's ability to understand a robot's behavior from
the viewpoint of the robot. While we originally hypothesized that augmented reality could improve
the issue of switching perspectives between the screen and the robot while running live code,
we didn't take into account that user still needed to look at the terminal
to see if any errors or warnings appeared. As a consequence,
users were still frustrated by having to switch between the HoloLens viewpoint and the computer.
From these codes, we synthesized four major themes to guide designers on the types of support
roboticists need with debugging tools. Again, these are described more fully in the paper.
To briefly summarize the key takeaways from these themes, here are some guidelines for future tools.
The first guideline is to group visualizations by the task they inform rather than simultaneously.
For instance, to reduce the visual clutter on the right, perhaps we can separate
perception visualizations from navigation visualizations, so users can choose to observe
only the most relevant information to support their current data task.
Second, we should provide frequently ref erenced data streams in the main interface,
this may improve perspective taking. For example, an important piece of information that was
displayed on our task was the path of the robot. However, we know from earlier in the presentation
that another frequently referenced data source was a user's terminal. While adding a terminal to the
AR debugging space could be a simple solution. Future work could explore how to provide errors,
warnings, and other debugging information intuitively situated within the viewing scene.
The third guideline is to allow for dynamic alterations of the visual saliency
of data renderings. For example, in the image on the right to let
users see the path of the robot a customization feature could be
to make the robot translucent when it occludes important information.
Lastly, we should consider already established abstract data representations in the robotics
community when creating new abstractions. For instance, red is typically displayed for errors,
yellow for warnings, and green for success. We could apply similar encodings to our
visualizations by making paths successfully traversed green and paths being executed as
yellow as seen on the right. By understanding how design aspects improve user experiences,
we have the opportunity to improve the programming and debugging capabilities for roboticists.
While this may seem obvious, they try a community commonly takes a user centered design approach
when developing systems for end users. We believe it is critical that we start
applying the same ideas when developing our own tools. Thanks for listening
Transcribed by https://otter.ai