I am Pragathi Praveena from the University of Wisconsin-Madison and I will present our
work in the space of robot teleoperation. This talk is divided into four parts. Through
this talk, I want to introduce a novel way of providing feedback to a user who is remote
operating a robot. In robot teleoperation a user controls a robot
from a distance, especially so that they can work in remote, high-precision or hazardous
environments. During teleoperation, it is important to provide
details about the remote environment to the user. For example, in this case, the user
receives visual feedback from a camera on the robot.
Designing force feedback is particularly challenging. For example, if we want to communicate to
the user how hard the robot grasps an object or how heavy the object feels?
One approach is to use highly specialized devices that can give the user a “sense”
of the properties of objects in the remote environment. In this system, forces are measured
or estimated in the remote environment and these forces are applied to the user’s body
or arm through the force feedback device. There are a range of such force feedback devices
that can improve user experience and task performance. However, there are also challenges
with using these devices. Sometimes the force feedback can be unstable and result in unsafe
operation. They can also be bulky or fixed to the ground and thus restrict the user’s
natural range of motion. Can we use an alternate, complementary approach
to convey such force information? In this work, we look at an approach introduced
by Anatole Lecuyer called pseudo-force feedback where force feedback is simulated using vision.
For example, here the touchpad surface that the user is pressing down on with the stylus
is actually very stiff. But, by visually changing the deformation of the virtual surface, the
user feels that the touchpad surface is elastic. Prior work has looked at using pseudo-force
feedback for such desktop and virtual reality interfaces but it has not been applied to
a robot teleoperation scenario. So in this work we share our initial evaluation of pseudo-force
feedback for robot teleoperation. Our work uses an implementation of teleoperation
called mimicry control. The user’s hand movement is tracked in real-time and the robot
mimics their movement. -- During regular teleoperation, the user
can control the robot by moving their arm. The robot closely follows the user's motion.
This means that as the user moves their hand a certain distance, the robot’s end effector
moves by a similar distance. When we want to communicate a larger weight, the robot
moves slower. Thus, to move the robot by the same distance, the user has to move their
hand by a larger distance. Why might this work as a cue to communicate
weight of objects? During teleoperation the user receives feedback
from two senses: First, the robot movement that the user sees through their visual sense.
Second, the user’s proprioception which gives them a sense of where their arm is in
space. During regular teleoperation, these two feedbacks are consistent. However, in
the scenario where we add the cue for weight, the movement of the robot differs from the
user’s input and thus we create a conflict between the information from the two senses.
Through the remainder of this talk, I will refer to our cue as the motion cue because
variation in the motion of the robot causes the sensory conflicts.
Drawing from prior work, we design our motion cue such that the robot lifts a heavy object
slowly. We manipulate the motion of the robot only in the direction of weight, which is
the vertical direction. Here we see a graph of the vertical velocity
of the user’s hand in blue and the robot end-effector in green as the user lifts the
object vertically. The lift motion consists of three phases. As we go through the different
phases, please follow along with the video on the right. Phase 1 starts just as the grippers
close around the object. In Phase 1, we simulate inertia. The robot moves very little compared
to the user’s input thus exhibiting resistance. In Phase 2, the robot moves faster than in
Phase 1, but still less than the user input. This corresponds to a lower lift velocity.
In Phase 3, the motion cue switches off and the robot is in regular teleoperation mode.
The robot motion in the different phases is determined by the equations on the right.
This factor g, which we call the gain factor, allows us to generate cues for a range of
weights. A gain of 1 means regular teleoperation and lower the gain, larger the weight.
In this section, I will be posing our three research questions and a subset of our results
from the user study with 10 participants. Let’s start with the first research question.
Will participants report perception of weight without explicitly being asked about it?
To test this, in our first task, participants lifted two bean bags with identical weights.
One of the bags was simulated to be heavier with our motion cue for weight. Participants
repeated this lifting three times and freely described any differences they experienced
when picking up the two objects. We found that eight out of ten participants
reported the effects of the motion cue, for example that they needed to make more effort
to grab the cued bag. But, only two participants attributed this difference in robot movement
to difference in object weights. The rest thought that the difficulty of movement was
due to lack of practice or bad positioning. The second research question is about the
mechanism of the cue. Is the cue only a visual cue and based on the user watching the robot’s
motion, or is it dependent on the conflict that we create between the visual and proprioceptive
senses? To test this, in the next task we had two
conditions. One where the participant controls the robot and the other where the participant
only watches a video of the robot movement. We call these conditions Controller and Observer
conditions. For each trial, participants lifted two metal cans that looked similar and weighed
the same. We created the perception of a virtual weight by using one of five levels of gain
to generate motion cues. Participants were not aware that the cans weighed the same and
they were asked to respond as to which can was perceived as heavier. Each participant
had 15 trials in each condition. This is a heatmap of participant responses
to which can was heavier in the two conditions: Controller on the left and Observer on the
right. In this heatmap, if more participants got the answer right, we see darker colors.
We see darker colors on the heatmap for the Controller condition. What we find is that
the cue resulting from sensory conflict leads to better task performance as compared to
visual cues alone, especially when it comes to being able to differentiate between different
levels of the cue. Our third research question is whether the
motion cue improves task performance and user experience. For this we designed two tasks.
Similar to the previous tasks the cans used in this task were visually similar and weighed
the same. In the first task, participants categorize
cans into two bins – Recycle or Stock – based on perceived weight of the can. Lighter cans
go into Recycle, while heavier ones go into Stock. In the second task, participants sort
four cans from heaviest to lightest. Both tasks have three conditions. In one condition,
participants receive a gold-standard cue, for example the cans were labeled with a number
indicating the virtual weight. In the second condition, participants receive the motion
cue and in the third condition they receive no cue.
We aggregate the scores for each task and we find that in terms of task performance,
participants perform better when they receive the motion cue as opposed to no cue. Task
performance in the presence of the motion cue is comparable to the performance in the
presence of the gold-standard cue. However, in the binning task, despite this good performance,
participants complained that they did not readily notice the cue in the presence of
complex robot movement. In terms of user experience, we found that
in the sorting task the presence of the motion cue significantly reduces workload on the
user. This may be because as compared to the binning task which was a binary classification,
the sorting task challenged participant sensitivity to small differences in perceived weight.
I will conclude by summarizing our three main takeaways.
Users rely on coupling between visual and proprioceptive feedback. Complex robot movement
in teleoperation creates challenges and opportunities for designing cues for pseudo-force feedback.
As tasks get more challenging, presence of the motion cue is more valuable to the user
by improving their performance and reducing workload.
Thank you for your time.