Hi I’m Joseph DelPreto from MIT, and our work uses wearable muscle and motion sensors
to detect gestures that can remotely control robots.
As human-robot interaction continues to become more effective, robotic assistants can become
more pervasive and benefit a wide range of societal applications.
But whether working directly with a robot or remotely controlling a robot, there is
still often a communication barrier.
We’d like to reduce this gap by allowing robots to understand more of our nonverbal
commands and intentions.
We’d also like it to be easy for new users without robot experience to start quickly
interacting with robots.
Our current work explores these goals within the context of remotely controlling a robotic drone.
The system creates an expandable vocabulary of 8 navigational gestures that are detected
using wearable muscle and motion sensors.
The gesture detection pipelines are designed to be plug-and-play, meaning they do not require
offline calibration or training data from each new user.
This builds on past work related to robot and computer control interfaces.
Various modalities have been successfully leveraged for robotic control including speech,
virtual reality, web-based interfaces, or physical interactions.
Our current work focuses on discrete gestures, to allow for a natural user experience while
reducing model and sensing complexity.
Gestures are often detected using vision, sensorized gloves, or muscle signals.
We decided to focus on wearable muscle and motion devices to minimize sensing infrastructure
that a new user would need to deploy.
Muscle signals can also provide information about force, stiffness, or fatigue that may
be hard to observe otherwise.
They can also infer hand motions without requiring sensors on the hand itself.
Within this framework, we will focus on designing a plug-and-play system for robotic teleoperation
that requires minimal user setup.
We use EMG electrodes to detect muscle signals from the biceps, triceps, and forearm.
The forearm device also includes an accelerometer and a gyroscope.
We then define a suite of 8 navigational gestures that serve as motion primitives for controlling the robot.
Briefly stiffening the upper arm is similar to a cringe response when seeing something go wrong,
so this gesture is used to stop the robot.
Squeezing the fist is similar to a grasp or a selection, and is used to move the robot forward.
Briefly rotating the hand CW or CCW causes the robot to turn in place.
Finally, waving the hand inward or outward causes translation motions.
If the hand is held vertically, this motion waves the hand left and right.
But if the hand is held horizontally, this waves the hand up and down.
We use two types of gesture detection pipelines.
Most of the gestures leverage an algorithm for clustering streaming sensor data online.
This essentially creates an adaptive threshold that does not require any past calibration
or training data, but it uses expert knowledge of the expected signal shapes.
The wrist motions are detected using a neural network that is trained on labeled data
from previous users of the system.
Let’s take a closer look at how these pipelines operate, starting with the stiffening gesture.
The biceps and triceps EMG signals are multiplied together to indicate stiffness,
which is high when both muscles are tense.
We pass this indicator to an adaptive clustering algorithm to detect periods of high stiffness.
It uses a Gaussian Mixture Model to cluster the unlabeled streaming data.
This model is periodically recomputed a few times per second to adapt to newly collected data.
To reduce the amount of data that needs to be stored, we use a rolling buffer of data
for each of the mixture components.
Here, there are three components and three buffers of data.
These represent when the user is relaxed, performing other task motions, and the high stiffening state of the gesture.
Using rolling buffers limits the computation time and allows the mixture model to be fit in real time.
Having a separate buffer for each state also means that gesture examples can remain in
the history even if they’re only made sporadically.
We can then look at when the streaming data is classified as the highest mixture component
to detect when a gesture is being made.
To illustrate this, we can look at how the algorithm operates for the stiffening gesture.
The left graph shows the stiffness estimate, and the right graph shows the contents of
the three rolling buffers that are used to create the Gaussian Mixture Model.
Initially, the model hasn’t seen any examples of a gesture so it thinks that small arm motions
might be gestures.
But once it sees a single gesture example, the cluster thresholds adjust and learn to
separate small movements from the gesture.
So it essentially performs calibration online while the user is performing gestures and
interacting with the robot.
This approach can allow gestures to be adaptively detected without labeled training data and
without significant calibration routines, as long as the detection problem can be cast
as a dynamic thresholding problem.
This gesture can then be used to control the robot, in this case to stop and hover.
This clustering algorithm is used as a building block for the other detection pipelines too.
For example, fist clenching can be detected by estimating lower arm stiffness
instead of upper arm stiffness.
EMG signals from the inner and outer forearm are multiplied and then clustered online.
The resulting gesture detections are used to command the drone to move forward.
To detect rotation gestures, we process the gyroscope data to create an indicator that
highlights the expected signal shape.
Since the gesture consists of briefly rotating the hand and then returning to neutral, we
expect two peaks of opposite sign in the angular velocity.
To detect this, we keep a rolling buffer of the angular velocity around the forearm axis
and compute what we call a "folded product."
At each timestep we split the buffer in half, negate and flip one half, and then multiply the results.
Conceptually, this folds the buffer onto itself
and creates a signal that is high when there are two velocity peaks of opposite sign.
We pass this indicator to the online clustering algorithm to detect when it is high enough to indicate a gesture.
This gesture is then used to turn the drone in place, either to the left or to the right.
The final classification pipeline uses a neural network system that was developed in a previous
work to detect left and right hand gestures.
It is trained on data from previous users,
so a new user does not need to provide additional training data.
Our current system augments this pipeline in two ways.
It uses the forearm accelerometer to infer the arm orientation, and then interprets the
wrist motions as being left and right or up and down.
It also aims to reduce false positives by detecting when the forearm is activated enough
so that the user is probably making a gesture instead of other small motions.
To do this, the inner and outer forearm EMG signals are added together and then passed
to the online clustering algorithm.
The result is used to gate the neural network predictions.
The final gesture predictions are used to control the drone to move left and right or up and down,
depending on the orientation of the hand when the gesture is made.
To evaluate the gesture detection pipelines in a controlled manner, LEDs cued users to
make specific gestures at specific times.
All gesture detection pipelines operated online, but the robot was not controlled.
Six users participated in these experiments, and the individual classifiers correctly detected
over 97% of their 1,200 gestures.
When a gesture hierarchy was imposed to make the outputs of the separate classifiers mutually exclusive,
the overall gesture detection accuracy was 93.7%.
After completing the open-loop sessions, each user was allowed to make any of the gestures
at any time to remotely control a drone.
They made over 1,500 gestures in total, and the drone correctly responded to over 81% of them.
Some of the performance decrease compared to open-loop sessions may be due to the wrist
motions being performed with different timing than the neural network expected.
It may also be due to the clustering models requiring a couple of gestures to adapt to
changing signal levels.
During these closed-loop experiments, the six users were able to successfully pilot
the drone through hoops and around obstacles.
They flew the drone for a total of 2 hours over 25 sessions.
The drone flew through a hoop 38 times, navigated around the posts 6 times, and hit a hoop 12 times.
The robot motion was discretized for the current experiments, but could be made continuous in the future.
These experiments indicated that the classifiers successfully adapted to varying signal levels,
and that the users could successfully use the gesture vocabulary to teleoperate the robot.
In the future, this gesture vocabulary can be extended for increased control and for
other robotic platforms or electronic devices.
In conclusion, this work presents a gesture vocabulary designed for remote robot control
using muscle and motion sensors.
It leverages an algorithm for clustering streaming data in real time to detect gestures from wearable sensors.
It’s also designed to be plug-and-play,
which can reduce the barrier to casual users interacting with robots.
We look forward to expanding this vocabulary and to evaluating it with more users
and with more robots in the future.
Overall, the system takes a step towards improving human-robot communication
and leveraging the benefits of more pervasive human-machine collaborations.