A framework and call to action for the
future development of EMG based Input in HCI. I am Ethan Eddy, a PhD student at the
University of New Brunswick, where pioneering electromyography research has
been explored since the 1960s at the Institute of Biomedical Engineering.
Throughout this long history, the majority of research has largely focused on
improving prosthetics technology for amputees.
However, for the past seven years, the
UNB HCI Lab has been working closely with the Institute to explore how HCI can help
inform prosthetics research and how EMG technology can be improved and repurposed
as an intuitive and reliable handsfree input modality.
The paper that I am presenting is the culmination of this seven year
collaboration and highlights the need for a fundamental shift in how we as a
research community should be approaching the use of EMG as an input modality.
Because despite the compelling uses of EMG
for input and its growing popularity, it is still far from the level of usability
that would be needed for consumer level applications.
In this paper, we emphasize what parts of
prosthetics research and technology can be applied directly to general purpose EMG
input, but more importantly, where EMG research needs to diverge and evolve for
its future success as a general purpose input modality.
Because fundamentally, using a prosthesis is very different than playing a video
game, controlling a drone or interacting in mixed reality.
Electromyography, or EMG, is the
measurement of the electrical activity generated during muscular contractions.
Using surface electrodes, this
physiological signal can be recorded, processed and passed to machine learning
algorithms, which can discriminate between contractions.
Effectively, this turns the human body into a controller where muscular
contractions or gestures are associated with interactive commands.
But before we explore the evolution of EMG
in HCI, we must first understand its use for prosthesis control.
Since the mid 1900s, EMG based
control has primarily evolved in the prosthetics community
to enable amputees to control their prostheses. The very first controllers
took a one muscle, one function approach where EMG was recorded at antagonistic
muscle sites to enable one degree of freedom control.
However, this was not necessarily intuitive and so to increase the input
space, pattern recognition based techniques were leveraged.
Inevitably, the success of this technology in the prosthetics field led to its
exploration for general purpose applications within the HCI community.
EMG became an attractive solution in the
early 2000s for HCI researchers due to increased attention from DARPA funded
research and the release of commercially available devices.
Since then, the HCI community has and
continues to explore EMG based control for a variety of applications.
And although it is gaining attention as a promising handsfree input modality both
academically and from industry, its use is still limited.
In this work, we make three contributions
to our goal of unlocking the true potential of EMG as a robust and reliable
input modality for general purpose applications.
We first synthesize the main limitations
of EMG based input that exist in a sampling of HCI work.
We then outline a framework for designing and building EMG based control systems.
And finally, we provide a call to action identifying the key areas of research that
need to be addressed for the continued and improved success of this technology.
So what is hindering the widespread
adoption of this inherently promising technology?
Addressing this question was the first goal of our paper.
To start, we sampled approximately 50
papers from the ACM digital library. Using trends found in this body of work
in combination with our research group's experience in this space, we found four
main themes relating to how EMG based interactions are designed, limitations in
the design of control systems, the suitability of system evaluations, and
limitations in the reproducibility of results.
Firstly, we found a reoccurring theme of
insufficient focus on interaction design for use in EMG based interfaces.
The issue is that the interaction design of general purpose EMG input and
prosthesis control are fundamentally different.
To understand exactly what this means,
let's explore the difference between the two.
The use of EMG based control in the prosthetics community is now quite
established and is commercialized through several vendors.
In these systems, electrodes are placed on
the residual arm of an amputee, meaning every fit and setup is unique.
Additionally, the inputs that are
recognized are physiologically appropriate contractions.
These are contractions that correspond to an equivalent device output.
For example, think hand closed to close the prosthesis and hand open to open it.
But maybe the most unique aspect of prosthesis control is that control schemes
are developed to enable the continuous control over a device.
This means that predictions are being made on milliseconds worth of data to enable
responsive control to microadjust a prosthesis.
Although suffering from many of the same
limitations of prosthesis control, such as the electrodes for capturing EMG signals
physically shifting on the skin, muscle fatigue and noisy signals, EMG for general
purpose use has many of its own unique challenges.
For example, the typical muscle contractions that naturally occur during
our daily lives are not necessarily ideal for EMG devices that are always on, as
they can inadvertently be interpreted as a command.
Think about how many times you might
inadvertently open or close your hand while performing your daily activities.
However, potentially the biggest difference between general purpose and
prosthesis control is that HCI is full of discrete inputs for example, button
presses, clicks, swipes, flicks and gestures.
These actions are fundamentally different
than the continuous ones used to make micro adjustments to a prosthetic device.
The second theme we discovered related to the limitations in the design and
optimization of control schemes for the designed interaction.
While HCI researchers can draw from the
large body of work done by the prosthetics community, including guidance on
algorithms, features and signal processing, we must also recognize that
our control systems are fundamentally different and have unique requirements.
This leads to a fundamental shift in how
we design our control schemes and opens up many new directions for future work.
Potentially, the biggest difference in
model design is that amputees most often have unique residual musculatures, so
creating user independent models or control systems that work for any users is
extremely challenging. In general purpose applications,
however, this becomes a much more realistic possibility for ablebodied users
and is a crucial next step if this technology is to gain widespread adoption.
In general, there are two types of evaluation for EMG based control systems.
In offline evaluation, the performance
assessment of a control system is done on a prerecorded data set.
This has been normalized in the prosthetics community since finding
amputee participants is difficult and prosthetic fittings are expensive.
But offline performance are not indicative
of online usability, as there are numerous confounding factors that can affect the
real world performance of a control scheme.
So, while offline evaluation is good for parameter tuning or experimenting with
different control systems before deploying them to the real world, it is not enough.
In contrast, online evaluation focuses on user in the loop interactions, where the
user gets feedback from control systems being tested.
This is fundamentally different than
offline evaluation, where the user has no stimulus to respond to.
As a community, online evaluation must become part of all EMG related research.
to truly assess the viability of a control scheme. EMG based system evaluations must
mature to the standards of the broader HCI research community.
The fourth and final theme is reproducibility, a challenge that has
recently garnered increased attention in HCI.
In general, we found that there was often
insufficient experimental control or detail provided to enable the replication,
corroboration or extension of previous work.
As a community, we should work together and collaborate through shared code and
data if we want to accelerate the advancement of this technology.
The four themes I just described informed
our second goal to facilitate the design of EMG based control schemes.
We did this through the creation of a straightforward design framework.
In the first step, we need to consider
whether the inputs we want to recognize are continuous static contractions such as
holding your hand closed, or dynamic discrete gestures such as a finger snap.
We must then consider the technology that
will enable the recognition of these inputs.
Things to consider are the sampling rate,
number of electrodes, electrode location, and the inclusion of additional sensors.
Then we need to select whether the control system will output device commands based
on a predefined periodic parameter or continuous control, or only after a
discrete action with a particular start and end, or discrete control.
Next, we must select an algorithm that
will best enable the recognition of the inputs.
In general, stationary models should be
used for static contractions and temporal models for dynamic gestures.
Another important consideration is whether the problem is closed set, where the model
assumes each action is associated with one of the gestures used to train the system,
or whether it can handle out of set actions. After
we must fine tune the models using adequate windowing and features.
Due to the stochastic nature of EMG, this
is required to increase the information density of the signal so that machine
learning models can discriminate between contractions.
Then we need to decide about the context dependence of the model.
Is it susceptible to confounding factors? And its user independence
Does it work across users?
And finally, we must evaluate the model.
This step should consist of some sort of online usability assessment.
So what does all this mean moving forward?
Well, we believe we are at an exciting moment in the progression of this research
for the HCI community to be able to contribute to the adoption of an
interaction modality that is more applicable than ever with the availability
of new display technologies like mixed reality headsets and the continued
movement toward ubiquitous computing technologies.
And these contributions are needed in a number of important directions.
For example, we need to explore the future of EMG wearables.
How can we create wearables that are comfortable, reliable, and less prone to
confounding factors such as electrode shift?
Or potentially, how could we facilitate
the design of EMG based control systems through toolkits, which is a direction
that we're in the process of working on at UNB?
Most importantly, how can we make this technology easy to use and convenient?
While there exists many use cases involving handsfree interactions where
camera based input is less feasible, EMG based control must offer solutions that
offer comparable recognition, accuracies and reliability.
In this work, we have highlighted the major factors that are limiting the
progression of EMG based input, provided a framework to facilitate the design of EMG
control systems, and provide a call to action for future research in the space.
While we can learn from the large body of prosthetics research, moving forward,
there'll need to be some sort of divergence from the prosthetics field as
our understandings and practices around EMG continue to mature.
We hope this work serves as a stepping stone to enable the HCI community to truly
unlock the hidden potential of EMG as a robust input technology.
Finally, I'd like to acknowledge my
supervisors and co-authors, Erik Sheme and Scott Bateman, and all the staff and
clinicians at the Institute of Biomedical Engineering at the University of New
Brunswick for making this project possible.