Hi Everyone, thanks for your interest in our research. My name is Mengjia and I am a
phd candidate from UC Santa Barbara. It is my honor to present our research on
PneuSleeve. This work was conducted with Facebook Reality Labs and I really
appreciate the valuable support I received from all my coauthors.
Pneusleeve is a pneumatic fabric sleeve that is able to render controlled haptic
feedback with compression, skin stretch, and vibration.
It is soft, compact, and can render a large range of expressive haptic feedback on
user's forearm.
So why we are interested in haptics?
We feel things in all different ways in our life, from perceiving materials, to
interpersonal interactions
Wearable technologies with haptic feedback have also entered our life in recent years.
However, most human-computer-interactions don't use haptics, and when they do,
it's simple vibration. Wearable haptic displays that can communicate more
information would be valuable.
The haptic information we received is handled by our brain in a complex way
through multiple types of haptic receptors in the skin. The target location of
PneuSleeve is user's forearm, which is covered by hairy skin. The hairy skin contains
several types of tactile mechanoreceptors.
Through these mechanoreceptors, mechanotactile stimuli, such as pressure, vibration,
and skin stretch can be felt and interpreted by human. Because different types of the
mechanoreceptors have different sensitivity over different stimuli, It is believed that
stimulating multiple types of the haptic receptors can help generate more expressive
haptic feedback.
There are many wearable haptic devices made of soft or rigid components. These
devices are mostly designed to be worn on
fingertips,
wrist
hand
forearm
and upper body
using pneumatic, motor-driven, shape memory alloy and other actuation methods.
Many of those haptic wearables use multi-modal mechanotactile stimulations, in
order to render a large range of haptic stimuli.
For example, bellowband and force jacket combine compression and vibration in their
actuation
Researchers have also combined compression with skin stretch in their devices.
Here is a brief summary on existing research on haptics. It is clear that
none of them have integrated compression, skin stretch, and vibration all together in
a soft wearable device.
Besides multi-modal actuation, we further designed soft sensors to enable closed-
loop regulation of compression force, to render controlled stimuli across different
users. We also mechanically validated the performance
of the actuator, sensor, and the closed-loop control.
To show that users can actually feel the stimuli, we conducted psychophysical studies
on absolute detection threshold and just-noticeable difference.
Finally, to investigate how users can understand the meaning from multi-modal haptic
cues, we conducted subjective assessments on 23 selected feel effects
With the capability of the device, we also envisioned some potential applications of
PneuSleeve including notification, navigation, discrete communication, and gaming.
When paired with other technologies such as tracking, we expect that PneuSleeve
can be of great use to people's daily life.
To implement PneuSleeve system, we developed soft actuators, soft sensors, and
closed-loop force control method.
With the goal of rendering compression, vibration, and skin stretch with only one
type of soft actuators, Fluidic fabric muscle sheet (abbreviated as FFMS) came as a
good candidate.
These actuators can extend and contract without radial expansion comparing to
mckibben muscles.
Comparing to shape memory alloys, FFMS actuators change length without the need
of temperature change.
Therefore, these actuators can provide compression and skin stretch without
introducing other haptic cues.
By using pneumatic actuation with fast-switching valves, vibration could be generated.
To evaluate how those actuators render forces and vibrations, we fabricated the
actuators based on the method described in the FFMS paper.
We first stack two layers of non-stretch cotton fabrics,
Then put straight stiches on top,
After that, we insert the soft tubing in between the fabric layers and the stitches
We seal one end of the soft tubing and wrinkle the fabrics
And attach rigid tubing on the other end.
Therefore, when the soft tubing is pressurized,
The actuator extends longitudinally because of the radial constraints by stitches and
fabrics. We use this state as the initial and default state,
When the soft tubing is depressurized, the actuator contracts and exerts force
longitudinally.
We conducted evaluations on the quasi-static linear force for those actuators. The
results show that the compact actuator can generate more than 4 N when both ends
are fixed, and can still generate more than 1 N force when one end has moved 16 mm.
For vibration characterization, we conducted transient analysis of the compression
force across different frequencies. Results show that although the force magnitude
attenuates with the increased frequency, an easily perceivable force of 50 mN can be
achieved at 20 Hz.
We made six of these actuators, and integrated those actuators to a stretchable fabric
sleeve
Two of them are compression actuators located on the proximal and distal forearm.
The rest of four are skin stretch actuators uniformly distributed around and tangent
to the forearm
All six actuators can render vibration stimuli.
After finalizing the actuation, we see a need of developing a sensor to control the
compression force.
Different users have different forearm sizes and anatomies. Open-loop pressure
control method may result in different compression forces among different users.
A suitable grounding force is also critical to prevent the slippage of compression
actuators.
Therefore, we use soft sensors located on the ventral side of the forearm to get the
compression force feedback.
We choose capacitive sensing for its insensitivity over temperature change. A press
on the sensor results in a change in the gap between the electrodes, and accordingly
a change in the capacitance.
Based on this theory, we used a layered structure of dielectrics, electrodes and
insulators for the sensor. We tried various dielectrics and finally selected silicone
with microgrooves to tune the sensor sensitivity
to an appropriate range for our actuators. We also use conductive fabrics as the electrodes
and flexible insulators as the backings.
We used the same compression force testing setup as in the transient compression
tests to evaluate the sensor.
Results show that the sensor has more than 12% relative change in capacitance under
the force applied by the actuator, and exibits a small hysteresis. A linear fit is
sufficient to capture the variance of the data.
To show that the sensor feedback is consistent on different sized arms, We repeated
the test on cylinders with different circumferences.
The results indicate that the capacitance outputs are highly consistent, therefore,
we expect the sensor output does not change significantly
with varying arm sizes.
Recall the importance of compression force control, we further implemented a
closed-loop control strategy to use the feedback from the soft sensor for force
control.
Here is an overview of the hardware for Pneusleeve control system.
This system takes measured capacitance from the sensors into consideration and
adjusts commands to the pressure regulators to achieve desired compression force.
The control structure consists of an inner loop, an outer loop, and a feed-forward
term. The inner loop comes with the pressure regulator
for precise pressure control, which is faster than the outer loop.
The outer loop takes feedback from the sensor for error correction
The feedforward term uses a personalized user stiffness model to improve the
convergence speed of the controller.
Based on the feedback from the soft sensor, the feedforward model, together with
the integral controller, controls the pressure regulator input signal.
With the pressure change in the actuator, the compression force on user's arm can
be adjusted.
To evaluate the performance of the proposed control method, sinusoidal waveform
signals are used for tracking error evaluation and squared waveform signals are used
for response time evaluation.
Results show that the root-mean-square tracking error for the sinusoidal input was
0.10 N, and the settling time for squared wave input was less than 0.3 s.
After the mechanical evaluation on the Pnesleeve system, the next question is
whether people can feel the stimuli or not. To investigate on that, we choose to
measure absolute detection threshold to get the lowest level of stimuli for a person
to detect the feedback.
The results of absolute detection threshold show that thresholds are lower for the
compression actuators than for the skin stretch actuators, indicating that users are
more sensitive to compression actuators.
Next, we want to know if people can distinguish different levels of the stimuli, so that
we have references when designing multi-level haptic feedback.
Results of the just-noticeable difference study show a higher variability for the skin
stretch actuators,
which is consistent with the reduced sensitivity to those actuators in the ADT study.
It is important to note that the maximum values in both figures are about 40 kPa,
which is small comparing to the maximum range of the stimuli (200 kPa)
The above psychophysics studies established that all six actuators of PneuSleeve can
generate perceivable and distinguishable compression and skin stretch sensations.
As mentioned earlier, PneuSleeve can also generate vibrations.
Recall that multi-modal stimuli could potentially
enrich the expressiveness of the haptic messages being sent.
Therefore, we conducted subjective assessments using a vocabulary of 23 multi-
modal feel effects to show the expressiveness of pneusleeve.
For those who are not familiar with it, feel effect is defined as an explicit pairing
between a meaningful linguistic phrase and a rendered haptic pattern.
When designing the input signals, we think in terms of single actuator response
design and combined actuation design.
Here are some examples of the control signals we designed.
For example, for the acceleration feel effect, the 2 compression actuators operate at
a higher and higher frequency to express the message of acceleration.
The rotation cues use a sequential actuation of the skin stretch actuators to express
the rotation direction, and vibration is used to make the rotation more continuous.
We tried out many possible combinations and finally selected 23 feel effects grouped
into 7 families, including notification cues, physical objects simulations, navigation
cues, as well as some life experiences. These effects use combinations of
compression, skin stretch, and vibration. Force control is selectively used when skin
stretch actuators are involved.
We asked participants to rate on Goodness of fit, and goodness of feel of those
effects.
Goodness of Fit indicates how well the haptic sensations fit the name we give for the
feel effect. Goodness of Feel indicates how well the sensation feels or how pleasant
the sensation is.
The ratings are generally high for both of them. Some feel effects have relatively low
Goodness of Fit, such as spring, snake crawling and underwater, which some
participants indicate that they do not have enough real-life experience to compare
those effects with.
During the follow-up discussion after the rating, Multiple participants commented on
liking the soft sleeve form factor.
It is interesting that participants have different opinions on the compression feeling.
Some think that the squeezing is too strong, while some think it is perfect in order to
grab the user's attention.
Many participants have indicated the applications of these feel effects, such as
gaming, VR, and emotion induction.
There are still a lot of improvements we could do on PneuSleeve. For example, we
used air pressure ceiling for the testing in this paper. The portability of the device
could be improved by using a portable compressed air reservoir.
Multi-sensory integration with visual and audio cues could possibly enhance user's
immersive experiences.
As of noise concerns, our current setup uses an off-the-shelf muffler to lower down
the noise from the pressure regulators. Future work could use a customized muffler
and sound proofing foams to further reduce the noise.
Thanks again for watching this presentation. I would like to thank all colleges who
supported this research and thank all volunteers who participated in our user studies.
Thank you!