Hi. My name is Jom. I'm currently a PhD student at the University of Washington - and collaborator at Facebook Reality Labs.
Today I'd like to talk to you about our paper for Chasm, a simple, yet expressive compact haptic actuator based on a screw.
This paper was co-authored by my mentor Ali Israr of Facebook Reality Labs and Majed Samad.
So - imagine yourself in a Virtual or Augmented Reality setting
As you roam around in VR, your sense of locomotion is thrown out of whack.
You feel no inertia in each step you make and simply glide around.
As you handle and touch objects - the most you can feel are slight vibrations to the hand
- that only represent impact or textures. Its hard to judge the stiffness or weight of anything around you.
So what are you actually supposed to feel?
Well, if you peek inside your skin,
you'll find four mechanoreceptors - that are responsible for feeling different 4 different signals.
The Meissner and Pacinian Corpuscle and the Merkle Cells and Ruffini Endings.
Each mechanoreceptor is sensitive to different bands of frequencies.
The Meissner and Pacinian Corpuscles
are sensitive light touch and high frequency effects, such as impacts and vibrations from 10 to 1000 Hz.
While the Merkle Cells and Ruffini Endings
are sensitive to shear, pressure and lower frequency vibrations in the 0.5 to 100 Hz range.
All four cells combined are responsible for the sensation of your skin,
allowing you to feel textures, grab and manipulate objects, and respond to external forces.
So where are we in terms of stimulating these organs?
Commercially available haptic devices - like the Linear Resonant Actuator or LRA,
found in most modern phones and game controllers, can only render a small band in the hundreds of Hz range.
It's improved from its predecessor, the Eccentric Rotating Mass or ERM, by only a tiny margin in terms of bandwidth.
Larger ERMs can produce lower frequencies too, but again, at a very narrow bandwidth.
Piezoelectric haptic actuators have a wider bandwidth - but can only provide miniscule displacements
- in the order of microns
The Lofelt L5 operates similarly to the LRA - and can render an even wider range
but as you can see, it does not cover anything below 40 Hz.
So far, shear and pressure found in the lower frequency band
– important for the perception of mass and force,
is completely missing from the picture.
Before looking at the research landscape, I'd like to put our friend - the LRA – back on this graph.
From 2009, Kim and colleagues presented a triplet of haptic actuators
that can render both constant shear and high frequency vibrations using a complex 6-bar linkage.
In 2017, Schorr and colleagues presented a 3 degree of freedom haptic device that can be worn on the fingertips
the device could also render shear and low frequency vibrations in 3 axes in a compact formfactor.
In 2018 and 19, Mintchev and colleagues presented the Foldaway,
a lightweight 3 degree of freedom haptic device designed for tactile input that could render at an even larger bandwidth.
In 2019, Sreetharan and colleagues introduced a compact haptic actuator using a leadscrew and a miniature stepper motor as the primary mechanism.
Although achieving a tiny form-factor – performance worsened significantly while under load - due to the open-loop control.
Our work, Chasm, presents a good compromise between compactness and bandwidth,
reaching into a wide band of perceivable frequency - including constant pressure or shear with a straightforward mechanism.
Now, we're well aware that we're not comparing apples to apples here,
but there aren't that many papers that publish these details!
We're just trying to frame where Chasm sits in the haptic device landscape.
Now, moving on to the design
Chasm at its core is very simple. It boils down to 3 components.
The first is a DC micromotor.
The second is a leadscrew. Leadscrews convert rotation to linear movement, just like how a car jack works.
The third component, the nut serves as the interface to the skin.
As the leadscrew rotates, the nut travels up and down along the length of the screw.
A few other components make up the rest of Chasm.
A magnetic encoder is used to track the position of the nut - and a motor driver provides power to the motor.
We use a Faulhaber 1016 micromotor and a 2 mm custom leadscrew in our prototype.
All the components combined gives us the following characteristics:
Chasm can render constant shear and vibrations perceivable up to 170 Hz - both independently and simultaneously.
It has a 3.4 mm of travel
And 4.8 N of shear force.
Chasm is about the size of a triple A battery and weighs about 15 grams.
Chasm consumes less than 0.6 Watts of power during normal usage - and up to 2.7 Watts in bursts.
Let me explain how we got these numbers.
We ran Chasm through a number of engineering validation tests.
First, we validated the output of the encoder against a laser doppler vibrometer.
From the plots, the encoder has an average tracking error of about 100 microns, which is acceptable given our use case.
From here on out, we will use the encoder as ground truth.
Now without a load, and ordering a step command,
which is a command to move the nut from on end to the other instantaneously,
Chasm has an average rise time of 12.9 milliseconds and a steady-state error of 40 microns.
The simulation on the left is from recorded data, shown in real-time to show how fast Chasm can get.
The white interface is the intended location while the blue interface is the actual position.
Under a 3.6 Newton normal load on the interface, which was determined to be the typical loading condition,
Chasm still performs well with an error of 150 microns.
To test the maximum output force, we created a jig to hold Chasm vertically
and an attachment for Chasm to push against a capacitive force sensor.
Our 4.8 Newtons of shear force comes from this experiment.
Here, we play a sweeping sine wave from 0.5 to 40 Hz in both the unloaded and loaded conditions.
Chasm maintains its full bandwidth up to about 20 Hz.
And Chasm can still render vibrations higher than 20 Hz at a depreciated amplitude.
We'll later confirm that these higher frequencies are still perceivable.
Here's a slow motion version of the same video.
Now that you know what Chasm can do, let's walk through how we control Chasm.
Chasm's end effector or nut position is controlled using a thousand Hertz PI control loop - through a microcontroller.
This gives up ample room for rendering high fidelity haptics - as you'll see later.
To talk to the microcontroller, we opted to use USB HID.
USB HID has two major benefits:
1. It works with almost every device without drivers, just plug-and-play out of the box.
2. HID has a typical latency of 1 millisecond. Greatly reducing the guesswork of latency during haptic rendering.
This means Chasm's microcontroller can readily connect to many different systems,
such a PC running Windows, macOS or Linux, or Android devices such as smartphones and the Oculus Quest.
During development, users can prototype haptic effects on a PC.
In this paper, we use Unity's extensible editor to author experiences.
Through a USB HID plugin, Unity communicates with Chasm's microcontroller to send low-latency haptic commands.
Such commands include:
Stepping instantly to a position. Useful for streaming haptic textures to Chasm in real-time.
Linearly interpolating to a position. Useful for rendering shear forces.
And Chasm can also render arbitrary sine waves at any time on top of any existing command,
for responsive vibrations and impacts.
Chasm also supports arbitrary waveforms created using Unity's AnimationCurve tool
enabling authors to create unique sensations using a graphical interface.
Combining those techniques together can create extremely convincing haptics in a virtual setting.
Now let's go through some prototype use cases we had thought of.
Firstly, since Chasm has a simple mechanical design, its design space is quite flexible.
For instance, we can fold Chasm's leadscrew - using a pair of gears - to create a shorter but wider formfactor.
We can also place the leadscrew at a right-angle using bevel gears.
And most importantly, we can scale Chasm up and down in size.
For example, we can miniaturize Chasm further by using a smaller motor and trading output force for size.
Here, we can place the smaller Chasm units in the housing of a smartphone for creating distributed haptic effects.
We can also embed Chasm in game controllers for creating force feedback.
A slightly modified version of Chasm - with a capacitive sensor inline
can also be used to create force feedback for triggers in controllers.
Chasm can also be used to render height and textures in a pen by either modulating the height or friction.
In this paper, we investigated two main use cases for Chasm.
Our first prototype is a headgear with Chasm on adjacent sides of the temple.
Users would wear a VR headset on top of the headband - and Chasm can render shear forces and vibrations on the temple.
One use is to induce forwards and backwards motion cues.
Another use is to render shear forces to nudge users during navigation - and playing alternating taps while walking.
Our second prototype is a handheld marker for enhancing interaction in AR and VR.
Users can feel shear force and texture rendered underneath their fingerpad.
Coupled with a compelling visuals, Chasm completes the sensory experience of feeling realistic weight, forces and textures.
We ran some perceptual studies to validate our two prototypes, let's go through them.
We first ran a pilot study to determine how high of a frequency Chasm can actually render
We measured this on both the temple area and finger pad on the thumb.
I won't go into detail here, but you can read about this experiment in our paper.
From the results, we can see that despite the significant attenuation, a
participants can still feel vibrations up to 170 Hz on the finger.
Now let's look at some perceptual studies we ran with the headgear prototype.
Our first study was simple,
we wanted to see if rendering forwards and backward shear on both sides of the temple would induce locomotion cues.
To do this, participants wore a VR headset on top of the headband
and were subjected to an accelerating or decelerating starfield.
During acceleration, both actuators would move backwards
and during DECELERATION*, both actuators would move forwards.
However, as our stroke was limited to 3.4 mm,
we found that some participants were confused by the cues - and could not determine the direction.
In the future, we want to re-visit this experiment with a longer-stroke prototype.
Next, we placed participants in a maze-like environment and gave them haptic cues to navigate with.
Their objective was to reach a pre-designated door as fast as possible.
They used the joystick on the controller to move around.
We gave them three types of haptic cues
One - A cue on the left of the head moving forwards, suggesting them to turn to the right.
Two - An opposite shear cue on the right side, suggesting them to turn to the left.
And Three - A cue where both actuators move forward, suggesting them that this is the wrong way and turn back.
Participants were not taught these cues - and learned them on their own.
Most participants reported that the navigational cues were intuitive and useful,
- and completed the maze faster than without haptic cues, as expected.
We also had a bonus experiment giving participants ”haptic gaiting”
where Chasm renders alternating taps as the user moves around, similar in cadence to walking.
Although from our survey there was no significant effect for nausea,
participants reported through interviews that the haptic cues were pleasant and perhaps masking their peripheral vision from motion.
We also hope to investigate more in the future.
Now, let's move on to the marker prototype.
Before I talk about the perceptual studies, I need to explain how we achieved our haptics and visual pseudo-haptics.
The blue marker and hand represent the actual position of the user's hand – or the "god object" in some similar models,
while the black marker is what actually is seen in VR.
On the right is a big red button.
A linear spring connects the virtual marker with the actual marker with a stiffness of K_M.
The button, or any object with stiffness in our VR environment, has a stiffness of K_B.
As the marker comes into contact with the button,
the virtual marker maintains constant contact with the button, as forces are in equilibrium.
As the user pushes forward, because of the difference in stiffness, the marker and button moves different amounts.
The Control Display ratio can be calculated from this distance.
A higher ratio suggests higher stiffness – requiring more force exertion – requiring the user to push their arm more forward than seen.
A proportion of the displacement is also used as the amount of shear rendered on Chasm.
And since our model relies on Unity's physics engine - it can be readily used to interact with other rigid bodies in Unity.
Here's a short clip of the effect in use.
As you can see, the user needs to exert more movement for stiffer buttons and Chasm renders higher displacement.
And this is what the user would actually see.
Our perceptual study is quite simple,
we had thirteen participants come in to press two buttons repeatedly.
One button was kept at a constant stiffness, while we varied the other one.
We tested three conditions:
1. Using only the pseudo-haptics
2. Haptics only, where the pseudo-haptic illusion is turned off
And 3. Both pseudo-haptics and haptics together.
Our results suggest that there is indeed visual-haptic integration -
and users were definitely able to discriminate stiffness better with haptics - than with visuals alone.
Finally, here's a brief preview of our Demo.
As you can see, Chasm is constantly rendering both shear and vibrations to simulate friction while writing or drawing.
Here, due to the tight integration with Unity and its fast response,
Chasm can instantly respond to texture changes, switching between high and low frequencies, along with shear.
Here, using haptic map, where brightness indicates height,
we can encode haptic textures that correspond to what the user sees without any custom programming.
Users can also get a higher sense of the presence
by rendering impact and shear upon contact with physical objects in the virtual world.
Using custom authored waveforms,
realistic effects can be rendered that respond to discrete events. Like the fabric snapping away here.
And lastly, coupled with the pseudo-haptic weight illusion,
Chasm renders an extremely convincing sense of mass when interacting with objects.
Note that these blocks are simply rigid bodies in Unity. As our haptic rendering scheme takes care of both the illusion and haptic rendering automatically.
To summarize: Chasm is a broadband haptic actuator that can render both shear forces and vibrations both together and independently.
Chasm has few components, so its scalable, and has multiple use cases for AR/VR and beyond.
Chasm has robust software integration with low latency which means haptics can be extremely responsive to user input.
Thank you for your time! And please come try our demo at an HCI conference, soon?