Hi, my name is Romain Nith and I'm here to present JumpMod,
done in collaboration with my colleagues, Jacob Serfaty, Sam
Shatzkin, Alan Shen, and my advisor Pedro Lopes at the
University of Chicago. Here, Alan is wearing our haptic
backpack. Let me show you what he feels when he jumps with it.
"Wow, it feels like I... jumped higher!"
So he felt like he jumped higher - well hold on, there were no
cables or external structures, helping him jump. He also didn't
seem to have physically jumped higher either. So how does our
backpack with no means of pushing him higher, actually
gave him the illusion of jumping higher?
Well, let's rewind and take a closer look. As Alan jumped the
backpack move a two kilogram weight upwards. It produces just
enough force to feel a vertical momentum, but not enough to make
him feel physically jump higher. And as such, we encourage you to
think about this as a perceptual effect more akin to an illusion.
By moving the weight up or down at different phases, we can
actually achieve five different sensations. For example, the
sense of jumping higher, but also the sense of being pulled
up or down by an external force, such as someone tugging you with
a rope, or even the sense of landing with a hard or soft
impact on the ground.
So why did we build his backpack. Well gesture input,
were completely untethered and ungrounded, they work very well
in freely moving VR. Alan here didn't jump high enough and
knocked the cow over. So he collects his power up and jumps
higher over the next one. Success! Now he's about to jump
even the full body as you saw with Alan's jumping in VR, has
over a haystack, but the wind will blow him down and mid-jump,
creating the sense of being pulled lower. Next, there's a
puddle of mud. By jumping into it, he will feel a sense of
landing softer. And finally, there's a pumpkin up ahead, Alan
will jump on it, crush it, creating the sensation of
landing harder.
become very popular in the last decade. As such, researchers
have been actively closing the loop by adding output sensations
to match the input commands.
For example, handheld controllers can emulate the
feeling of shaking a virtual ice cube in cup. Hand exoskeletons
allow us to render the shape of the virtual cube and manipulate
it with our hands. Similarly, electrical muscle stimulation
have been used to render the cube's weight in addition to its
shape, and other researchers have created shoes to step on
these virtual objects.
What about jumping? Well, turns out the vertical force feedback
is extremely rare in mainstream interactive experiences. Because
a device capable of moving would need a lot more power, mass and
battery than all the ones that you see on the screen. So let's
see how researchers add realism to vertical experiences.
Researchers use grounded devices such as motorized pulleys. To
suspend the use in mid-air, they need to have an entire
infrastructure dedicated for the device, making it not very
portable. Another popular approach is to use powerful
propellers to push air away from the user, generating forces.
While some devices using this approach are hands-free, others
require the need to hold it, which can greatly limit users'
movements. Additionally, most are not powerful enough to lift
the users off the ground. But the ones that do require
powerful hardware, which comes at the cost of weight. Augmented
Jumps, for example, weighs 60 kilograms by itself and can only
produce a 25% sense reduced gravity. Plus, it creates quite
a bit of noise which can easily break the immersion. So it is
not surprising that researchers have been searching for a
smaller way to achieve these vertical effects. For instance,
by altering what the user sees in VR, they can create the sense
of jumping higher. While these redirected movements are very
effective, they only work in VR, and not in all vertical
experiences. Moreover, we know that real haptics tend to
outperform pseudo haptics so we decided to explore this in a new
way.
Our approach is to find a force-based illusion that
requires a much smaller actuator, and yet still effects
someone's sense of jumping. And this is exactly what we did with
JumpMod. We implemented our backpack with a motor driving a
belt, which moves a two kilogram weight, also doubling as our
battery along a linear rail. The belt tensioner ensures a good
power transmission from the motor to the weight. We also
keep track of the position of the weight with an encoder and
limit switches located at both ends of the rail. Finally,
everything has been controlled with a microcontroller equipped
with Bluetooth and a motor controller.
In addition to two traditional shoulder straps of a backpack,
we further secure JumpMod to the user using a sternum strap and
two leg straps. To better understand our backpack, we
conducted a perceptual study in extreme conditions: participants
could not hear nor feel the motor running. Participants were
asked to jump and describe the sensation that they felt. We
studied our backpack by actuating at different phases of
a jump in different directions. And this is what we found. Even
in the most extreme conditions, participants were able to
distinguish five different effects. Note that some effects
can be replicated with different jump stimuli. Now, do we really
need JumpMod? Let us compare to the popular visual illusions
used in VR. The goal of this study was to understand if
realism induced by JumpMod outperforms the one used in
visual illusions. So, we designed the VR experience to
evaluate exactly this, the same one that you saw at the
beginning of the talk. And we found out that immersion is
superior with JumpMod compared to the baseline, as well as
realism. And this is quite exciting as it suggested that
JumpMod did add value to the experience, and most
participants prefer the JumpMod condition, despite the added
weight and sound. So far, I've shown you JumpMod, working in a
VR game, in which you have to jump in place. Let me show you
more advanced experiences, some even go beyond VR. JumpMod can
be used as feedback guidance. For example, learning how to
jump rope. This is different from the previous experience, as
this does not render forces for the sake of realism, but rather
the backpack informs the users when to jump. It can also be
used for other VR experiences emphasizing on exploring large
spaces, for example, playing a VR escape room game. Here Alan
is trying to reach for a key stuck on the chandelier right
above him, which is out of reach. So he uses this
trampoline on the floor to help him jump higher. He knocks the
key out of the chandelier, grabs it, and now he's able to escape.
Since JumpMod is completely ungrounded, it can be used to
play an augmented game of basketball. Here's Shan-Yuan is
trying to defend against Alan and sends a random haptic
effect. But it seems like Alan got the "jump higher" boost and
actually managed to score. To conclude, looking back at
previous works, these ungrounded devices allow rendering similar
types of sensations as JumpMod. However, JumpMod achieves it by
using perceptual illusion and hence is able to miniaturize the
hardware. As such in this graph, haptic expressivity versus
device size, we can put JumpMod right here. If we continue to
populate this graph, we have at the very bottom, the handheld
controllers using linear resonant actuators. While these
vibrations are very high resolution, it does not provide
enough force to move an entire limb. On the other side, we have
grounded devices using powerful motors to be able to lift the
user. We can actually see an exponential curve covering the
trend of these devices. But here with JumpMod, we decreased the
device form factor while still keeping the same level of haptic
expressivity. In a previous project, we shrunk the overall
form factor of the hand exoskeleton by using electrical
muscle stimulation as the main actuator and a brake-based
exoskeleton, reducing the actual form factor and weight to 68 grams.
My research is to argue that there's a way to make devices
using perceptual, as well as engineering tricks to get more
linear trend. This will allow us to enjoy the same level of
haptic expressivity, but in a much smaller form factor.
My mission is to keep populating this plot by designing body
scale haptic devices, while keeping the form factor at body
scale. Body scale haptic is important because it will first
enable force feedback in highly mobile VR and AR experiences
that can be enjoyed at home for instance, but also enable
completely new interactive domains that we've never seen
before. We will be able to carry physical assistants around all
day long, such as this fingerspelling assistant that I
showed at UIST 2021 or even delivering haptic effects in
Today I showed you JumpMod, a haptic backpack that alters a
augmented sports.
user's perceived jump. JumpMod achieves this by moving a weight
along the user's back, which causes an inertial force that
modulates the user's perception of their own vertical momentum
while completely being untethered and hands-free. Thank
you for listening. Please check out our paper for more details.