- Hello everyone, I'm Da-Yuan Huang,
and I'm here to talk about RetroShape.
Leveraging rear-surface shape displays
for 2.5D interaction on smartwatches.
I was involved in this project
when I was a post-doctoral at Dartmouth College.
This is the happy crew at Professor
Xing-Dong Yang's ex-discovery lab.
Recently become assistant professor
at National Taiwan University of Science and Technology
so if you come to Taiwan, please come to visit us.
I remember when I got my first smartwatch
I liked it a lot, it contains several useful
applications with sleek interface design.
I could even play Angry Birds on it
when I was on a bus or on a train.
At the time, it was a big deal to me.
So smartwatches provide quick access
to short-time entertainment applications,
however, my user experience dropped pretty fast.
Two weeks later, I just forgot my smartwatches
and continued using my iPhone.
I thought about the reason, and believed that
the major issue came from the sizes
between phones and watches.
Compared to phones, the screen size of watches
is small, which limits both of the input
and output capabilities.
Speaking of the screen size, I once thought
maybe in the near future, like the increasing
screen sizes of smartphones, the smartwatches
will equip bigger touchscreens.
However, the market proved me wrong again.
So many smartwatches were introduced
after then, but their screen sizes are
pretty much the same.
I think that is because we want
our smartwatches to be small.
Since the smartwatch is a long-existing
wrist-worn accessory on our wrists.
Most importantly, we want them to stay
small and unobtrusive while not in use.
Although these kinds of designs are really cool,
most of the users may not be ready to wear
this smartwatch constantly, or recognize this
as a smartwatch.
So now, imagine with bigger screen resolution
and ideal feedback, the entertainment experience
has been improved a little bit.
However, now imagine there is a future smartwatch
which offers a super satisfactory entertainment experience.
But, as a wrist-worn device, it's phone-factor
and the size should be remained.
What can we do to progress towards
these ideal smartwatches?
We proposed generating haptic feedback
on the back of the smartwatch.
It's not simply applying vibrotactile feedback
but generating collocated visual-haptic feedback.
For example, you see an explosion on your wrist,
you should sense the explosion wave on your wrist as well.
The visual and haptic feedback should be
coupled with each other.
So in brief, our strategy is to utilize
haptic feedback to amend the
insufficiency of visual feedback,
and further render realistic physical properties
on your wrist, as if the virtual objects
are extended to the physical world.
So each pixel on the screen has a
corresponding tactile pixel or taxel
on the back of the watch face.
Users can feel the virtual objects
and see the objects at the same time.
This project was inspired by many research works,
for example, vibrotactile feedback is a common
cutaneous haptic stimulation in the
VI environment or in game controllers.
For more advanced effects, previous works
propose rendering moving stroke on the back of the chair
to enhance multimedia experience.
However, more often, researchers utilize
vibrotactile feedback for delivering messages,
for example, rendering stroke or character best
patterns on the back of the smartwatch
or across the whole body.
Force feedback is also a popular haptic channel
in VI environment, for example, rendering
mechanical force feedback on the fingertip,
or using ultrasonic haptic array
or an air compressor to generate
mid-air force feedback, or generating skin jerk
sensation on the wrist or on the finger segments.
However as you can see, force feedback
on the smartwatch platform for entertaining
applications, has rarely been explored.
Various types of shape-changing display
have been proposed, for example,
line shape displays can be a good addition
for motor or physical assistance.
We are especially inspired by the pin-based
shape changing display at different scales
as they have been proposed for extending
digital information from 2D to 2.5D for many years.
For example, MIT media lab developed inForm,
a large-scale shape changing display
allows users to interact with visual
and haptic information at the same time.
On the other hand, researchers at Microsoft
Research built a mobile-version shape display
to enhance the haptic feedback in VI environments.
Inspired by these amazing works,
we developed RetroShape, a small-sized
shape display placed on the back of the smartwatch.
So, RetroShape is composed of a 4x4
actuated pin array, containing 3D printed
taxel with no space between them.
As you can see, our display adopts the modular design,
each small module is composed of a 2x2 pin array.
A taxel is connected to a rod
and linearly actuated to a bio-miniature servo motor.
The four motors are in a stack,
so to ensure that they occupied a minimum horizontal space.
The servo motors and the rods are placed
in an interleaved manner so each taxel
can be driven independently.
In order to hold the taxels together,
we add the constraint on the top,
and then acrylic glass watch case
surrounding the taxels, and this is how it works.
And finally, the watch case holds a two-inch
TFT watch display.
Although the prototype is thicker
than the normal smartwatch, it can be worn
on the wrist comfortably.
This is how it works in the end.
So in this work, we discussed several
physical properties that RetroShape can do,
including location,
motion,
size and number,
and the material of the virtual objects,
in addition shape and pressure can be rendered
by multiple pins or different height of the pins.
With this prototype, we now need an
authoring environment to help users
create an at-ease shape-changing tactile feedback
for videos and games, and use these to create
our RetroShape demo applications.
So, we first designed the pre-made effects.
Pre-made effects can be created using animated GI files.
The pixel intensity stands for the height of the pin.
However, it's resolution is higher than
the resolution of our prototype,
so we applied a simple pixelated algorithm
to partition the screen into a 4x4 grid layout.
We implemented an interface to make the
pre-made effects easily used in games,
for example, when a ball hits the ground
the game triggers an event, and the processing
program allows the pre-made effect
and around the pixelated algorithm.
Finally, through an Arduino board,
the distance and the speed of the pins
can be dynamically controlled.
This is another example for our
first-person shooting game.
And we implemented three games
on our RetroShape prototype, including the
first-person shooting game,
a space shooter game,
and finally a whack-a-mole game.
So, these are games.
We also implemented a haptic video editor.
The editor is composed of two views,
a key frame editor view, and a taxel view.
The user can either drag and drop a pre-made effect
to the desired location in the key frame editor view,
or just adjust the height of the pin in the taxel view.
Users can select the taxel they want
and use the slider to specify the pin height.
The authoring result is stored in an excel file
and can be reloaded while playing the video.
This is the example of our authoring result.
The movie clips from Kung Fu Panda.
This is how it looks on the RetroShape smartwatch.
So, we designed rich haptic effects
with this editor, including increasing pressure,
linear explosion,
random explosion,
circular explosion, and the vortex movement.
Right now you may wonder, hey wait,
the resolution of our RetroShape is only 4x4,
it is not a 100% visual haptic feedback.
True, ideally the resolution of visual
and haptic displays should be the same,
as I mentioned before.
However, we noticed that there is a trade off
between the resolution and engineering effort,
especially for a smartwatch platform.
So, during the design of RetroShape,
we began to wonder, can a low-resolution
shape display still keep coherent illusion?
The problem soon became, can a low-resolution
haptic display, not shatter the illusion?
We are especially interested in the coherent
illusion of the sizes between visual
and haptic feedback, and the distance
between visual and haptic feedback,
because these two factors directly effect
the least resolution of our prototype.
To understand these questions,
we implemented a robotic arm
that can render a tactile feedback
on the desired position of the participants' wrist.
As you can see, a smartphone was mounted
above the taxel to provide visual feedback
and rendered a mock-up smartwatch screen.
The robotic arm holds a 3D printed taxel
and a civil motor actuated that taxel
to the user's wrist.
So we developed a tracking system,
a second smartphone face is downwards
and it is in contact with a stylus tip
right above the taxel.
The stylus touchpoints represents
where the taxel was located,
which was used to determine the location
of the visual stimulators on the top of the screen.
This is the self-calibration process.
So with this robotic arm,
when the participants see a visual object
on the screen, the robotic arm can render
corresponding haptic feedback onto
the participants wrist.
So we use these apparatus for the following studies.
For the size study, we modified a regular JND study.
Sorry for skipping some details,
but the idea is that we render haptic feedback
on the wrist and ask the participants
if the size of the visual object
fits the size of the haptic feedback.
Through this manner we can gather ratio
of visual haptic feedback that provides coherent illusion.
In a study we examined three taxel sizes,
10mm pin, 6mm pin, and 2mm pin.
For each trial, we gave the participants
a reference, where the visual and haptic stimuli
are in the same size.
Since we found participants did not contain
any prior knowledge of an equal size
visual haptic feedback, for the upper part
of the visual size we provided a visual stimulus
larger than the haptic stimulus,
and asked participants if they agree
the visual haptic stimuli were coherent with each other.
If yes, we enlarged the size of the visual stimulus,
if no, we reduced the size.
We followed a one-up one-down staircase method
and averaged the last five values as the results,
we applied the same procedure for the lower bounds.
So this is the result.
The results suggest that a taxel
can stimulate an object that is bigger or smaller
than it's actual size without shattering the illusion.
For example, a 2mm taxel can stimulate
an object as small as 1.3mm or as large as 2.9mm.
These results also suggest the limitation
of low-resolution shape-changing display.
As for the distance study, the distance between
visual and haptic stimuli kept changing.
Participants were asked to report
if they think the visual haptic feedback
was co-located or not.
Through this manner, we can get a
discrimination threshold of location.
So, for the reference, a visual and tactile stimuli
fired in the same location, for the test trial,
they were set a certain distance apart.
The reference and test props were displaced
in the rendering order, and the participants
need to instigate which one is the prop.
If the answer was correct, we reduced the distance,
otherwise we increased it.
So the procedure follows one-up two-down
adaptive staircase method.
This is the result.
Surprisingly as you can see,
taxel displays are able to stimulate
a realistic experience within a large
location different thresholds.
This suggests that even when visual
and tactile stimuli are not aligned,
they are still perceived as co-located
as long as the distance between then
is kept within a suggested threshold.
So, our RetroShape prototype was actually made
based on the study results.
We aim to explore more factors in the near future.
So, with the prototype and the study results,
we would like to know if haptic feedback
rendered by RetroShape prototype
is a valuable addition or not.
Our goals are two-fold, we examined the realism
and the enjoyment made by our RetroShape prototype.
For the realism, participants had to
compare a vibrotactile system on the RetroShape prototype.
We implemented a vibrotactile system
using a 10mm EIM vibration motor,
which is commonly used in previous works,
and for the configuration also followed those works.
During the comparison, the participants
saw a bouncing ball animation on their wrists,
and we rendered the size and pressure of the ball
using either shape-changing tactile feedback,
or vibrotactile feedback.
For different sizes, we used one and four taxels.
For different pressures, we used different levels
of pin heights or vibration frequency.
But overall, RetroShape prototype
received a significantly higher score
than a vibrotactile feedback.
The participants commented that
it is so cool and I felt the ball was actually
sinking into the skin.
In contrast, most of the participants
rated the vibrotactile feedback below three
as weight has nothing to do with the vibration.
The second part measured user enjoyment of RetroShape
in comparison to a tactile-free smartwatch.
So participants had to experience our
demo applications using RetroShape.
For comparison, they then used applications
without any tactile feedback,
it's a visual-only smartwatch.
So, participants rated shape-changing
tactile display, having no tactile feedback
for most of the games and videos.
Interestingly, the tactile version
space shooter game did not receive higher scores,
since the participants thought the visual
and haptic feedback do not match
the close explosion very well.
This is a very useful finding because
it shows that mismatch between visual
and haptic stimuli can largely mitigate user experience.
The possible reason for this mismatch
is that the resolution of the taxel display
was too low to render the fine grand details
in the exploding scene,
but overall they enjoyed our prototype a lot.
Right now let's go back to the very first question,
how can we progress toward an ideal smart watch
that offers super satisfactory multimedia experience?
In this project, we extended the interaction region
from the watch face to the back of the watch.
The user studies informed us what is
the proper resolution of the haptic display
for coherent visual and haptic feedback.
But, I don't know, maybe our imagination
of the future smartwatch is incorrect,
maybe in the future, a huge smartwatch
on your wrist is not a big deal at all,
or maybe the phone factor is not a watch anymore.
However, this work, what we would like to do,
is to demonstrate a possibility,
a possibility of not changing the phone factor
and the size of the smartwatch,
but still providing satisfactory output experience
on the smartwatch platform, and our current answer
is to extend the interaction region
from watch surface to our wrist skin.
Thank you for your listening,
and I'm happy to take your questions.
(applause)
- [Audience Member] Hello, thank you for the talk
it was very impressive.
Jeremy Warner from UC Berkeley.
So I was wondering, in the visualization
for the overlay, you used a second smartphone
for tracking, what was the motivation for that?
Couldn't you have just have used
a simple capacitative sensing pattern
or something, why use another smartphone?
- You mean why we need a tracking system
on the smartphone?
- [Audience Member] Yeah, for the tracking
on the other side.
- Because the reason is we have to
keep changing the stimulus positions,
because if you continue to offer stimulus
on the same location, the user will get
used to it, and finally they cannot
differentiate different stimuli,
so we have to keep changing the position of the stimuli.
However, to do that we need a good tracking system
that can provide the tactile stimulus
on the desired position of the participants' wrist,
and that's the reason why.
- [Audience Member] Okay, thank you.
- Thank you very much.