- I'm Jotaro.
- And we'd like to start with some gameplay footage.
(mechanical noises)
Now this gameplay footage would be completely unremarkable,
except for the fact that the person playing it is blind.
(mechanical noises)
This is Eddis using a haptic device to be able
to play this game.
The device he is using is called DualPanto.
Now current displays for blind users, for video games,
might look something more like this.
This is an audio game that might run on your smart phone.
And what's gonna happen is a little target, little cursor,
is gonna start sliding across,
and the volume will get louder
as it approaches the bullseye.
And you tap on the screen to be able to shoot.
- [Game Voice] You've got one shot left.
(insect chatter)
- You've got seven points.
I think you need more practice, my dear bowman.
- So when I said typical, I actually meant
an award winning audio game.
This is probably pretty good for what's out there.
But it still reduces space to scanning.
The users have to wait for that target
to cross the entire space, to cross the space.
So what happens when we actually use spatial interaction?
So this is Terry.
He's a YouTuber.
And on his channel, he's playing through
the Legend of Zelda Ocarina of Time.
He's also blind.
His setup has two speakers which give him spatial audio
and this is critical, he's playing this in a virtual machine
which allows him to save the state and then restore
that state when he's playing the game.
And he's gonna try and attempt a very targeted jump.
- Now this is my challenge I have.
I have to try to figure out where to jump still.
And it's kind of a trial and error thing
where the safe state comes in really handy
so I'm gonna try it here.
(character screams)
Nope.
(character screams)
- So unfortunately he doesn't have the information available
to be able to track this target.
And this target is static.
This is probably an easier condition.
It gets even harder when objects move.
When you only have audio, the player might get stuck
because they can't have that information
about the environment or about where their target is
and how their target is moving.
And so when he gets later on in the game,
he runs into this situation. (suspenseful music)
He saves.
- [Terry] Save.
Tried to get outta the way.
- And unfortunately it was a very poorly timed save
and it takes him a little while to get out of the situation.
He just doesn't have the information
to track these moving objects.
Now most solutions would typically use sound
or vibration as sonar.
You would point towards an enemy, or you kind of listen
for that volume to increase or decrease
or pitch to go up or down, maybe fiber tactile feedback.
But users still need to scan or wait.
They need to look around in the room or they need
to wait for that feedback.
We have a different approach.
Which we call DualPanto.
DualPanto is a haptic device that enables blind users to
continuously interact in virtual worlds.
It consists of two parts.
There's a me handle which lets users act
in their environment.
And an it handle which shows users moving objects
in the world.
What I'd like to do is give you a demo.
Hopefully you've checked out our demo
at the demo session yesterday.
But if you missed it, we can do that right now.
So what I need everyone to do
is please shake your neighbor's hand.
Everyone, come on do it, let's do it.
Now hold the hand.
You're going to just keep holding your hands
for a while, okay?
We're get nice and cozy.
Everyone on my right is going to draw a shape,
like a circle, a square, a triangle,
whatever you wanna do.
But you're both going to close your eyes, okay?
The person on the left,
you're just gonna let them lead you around, okay?
So three, two, one.
Close your eyes and draw.
(audience chatter)
Okay, people on the left, were you able
to figure out what shape it was?
Yes?
No?
Mostly so?
Yeah.
(audience chatter)
So you just experienced a single Panto.
Which gave you a continuous spacial interaction
with your neighbors without sight.
And no other game for visually impaired players does this.
- Alright.
Let me explain how it works.
So the system consists of two main handles,
main components, so it's called me handle and it handle.
So let me take an example of a soccer game,
and explain how me handle works.
Okay.
And then user can grab the me handle as they move around.
This will also move a soccer player on a virtual field.
The soccer is a virtual area rolled in the soccer field.
And user can also turn virtual avatar there
by rotating a handle.
But since this is a haptic device,
the user can also feel the virtual environment,
like a wall or obstacle.
Okay, so how about when we encounter moving objects
in the virtual world.
And that says it handle can do the job.
In this example, a soccer ball.
So the it handle, the system will actuate the it handle
to show the position of the soccer ball on the field.
And then our user can hold the it handle and then know
and sense the position of the soccer ball at the moment.
Just like this.
The key idea is here.
A me handle and it handle has the same shared workspace.
And that says you can buy all these things.
User can approach the soccer ball to take it.
Just like this.
(thump)
And give a kick.
(thump)
And do it again.
(thump)
And finally
(thump)
(ball hitting net)
score a goal.
So the key idea is here.
To DualPanto, the DualPanto user can act in a virtual world.
But at the same time, user can sense,
or the system can provide information to the user
at the same time.
So this enables blind users, it continues
in spatial interaction without any necessity of seeing
or waiting or scanning the virtual world.
So I would like to explain the hardware setup.
The key about this that we used two handles,
but still, we need a haptic device.
In this project, we started
from the haptic pantograph design as a starting point.
Just look, the device looks like this.
And then, this is our prototype.
And then the haptic device has to be,
in order to give a precise haptic rendering,
the linkage just has to be really strong and rigid,
but also smooth.
So we used aluminum linkages
and we also used Teflon washers
on the joints, on the linkages.
For rotation, we used a small motor with encoders
with a low gear ratio so the user can easily turn it on.
On the bottom it handle, the system to actuate the rotation
of the handle to let users know what objects it's facing.
So we used a higher gear ratio than the top handle,
or the me handle.
A higher gear ratio of the motor encoder.
So what about the shape of the handle?
So we actually tried a bunch of,
several handle shapes and sizes.
And our final design yields into this shape,
the teardrop shape, which allows users
to know where the handle is pointing
between different finger postures.
So with haptic rendering techniques,
we utilized existing haptic rendering techniques.
For example, for me handles,
we used god object techniques, which allows
to render walls or curvatures, polygons in the virtual world
that are complex-shaped.
Also we used virtual coupling techniques in order
to render our object
which is away from the end of the me handle.
For it handle, we used PID controllers
for position rendering and also for rotation rendering.
- So to evaluate DualPanto, we recruited six participants,
all of whom were visually impaired, two from birth,
to use the device.
We introduced them to the device and to the soccer game,
and this process took about eight to 15 minutes.
After training, each participant played a match,
and then they provided their feedback.
All participants were able to play the game,
and they reported very high enjoyment, so 6.5 out of 7.
(repetitive bumping)
(loud mumble)
(laughter)
(clicking)
- [Background Voice] Ah!
(laughter)
(clicking)
- [Background Voice] Oh, come on.
(laughter)
- That wasn't planned, by the way.
So we did some video and interview analysis
to understand what the experience was like
to use the device, and we came up with three main findings.
The first was that participants had a spacial awareness
of the world.
Players were able to pass to their partners during training
and shoot past their opponents during games.
Which meant that they could act while tracking where
the other person was.
And they were able to do corner shots
and other tricks like that.
So we had quotes like, "Yes, I could definitely move
"the player to the ball and everything."
And, "That shot was on my goal."
They could also move to the center of the field
or either end, just by spatial memory of the environment.
Multiplayer experiences naturally emerged from gameplay.
So we had participants playing both with and against sighted
and visually impaired players.
So yeah, people are really excited about this like,
"Ah, cool there are two players."
And some people thought, "Hey, it'd be great
"if there were even more."
And we had some taunts like, "I'll crush you, but you have
"to close your eyes!"
This was actually really cool,
because it meant we had sighted players
and blind users playing together and all having a good time.
A really inclusive experience.
We end up having these little micro tournaments.
That's a second participant, another participant saying
they like crushing the opponent.
So we have some competitive participants.
And you've already seen this impromtu co-op mode
that just kind of emerged.
So one person's operating the pantograph,
the other one's operating the foot petal.
It's just because they both just really wanted to play.
This was really fun.
The final thing is that DualPanto could be some sort
of audio haptic console,
and this could be really beneficial for users.
Participants like that you could instantly hear sounds.
They like that it was really interactive.
And there's some extra insight we had too.
"For blind people it is nice when you have these things
"at home because you don't need to adjust many settings."
Heard it's been said that they could just play
for a half hour when they feel like it.
So it's not only accessible in that
it's replacing modalities, it's also reducing the barrier
to being able to actually use the devices.
You can kind of pick up and play.
So this seemed like it would be the right model
to use this device.
One participant even said, "150 euros, that'd be worth it
"even if it's only a few mini games for now."
So conclusion, DualPanto is a haptic device
that enables blind users to continuously interact
in virtual worlds.
With DualPanto, apps can move beyond 1D scanning
to actual spatial interation.
And here's Eddis defeating some zombunnies once again.
Now previously, we presented two other devices
displaying interactive spatial content
for people who are blind.
At CHI 2016 we presented LineSpace, which printed lines
in a large area, so you had tactile spatial content
and dynamic audio interaction.
And last year at World Haptics, we presented LinePod,
a mobile version of this that allowed you
to feel interactive spatial content again
while doing dynamic audio interaction.
And these represent one modality
which is discreet turn-taking.
So you had to take your hands off when something changed,
or if something changed, you wouldn't know where it was,
so you'd have to move you hands around to scan again to see.
And this is why we're so excited about DualPanto,
because it really offers continuous interaction.
You get push notification of when something changes
in the world when you have something that's spatial.
And we think that's really cool, and we're hoping
to explore it more.
Now of course, we didn't do this alone,
so I'd like to thank all my coauthors,
all of our coauthors on this work.
And if you're interested,
if you know someone who's interested in working
on cool haptics work like this, you can of course talk
to any of us.
Especially Patrick Baudisch at HPI, or you can tell someone
to talk to me because I'm starting a new lab
at the University of Waterloo.
And so if people are excited about this, tell them to get
in contact with me.
Thank you so much and we'd be happy
to answer some questions.
(audience clapping)
- So we have two microphones.
One is right there, one is there in the center.
Hmm, post-lunch fatigue.
Okay, I start with one question, I expect another.
Okay, so what I like best about this
is that you're encoding state
in like a spatial reference system.
And the system is supposed to sound,
which when it encodes state it's kinda noxious, right?
- Yeah.
- [Host] But at the same time,
sound you can do massive amounts of sound overlay
and are pretty good at telling them apart.
So I wonder if you've thought of like sort of
complimentary approaches here to not just supplement that,
I mean, it's sort of this trade-off, right?
- Yeah.
- [Host] You have a stable reference here with haptics
which is really great, right?
And encoding this state costs you nothing, right?
You should hold on to it.
Whereas in the sound,
you can do multiple things at once.
So I wonder if you've thought of this, sort of,
the complimentary nature of it.
- Right, so I mean, I'm constantly debating this,
whether it's primarily haptic or an audio design
when you do these things.
Certainly, a lot of the effort is audio design, right?
The entire interaction needs to be explained.
Most of the rich information,
in any ephemeral rapidly dynamic information
we could provide through audio.
But you're right, it is anchored by this.
So yeah, we had to give a lot of thought to audio design
in all the things we do.
For the soccer game, that tells you maybe who has the ball,
it tells you when you score.
You have that kind of stateful stuff also updated.
So yeah, we thought a bit about how you can load audio.
We're thinking more about how to do that.
Does that answer your question?
- [Host] It's pretty good.
- Yeah, cool.
- [Host] Don't we have a quick, is that Yung, okay.
- [Yung] Hi, Yung, UNG University.
Great talk, thanks.
- Thanks.
- [Yung] I had a question.
What you have is you have one dynamic object.
Have you thought about how can you extend it
to have multiple?
Say, if I wanted to know my players I can pass to
and my ball.
So do you need more hapitcal things
or can you just extend it through round dropping
what's most close or interesting to that object?
- Right, so that's a tricky issue.
So that's a big limitation with the devices implemented
is you only get one it right now.
So when it comes to something like Zombunny,
or the soccer game, we multiplex.
So it would show the nearest target.
We've thought about maybe you can switch
between the different targets.
So we haven't solved that yet.
We've thought that maybe you can do either haptically
or maybe you can do an interaction design,
and it depends on the complexity of the world.
So that's sort of the active next step.
So if you have ideas, let us know because this
is sort of the big thing we're trying to improve.
Anything to add?
- Nope.
- Cool.
- [Host] Okay, Merrie.
- [Meredith] Hi! Meredith Ringel Morris
from Microsoft Research.
So this is really interesting,
and I definitely see how it applies to the soccer game,
but I'm having trouble envisioning how this might generalize
to other genres of game.
Could you talk a little bit more about other kinds
of activities that this same system's a good match for,
and what the limitations are?
- Right, so we focus on gaming because
we thought that was a really hard, fun use case.
And it seemed a natural fit for dynamic spatial content.
We have a couple implementations, not in this part
of this publication, but we've been exploring
how you might do this for other types of spatial content
that can change and how you might render that.
So we have some ideas of how to do that.
It's not just a soccer ball.
We think we can do a lot more complex interaction,
but we're still working out both the resolution
of the device.
You need to have a really good device to be able
to do lots of different things,
as well as careful interaction designs.
Especially with audio and how you interweave the haptics
in the audio.
Does that answer your question?
I feel like it kinda does.
- [Meredith] Well, I think gaming,
I'm not saying that gaming is too small a limitation.
I was actually wondering within gaming specifically,
it seems like games where you kick a ball are only one genre
of many possible games and I was wondering if you comment
within gaming, what categories of games
does DualPanto work well for, and what categories
of games do not express themselves that way.
- Yeah so, there are certainly several,
a lot of different kinds of games that can work.
This might be best offline,
because we don't have defined categories yet,
but we're kind of exploring those limits now.
So maybe we should talk in the break or something like that.
- [Meredith] Thank you.
- Yep, thanks.
- [Host] Okay, let's thank the speakers again.
(audience claps)