Hey, it's me, Destin. Welcome back to Smarter Every Day.
I want this video to be long and I want it to get down into the weeds
and just air out and let me get as technical as I want to,
so I'm gonna mention this sponsor right now upfront, which is different, I know.
But I want to say thank you to 23andMe for sponsoring this episode of Smarter Every Day.
I've been to their lab too, its a really cool place.
So, I'll tell you more about their at-home genetic test service later in the episode
but, for now let's talk about Haptic Gloves.
In the last episode of Smarter Everyday you got to see me interact with the Haptic Glove
and it changed my mind about VR (Virtual Reality)
and here's the deal, when you're a start up company, it's a double-edged sword, right?
You want people to know that you're doing these great wonderful things
so you can get investors and stuff like that.
But conversely you don't want them to know the "secret sauce" because
you gotta maintain your competitive edge, right?
So, the fun thing about this is when I got down to talking to the engineers
at like the fundamental Haptic Pixel level on how this thing works, you can see their eyes
dartin' around the room, like they freak out.
The reason they freak out is, because they haven't been permitted to talk about how this glove works
in forever apparently and then this dude walks in with a camera and sticks a camera in his face
and asks him very technical questions, its really fun to see their reaction
but, I was able to sign some paper work and we were able to work out a deal so that
I can take you right up to the edge of proprietary information, but not cross that line
so that they can still maintain their edge.
So after I met the founder Jake and the co-founder Bob, they turned me over to the head of engineering ops,
Lathe, to asks questions.
And in the last video we went through what the experience was like, this time I just wanted to focus
on how everything works
I decided to just go for it and ask Keenan what kind of actuators they used to simulate touch.
-So, I am going to wear a glove under the glove. -Correct, its just for sanitary purposes.
-I am really excited right now.
-So the question I have is, if...
-if I need... if I touch on my hand right here and I am supposed to feel the presence of something there.
-In order to have an actuator in that location you have to have a ton of different valves for example if you're using pneumatics.
-or you know, linear actuators if you're using electro-mechanical actuators. -Yes.
-So how do you get around that problem?
-We describe them as like, Haptic Pixels and each Pixel would be its own... pneumatic actuator
-and they are arrayed over your hand.
-Okay, I'm ready for it. Yeah. -Okay.
-How many of these exist?
-Umm...
-I don't think I can share that.
-Okay (both Laugh)
-Cool.
-So we'll just pull this down and I'll help you
-Put these on
-Ah, so you're aligning it to my finger. Oh, wow.
-Yup.
-Well, what are these?
-So, we...
-We, uh..., ahh...
-They don't want to talk about anything!
-Okay, so we developed a custom motion tracking solution that uses both optical and magnetic tracking
-to get really accurate positions of each one of your fingertips. -Yes.
-And then we do some crazy stuff in the software and using all that information we can figure out where all your fingers are in space.
Yes and what I learned from my class I'm taking at school is, there is a lot of cordinate transformations happening in my hand right now.
Right? Are they happening in my hand or are they happening off my hand in the processor?
They're happening in the computer
I know the math and that's awesome! Alright lets do it!
After running the simulation I learned one of the most important concepts, it's called the two point threshold.
You've had to have done a study on this. You figured out the specific bladder density that you need
in order to trick the brain
knowing that it's not gonna go
photorealistic and feel like complete imersion.
Is that true?
Yeah. There's actually a medically known
metric called the two point threshold.
Which is basically if you take two pencils and poke somebody on the skin,
how close together do those get before they can't tell that it's two seperate points?
I keep calling them bladders what's the actual word?
Ah...we call them tactors.
So the spacial resolution of the tactors
just has to be inside of the two point resolution
of the human hand.
It's a little more complicated than that.
I mean the two point threshold is one of very many
aspects of touch and I am not the most qualified person to talk to you about that.
and we do have a couple of folks here that can talk for a very long time about it
-So I'm Destin. -Hey I'm Adam.
So how do you develop software for something that has never been done like this before?
like do you create you own language
for the glove?
So...the way the software works is that it takes
information from the physics engine
and then we have to figure out how to render that
And you know it's pneumatic
We have to figure out how to open and close valves in the right order and all that sort of thing.
So the software does that.
So you have 150-something valves or whatever it is you have to control,
and you have to control it almost real time,
You know your latency don't you?
-Yeah -Yeah
So, and you're not gonna tell me that I'm assuming.
-I'm not talking about that
(both laugh)
So you have to, It has to be quick enough where a human can think that they're actually engaged with it.
So there is a threshhold there.
There is a threshold and they've done studies on it.
They found that anything longer than a quarter of a second is too long.
You lose the temporal connection between the things.
-But if you go below like..you know.. half that then the brain -You're fine?
Yeah your brain doesn't know.
They're discussing about whether or not a I get to see a tactor
I really wanna see a tactor
(laughs)
Who do I talk to to see a tactor?
Do you think..do you think Lathe will let me see-? Ok.
-We're looking at the website for that. -They're looking at the website.
Because thats one of the rules we were given
If its on the website then we can- we can show you.
Gotcha. Ok cool.
Have you ever met someone that you just click with?
You're on the same wavelength instantly?
This is Mark. The other engineers call him "Mr. Glove".
This is the guy that knows all the things
The fun part about this conversation is if you watch Mark's eyes,
it's clear that he just wants to "geek out",
but he can't,
because someone set up this proprietary borderline that he can't cross.
Anyway, this is one of the most enjoyable conversations I've had in a really long time.
-So that's Mark. -Hi
So...this is it huh?
Yeah, and they're really pretty simple.
As you said they're pnuematic, as you guessed they're a bladder.
So right here I'm looking at the end
-So I'm seeing...how many is that, 12? -Mhm
What do you call- so this is a tactor. So what are these individual dots called?
We actually call the individual dots tactors and the whole thing is a panel.
So we group a bunch of tactors together on a panel,
it makes it easier to route air to them,
makes it easier to support them,
and we can control each of them individually.
Look I know we're in the weeds here but Mark's about to describe a bunch of neat little things about your hands.
So if you'll check out your hands and do these little activities as he's describing them,
you'll understand things that you've never even thought about before.
Like the squish of your skin as you touch a hard object.
If you will use your hands to do them as he's talking
you will understand the sense of touch on a much greater level.
So this is the spot where the
research stuff that you have to get into kind of moves away from the traditional
electrical mechanical engineering and kind of moves into how the brain works.
It moves into ergonmics, it moves into human factors, and it moves into psychometrics
Psychometrics...I don't even know what that means
It's how people process things. It's how your brain works for sensation.
I don't know if you've ever seen something called the sensory homunculus
it's a picture of a person scaled
relative to how many nerve endings they have in different parts of their bodies.
It's basically a surface density of your your tactile receptors.
-Is that a good way to say it? -It's a very good way of saying it.
These panels, we think a lot about the huge hands part of that.
They're incredibly sensitive.
And so figuring out how to design a piece of equipment that is high fidelity and
capable enough of tricking those sense organs is a lot of what our challenge is.
-Okay so you're inflating this particular tactor here right? -Right
-It's like a balloon right? -Mhm
If I were to touch this table with my hand right now, okay
Touch the table.
So it's not a step function it's a curve, but it's not linear either.
-Right. -Because as you press in on something there's thickness to my skin
and so it starts as a line and then it ramps up and then it gets harder and harder and harder right?
-Absolutely. -Is it an exponential curve?
Do you know what that curve is?
You don't have to tell me what it is.
You're hitting on a great question which is what that approach and that that curve of displacement looks like.
Remembering that your fingers are sense organs. One of the ways they sense is by stretching, right?
They're deformed. And you actually feel that deformation.
So there's "what is your skin doing?" Which is, it actually gets stiffer as you push on it harder and harder
-And that's part of why we have such good... -Dexterity? -Right.
The other part of it is "what's the material you're pushing into?"
In some cases, if that material is soft, well that's gonna deform as well.
So you can have slightly different approach.
-Yeah -Like we're sacks of watery meat stuff
and so the relative difference between the hardness of the object you're touching
is going to affect the attack time
-I don't know what term you used but I would say attack time- -Rise time.
-Rise time? -Mhm
This is all making sense.
There's also little things like how well does your glove fit?
That's why when I put the glove on, it felt like there was...
a physical thimble that goes on the end of my finger
is because you wanted a tactor to be aligned with my finger just like that, right?
-Right -Is that because you have to have something to react against?
-Correct -Is that why you did that? -Yep
When you're touching something, it's not just the feedback you feel on your skin,
but also the feedback you feel in your joints, your tendons, as you push against it.
So you want to make sure to combine both of those.
You're absolutely right. And I've never thought about that at all.
If I grab something...if I grab this.
And I squeeze it,
I can tell how big it is because of where my tendons are.
And I can tell how hard it is because of how hard I'm squeezing it
and it's deflecting my skin between my bones,
and all that goes into...it rolls up into one transfer function...
...to give you the answer of what object is in your hand. -Right.
And that's where things get complicated.
Your world revolves around boundary conditions doesn't it? Like mathematical boundary conditions.
Pretty much yeah. Boundary conditions and contact mechanics.
So what is this right here? This tape that I'm seeing?
It looks like a tape measure that's pulling back.
That's the force feedback system.
That's what stops your hand from closing when you grab an object.
-It's a boundary condition. -Yes.
It's a boundary condition. So instead of pushing on this side of the hand,
you're actually pulling from the other side of the hand, and that's what stops the hand from moving.
Correct
And so you have to have some kind of braking mechanism back here...
So, can you crush things?
Like if you have something virtually can you set a resistance?
Like if I were to touch something hard versus something soft,
can you vary that force feedback?
(nervous sound)
-Yes. -Okay.
Yes. Okay. (laughter)
These guys are like: "everything's proprietary just wear the glove, stupid boy"
I think that one was just a complicated answer.
That was a complicated answer?
That one is really complicated.
So something that matters here is, the entire volume...
-There's a time it takes to do that compression. -Right.
So there's a lag associated with... It's basically the elasticity of the volume right?
What's the word?
well there's a question of "how fast you can push air through to this guy?"
and that gives you how fast you can run that tactor.
So it's this balance between having enough pressure to activate the system in fast time,
granted you're going through a small tube.
So the effusion rate of the gas is important. So you've got all that figured out.
And there's mechanical lag associated with the effusion rate,
or the head pressure is what it's called.
Yeah. With the velocity of the fluid.
The feedback diagram, or the the block diagram of how perfecting this looks
it's gonna look like a plus and a minus between you two guys right?
And include some electrical engineers in there for good measure.
We don't- nobody cares about electrical engineers let's be honest
(laughter) Just pretend I didn't say that.
I'm with Jeffery an electrical engineer.
So Jeffery I know how the tactors work.
Or at least I've seen them. I think I figured it out.
-You felt them? -I felt them
I make sure we have good information about where the fingertips and the other parts of the hand are,
so that we can turn on tactors at the right times to the right pressures.
Dude that's awesome.
I mean those valves that you're controlling, can you talk about that at all?
I can't talk about the type of valve we use,
but the timing and the hydraulics problem that you mentioned are things we work on pretty frequently.
It's important to get them inflated the right amounts at the right times.
We kind of get a pass on how accurate that is just because of how slow sensation is.
-Right -You know people's perception of touch especially
is not nearly as fast as things like vision or sound.
-Especially thermally. I'm assuming that's something you guys are... -Thats particularly slow.
Ed was telling me about the thermal component to this.
Both of these objects are 73 degrees but that metal feels way colder.
Yes when you're touching these objects you're not just feeling the temperature of the objects,
but you're feeling the heat transfer between them and your hand.
And that's because...
The reason the metal feels colder is because it has a higher thermal conductivity than the wood does.
-So it's like sapping more heat out of your hand. -That's exactly right yeah.
You're gonna model different types of objects by making more water flow or float a different temperature
to pull more heat out of your fingers that's basically what you're doing.
Yeah that's that's exactly right and it's important that we don't just flow water at the right temperature,
but we've flow it at the temperature that you makes you think you're touching
the real objects with different thermal conductivities.
Adam showed me another demonstrator where a dragon flies across the room
and breathes fire on my hand which I can feel both thermally and like touch,
and then it breathes ice on my hand, which felt cold.
Now what's clear to me is there they've got to be using fluid at this point,
because that's the only way you can remove that much heat.
So I think there's a tactor for touch and a pad for thermal.
but I think they're separated within the 2-point threshold.
They wouldn't tell me. I wasn't permitted to look.
But one thing that was really interesting,
If you switch from hot to cold, it (snaps fingers) happened really fast.
So they have to dump the hot from the lines and get cold in there fast.
So there's a lot going on here and I want to know mechanically how they do this.
This is a heat exchange problem.
-This was number two? -This is number two, you tried number three.
Awesome so this is second generation.
Ok so it's just one hand stationary in location.
Oh god that makes so much sense.
If I were to develop this, I would have started at moving the hand in space
and then add the haptics to that but instead you started with the haptics
We knew people could do motion capture -- that's been done before -- so we wanted to start
with the thing that hadn't been done. -Clever
keep your thumb beside your hand as you go in
How are you doing that? I would expect that that's fluid in there or that's
actually a solid that I'm touching. You're not allowed to tell me?
I'm looking at Lathe right now just to see.
(offscreen) It's hot and cold fluid - Gotcha
it's a chunk of snow it's a snowball oh it's a dragon oh it's gonna burn my hand
I don't think it likes being poked oh shit that's awesome okay that's good so
you guys like Game of Thrones so it's a combination of the tactors firing and
temperature control. so you haven't implemented temperature control into--
-Not in the glove, the glove does not have thermal
Is that a goal?
I don't think I can talk about that (Laughter)
That's awsome, so
so what we've learned is it's how far the tactors are fired, the rate at which
they they are fired, so it's not just the rise time. The timing is important, not just the rise
time but the the impact timing is important the heat transfer with fluid
which is something obviously they don't want to talk about obviously they're
working on it that's amazing that all that stuff actually tricks your brain it
does a really good job of tricking your brain which I did not expect and I did
not like VR up at this point seriously that's pretty cool we're living in the
future haptic gloves you're touching stuff in virtual reality I'm about to
tell you about the sponsor which is 23andme 23 pairs of human chromosomes
where the name comes from if you don't know about 23andme the way it works is
you go to 23andme.com/Smarter.
I'd appreciate it if you use that link you
get this vial you spit in it and you send it off you can do one of the two
kits you can do the health and ancestry kit or just the ancestry kit but if you
have any questions about where this goes I did a whole video at their lab the
cool thing about this though you get this report online you can learn stuff
about your body about your family it's it's amazing but here's the deal the
Food and Drug Administration the FDA just recently approved 23andmes ability
to release new genetic marker information and I don't want to go into
the specifics of what that is but it's stuff you care about and this
is new like this is future stuff okay so go to 23andme.com/Smarter
and at least read about the new information
that 23andme can release to you because
the FDA allowed it it's cool stuff so I would appreciate it if you participate
in the future with me and go 23andme.com/Smarter you're smart people
you know how this works: that helps smarter every day. I hope you love this cuz we're
stinking in the future like right now and it reminds me of when I was a kid
and used to watch Beyond 2000 with my dad I feel like I'm getting to make that
that show now, which is cool. Anyway, please consider going to 23andme.com/Smarter
that's support Smarter Every Day. Please consider subscribing,
if you enjoyed this video, if not, that's no big deal. Thanks for coming along on the ride with me.
Anyway, I'm Destin, you're getting smarter every day. Have a good one, bye!
So I feel like I sufficiently understand at least a little bit of the system yeah
thank you very much! It's Mark, right? Yeah. So I have found the wizard for the glove,
it's Mark.