[APPLAUSE]
IVAN POUPYREV: Thank you, Liz.
Thank you, Jacob.
So I want all of us to take a step back and just
marvel at how far we came with computing.
We used computers before, the size of the room.
And now have them in our hands.
And it's not just size that changed of computers,
it's how we use them has also changed.
This is how computers were envisioned to be used.
You know, you work on the spreadsheets.
You do your work.
You're very focused on the computer in front of you.
But this is how we use computers now.
And by computers, I mean mobile phone.
We are not focused on them anymore.
They're in the background.
We drive with them.
We run with them.
We even cook with them.
The phone is sort of this embedded companion,
always with us but it is not the center of attention.
And yet, the interfaces we're using for the phones
are stuck in the last century.
They're down for the office work,
for spreadsheets, for documents where the computer is a focus.
What would be the interface for a device that is not
just a tool to do tasks, but serve
as an embedded companion to support us in the background
and where the device is not the center of your attention?
So clearly, it would not be based on touch sensing, touch
interaction.
Because you have to actually focus on the touch.
And of course, this would not just
apply to mobile phone, this kind of interfaces.
It would apply to the whole categories of emergings
computing devices, such as devices mounted on the wall you
cannot reach, devices with small screens,
devices which don't have a screen at all.
And while voice interfaces, or verbal interaction,
has emerged as one of the important solutions
for these kind of interfaces, we focus
on non-verbal interaction, me and my team at Google.
And we're trying to understand how we can capture the user
intent through non-verbal cues like gestures, body postures,
proximity, facial expressions.
And these are very critical ways for humans
to interact with each other.
This is how we do this in the real world
all the time because a single smile can tell you
more than multiple pages in the book.
In certain situations, 93% of our communication between us as
humans is nonverbal.
And what people do is often more important than what they say.
We all know about that.
So why do we always have to tell our devices
explicitly what we want by voice, or commands,
or touching, and working with complicated interfaces?
Can we make our devices smart enough
so they can understand our intent
and what we want through non-verbal cues, gestures,
facial expressions, and so forth?
So what if the phone knows, for example,
when you're leaving home and alerts you to come and pick it
back, right?
So you don't forget it at home, back at home.
Or what if it can pick up information
from the screen of one phone and drop
another phone with a gesture rather than issuing
a complex command, and Bluetooth communications, and pairing
and so forth?
Wouldn't it be valuable?
Wouldn't it be great?
And what if it can't do this without compromising privacy so
you don't have to reveal your private information.
And that's what we started working on five years ago
when I started working on Soli.
Soli is a [INAUDIBLE] technology for non-verbal interaction.
Why do we use radar?
Well, radar has a lot of great properties.
It can capture large and small motions.
It can work through materials.
It's very easy to embed in different devices
like we did in Pixel.
And also, it's very robust and privacy-preserving.
The only problem we had when we started working with that
is that existing radars were huge.
And while our first design wasn't quite as
large as these bad boys, it was still
a large box like this one.
That was our first Soli device.
I used to go around and tell, it's going to be in the watch.
You can put in your watch here.
People didn't that believe me.
So we had to go through a lot of different iterations
of prototypes, and including some pretty wild variations
like this.
So by 2015, we drastically reduced Soli from this big box,
to this chip, single chip, as you can see here.
But still, it was not enough to get into the Pixel.
We had to go even smaller.
And today's device, you can see now on the screen,
that the latest one, 2019, is about one fourth
of the size of our first chip.
So this is really exciting for me
that we have made this whole journey.
So we went from the box like this size to a tiny chip
instead our Pixel 4 phones.
So I think it's a pretty cool technology development.
So the hardware was only one challenge.
We also had it address the complexity of radar signals.
The radar emits signals, radio waves and captures reflections.
That's how it works.
And it does it at the speed of light.
And as you move your body, this reflection will change.
So the goal on developing, using Soli and building
on top of Soli is to understand how the signal changes captured
by the radar correspond to your body motions.
What [INAUDIBLE] response is there.
And the challenge here was the biggest challenge.
It's that the signal that you have
doesn't look anything like this, something
that you can easily understand.
The radar recapture signal looks something like this.
You can't recognize any human identifiable characteristics,
which was one of the points of using radar in the first place.
But its made it very difficult for us
to understand how we can use it for our use applications.
So we had to go through endless experimentation for years
to understand which signal we want to capture and to get
the motions we want.
So here, you can see the person moving his hand.
And the red and yellow dots correspond
to motion of the speed and position of the hand,
so-called Doppler images.
So as we explored further, we started seeing patterns,
repeating patterns in the signal.
This is one of the early examples of the data
you can see.
And you can see that across the user, across the columns,
the signals look similar.
But between the rows is very different
and there is different gestures.
A gesture of a check mark in the middle
is very different from the hand tilt,
as you can see in this picture.
And this allows us to apply machine learning techniques,
signal processing techniques to separate this signal
and to understand what we want to get.
And that allows us to start exploring use cases for Soli
and think what Soli could do for the user, for the application.
So with regular devices like phones,
we can only use touch and voice.
That's all we can use.
Now, with Soli, in addition to touch and voice,
you have this sort of like a bubble.
You can use a hand motion, such as reaching and swiping,
inside of this small bubble around the device.
Now, this small bubble can expand to the bigger bubble,
about five to six meters is how far we can sense with Soli.
And now your whole body can be the bubble.
So you can figure out people's posture, or motions,
or big gestures like using your whole hands.
So what could we do instead of these big and small bubbles?
So we explored many possibilities.
Let me walk you through some of the early ideas
we worked on when we did Soli.
For example, can the large bubble see us exercise?
So, like you're working out and they Soli
can capture and tell you how to work out better.
Or can the small bubble see us making secret gestures
in the pocket, like for example, sign your receipt in a store?
Or can a small bubble, when a phone is in the pocket,
allows us to stop its calling in the middle of the meeting
by using the gesture like that?
This was one of the early ideas.
We also discovered that we can capture very small motions
of the hand in a small bubble, like micro gestures where
you hand becomes like a dial and you can use this dial,
like in your hand, your hands dial,
to control small screens like on smart watches, as an example,
or devices that don't have a screen at all,
or maybe there's no devices at all.
And we actually did build this into the watch
in the early prototype, s you can see in this video.
So what else?
In this small bubble, can your hand
become a puppet and you control a game?
You can pretend you're a little guy running, like a paper guy,
and the faster you move your fingers--
it's actually quite difficult. You get tired.
And then if you run fast enough, it becomes a plane.
And you can fly.
It can go back.
You can play for hours if you want to.
And then if you don't like that, [INAUDIBLE]
can use two hands like in the movie,
"Her," to have a little guy running
around this little world.
And he can jump by moving your hands forward.
So game controls was one of the fun applications
we tried to explore.
So how well can you track position of things
within this small bubble?
it is an early experiment.
And you can see how we can track position of things around this
slowly.
Can the large bubble see a person walking by?
Now, pay attention to the middle screen,
the middle window in the middle of the screen, the green one.
That shows the Doppler image of a person walking by.
You can see moving hands and a moving body.
So yes, you can see a person moving by in the bubble.
Can you use it for interaction?
Can we distinguish between, for example, a human and a dog,
like a pet?
Yes again, this is a person walking around.
This is the dog following up, the white dog,
matching the floor.
And you can see how a person moves.
And you see these big dots moving left and right
on the screen.
And when the dog runs.
You can see the tail moving up and down, left and right.
So a human, moving hands, a dog, moving tail.
So yes, you can distinguish between a person
and the dog using the radar.
So what if you can wear Soli in your pocket?
And then you can [INAUDIBLE] gesture-control your media
just by using your free hand gestures like opening your hand
to play back in the home.
So this was all the ideas, a lot of ideas.
We did all of the experiments we did, some of the use cases
we explored.
We explored many more.
And this was super exciting for us
in the early stages of the project.
Because it's validated, the Soli can
be used to capture non-verbal interactions.
And they are valuable and helpful for consumers.
We can create some exciting use cases.
And indeed, non-verbal interactions,
such as gestures, and presence, and body posture,
and facial expression, all of that,
represent the third major mode humans
use to communicate in addition to voice and touch.
To date, products offered the first two, voice and touch.
With Soli, we are finally making the third one
available to consumers.
What does it look like in the Pixel phone?
I would like to invite Shenaz Zack to come onstage
to talk about motion sense.
SHENAZ ZACK: Thanks, Ivan.
Hi, everyone.
My name is Shenaz Zack.
And I am a product manager on the Pixel team.
So as you just heard from Ivan, this was a long-term effort.
And there were a lot of ifs.
And so it was an extremely interesting challenge for us
on how do we take this new interaction
model to bridge the digital world with motion sense?
Now, the definition of focus areas and use
cases for motion sense followed an iterative process.
We first started with generative research
because we wanted to understand, how do people use their phone?
What are the challenges they're facing?
So after this extensive research,
we defined the focus areas and use cases
that motion sense could solve well.
And we went ahead and defined the experiences
and built prototypes.
This was just halfway through the process.
We wanted to validate those experiences.
And so we took those prototypes and conducted
both quantitative and qualitative user research so we
could validate our use cases.
Based on that research feedback, we
zoomed in on a few focus areas with a keen eye on helping
people, as that is in line with building a more helpful Google
for everyone.
Now, we all know we are distracted all the time, right?
And that was very clear from our user research as well.
Our phones keep ringing and we keep getting distracted.
So we want to stay in the moment even though we
need quick access to our phone.
So one of the key experience is we focused on
was silencing interruptions instantly
so you can stay in the moment.
Like for me-- this is my personal example--
my phone rings in the morning, my alarm goes off.
And I have to rush to find my phone
and quickly quiet it before my baby wakes up.
And so with motion sense, now I can just gesture,
and my phone silences immediately.
And then I can take the phone and silence it
before my baby wakes up.
The other thing we saw in our user research
was users were struggling to silence the phone when you're
in a movie theater, or you're in a meeting,
or you're in a kid's recital.
And all of a sudden the phone rings.
And you're like, oh my God, let me find that phone and silence
it.
You're scrambling to say find that button.
Now with just a wave, you can silence the phone
to avoid those embarrassing situations.
Pixel is the first device that actually understands
when you want to shh it.
You don't need to shout at it or hardly scramble
to tap a button.
You can just wave and silence your phone.
Now, based on the user research we did initially,
another big focus area that emerged
was how does the phone stay in step with you?
Phones today aren't proactive, they're reactive.
They're helpful.
But they're reactive.
They are helpful once you've pressed a button, touched
the screen, or issued a [INAUDIBLE],,
which I won't do now or all your phones will go off.
So this was another area we looked at helping
with motion sense for Pixel 4.
So with motion sense, it can accelerate
your common interactions with the phone
to ensure it's in step with you, like helping your device
unlock faster.
As you reach for your phone, motion sense
understands that you are trying to pick up your phone.
And it turns the camera on so by the time you've
raised your phone, your phone gets unlocked very fast.
Another key use case was truly helping your device
become smarter by only turning on the display
when you're near the device and turning it off
when no one is around.
With this, you can quickly check time, notifications,
or other information without even
having to pick up your phone.
You just glance at your phone, it knows you're there,
and it opens your UIs.
Again, with our research, we noticed that touch UIs
aren't always convenient.
When you're driving or eating or feeding your baby,
it's going to prevent you.
So we focused on how we can help extend the phone.
Thanks to Soli, on Pixel 4 it can understand your gestures.
You can use the phone when touch isn't convenient.
This can be again, like I said, when you're driving, or eating,
or while your display is off.
You don't have to bring the notifications down.
Your music controls are always available, and at all times.
You can just swipe to the next sound, go back, or go forward.
And last but not the least, my personal favorite--
waving hello to your favorite Pokemon character
or getting hearts from your Pokemon character.
You can use this on your wallpaper
or through the Pokemon app.
So as Ivan said, unlike vision, phone cameras [INAUDIBLE]
phone mics, radar is a sense that humans don't have.
So when we put it in Pixel and augment the user with it,
we're actually giving them a sixth sense and a superpower.
Thank you, everyone.
[MUSIC PLAYING]