[MUSIC PLAYING]
KONSTANTINE TSOTSOS: Hi, and welcome
to "What's Under the Hood" in ARCore.
I'm Konstantine, here with Luca.
LUCA BALLAN: Hi.
KONSTANTINE TSOTSOS: That's a great set you're on, Luca.
LUCA BALLAN: Yeah, fantastic, isn't it?
Yours is pretty nice too.
KONSTANTINE TSOTSOS: Thank you.
Today we're going to dive under the hood of ARCore
to discuss its inner workings, talk
about how you can design your AR apps to help them perform well
for your users, and we're going to give you
a sneak peek into some exciting new ML developments.
Let's get started.
ARCore is Google's developer platform
for building mobile AR experiences.
It packages a variety of immersion capabilities,
like motion tracking, depth, and lighting,
to help you integrate virtual content into your live camera
stream.
These capabilities are supported on over 1.2 billion
ARCore certified phones, and the algorithms
powering them are based on years of academic research.
These algorithms work together to enable
some incredible things, like the experiences here.
On the left, we're freezing a local scene mesh
and re-texturing it, letting you re-imagine your space.
In the middle, we're leveraging geometric understanding
to simulate convincing artificial lights.
Or on the right, where our virtual character navigates
real obstacles in the environment.
These are possible thanks to many new features
we've brought to ARCore working together,
helping you to bring the real world into the experience
you're designing.
Today we're going to focus on motion tracking in ARCore
for Android phones, one of the foundational capabilities
for AR.
This is what tells you where your device is
as it moves through space and powers
our understanding of the environment.
We're going to talk through what happens under the hood
during your AR experiences and discuss
key things to keep in mind for your users when building them.
Finally, we'll show you some of the latest R&D coming your way.
First up, let's build your intuitive understanding
of what makes ARCore tick.
It often works pretty seamlessly but sometimes
it may be difficult to understand
what's going on if it isn't doing what you expect.
Here's a basic scene on the right.
We start the app, play some content,
and get straight into AR.
But how do we know when we can place content and start
enjoying it?
To help, we're blending the image
with data from the depth API.
Once we can see depth data-- that spread of colors--
we can be pretty sure that ARCore is ready to go.
This is because ARCore needs environmental understanding,
estimated geometry of the scene in front of you,
to let you place content using our hit test APIs.
These APIs let you Raycast into the scene
to find detected geometry for placing your content.
Here, we're using the recent depth hit test
API, which uses depth data to place content.
Let's watch that snippet again, but with depth blending already
on, so that we can see when to start placing our content.
We can see that we go from dim colors,
when we have no depth available, to a range of colors, going
from orange to green, for near to far,
telling us that we have usable depth data.
Once these colors show up, we're ready to jump
into the experience.
We're going to keep an eye on that for some tricky examples
coming up.
But first, what's going on under the hood in this example?
Here's what it looks like inside.
On the left, we can see the camera image.
The colored lines and dots there represent something
called feature tracking.
Feature tracking is where we find interesting, textured bits
of the image and see where they go, frame to frame over time.
We use this to tell where the phone is moving.
Let's freeze that left side so that you can better
see the feature tracking.
On the right is ARCore's understanding of the world.
That funny triangle is called a frustum
and it represents the angle the camera's viewing from.
The lines following it are the motion path
that the camera has taken.
That's me waving the phone around here.
Let's freeze that right side now.
The yellow and orange dots are the depth image and raw depth
image from the depth API on certified devices.
Those blue dots are the point cloud API.
The brighter the dot, the more confident it is.
As I said, we've been working on these algorithms
for a long time.
And for those of you that have kept an eye on the area,
you're probably familiar with these kinds of visualizations.
Here's another fun example we've shown before,
where we took the underlying algorithms powering ARCore,
strapped them to a car, and drove
through the popular Lombard Street in San Francisco
to help stress test the systems.
Though please note, ARCore is not
intended for use in a moving vehicle.
What about when things don't go this well though?
For example, on the left, assets might disappear briefly
if you're moving really excitedly.
Or on the right, it can be difficult to get
the experience going.
If you keep an eye on the blend of depth,
you'll notice that it doesn't show up until the camera has
moved enough.
What's going on under the hood?
First, for the case where we lose our asset.
On the left, the colors on the feature tracking
that we see in the camera image tell us
about how long we can track an interesting point for.
The more red, the shorter we see that point.
The fewer points, the less information
we have to work with.
Here, when we move excitedly, there are very few features
and we only see them for very short periods of time,
making it really difficult to track camera motion.
In these cases, ARCore realizes there's a problem
and lets your app know.
Once we've recovered, your assets can come back.
But why is this happening?
We're all pretty used to using cameras,
but have you ever tried to take a video at night
and found that it might come out blurry or grainy?
Your camera collects light over time.
When there's less light, it needs more time to get enough.
But if you move during that time, what you're seeing
gets a little bit smeared over the image.
When that happens, we can't rely on the images
to tell you how you're moving.
In our previous example, a lot of the images were very blurry.
This happens frequently so ARCore has a strategy
to make sure that your AR content doesn't disappear
every few seconds.
This is something called visual inertial sensor fusion.
The camera, typically capturing frames
of at least 30 frames per second,
is the visual part of this equation,
but what's the inertial?
That comes from something called an inertial measurement unit,
or IMU, that consists of an accelerometer and gyroscope.
These give us linear acceleration and rotational
velocity of your phone along three different axes.
And all ARCore certified phones provide these measurements
at over 100 times per second.
By adding up these measurements over time,
usually called IMU integration, we
can get a pretty good idea of how your phone has moved
between camera images, or even across several images,
if the frames are blurry.
This is how we keep tracking going,
even if feature tracking degrades on a couple of images.
Later on, Luca is going to tell you
how machine learning is helping to stretch this integration
time even further.
This visual inertial sensor fusion
is the core of our AR motion tracking.
And it may sound a little familiar to you.
After all, when you move around the world,
you know where you are, thanks to a camera too--
often two of them-- thanks to our eyes.
And if you close your eyes, you can still
feel if you're moving, thanks to your vestibular system,
also known as your inner ear, giving us our sense of balance.
That's kind of like an IMU.
Your brain, like ARCore, combines these signals
to help you navigate the world.
That's a pretty effective strategy.
Like our blurry images though, things can go wrong.
If you've ever been sea or motion sick,
you've experienced what happens when
the signals from your body's IMU and cameras disagree.
When you focus on the horizon to steady your stomach,
you're helping the signal from your eyes
better line up with the actual motion of your body.
Making sense of these combined signals is pretty tricky.
Let's look at that in our challenge case,
where it took us a while to place content.
On the right, if you look at that 3D triangle--
the frustum--
you can see that the camera is barely moving at all.
It's mostly just rotating, not translating or moving
in a straight line.
Eventually, we do get enough motion to estimate depth,
but when the point cloud appears,
the points are all deep blue.
They're not very confident.
We need to view the world from a bunch of different angles
in order to confidently estimate distance.
The more that you move or translate, the more
accurately and quickly we can get you into the experience.
Later on, we're going to tell you
how we're bringing in some new machine learning
to further reduce how much motion we need to get going.
You can use a visualization like this
to help communicate this idea to your users.
You should encourage them to move the phone around,
combining both rotation and translation,
to give us lots of data to help your AR experience.
Putting these together, to help users get right into it,
first, avoid having them hold still at the start.
Guide them to move their device with smooth, not jerky motions.
Second, having lots of light in your environment
helps the camera to take crisp images for tracking.
We haven't mentioned the third one,
but you may have picked up on it.
To track interesting points in the images,
it helps to have the camera looking at interesting stuff.
Blank walls, for example, will be a challenge
for feature tracking.
So encouraging users to avoid low texture areas is important.
Here's another example for communicating
to users about lighting.
Our API actually provides a set of tracking failure reasons
to tell you when to trigger these,
like excessive motion, insufficient light,
or insufficient features.
But what else can you, as a developer,
do that doesn't depend on your users?
We have some APIs specifically to help with this called
anchors.
In your experiences, often you might
be able to walk around for a while
and give ARCore lots of information to work from.
Other times, you may have an issue.
Maybe you drop your phone or someone turned off the light.
Here, I'm simulating that by covering the camera
with my hand.
ARCore anchors are the right way to place your content
in all of those cases.
The fixed location and orientation
in physical space that you assign to an anchor
will update as ARCore's understanding of the space
improves.
Anchors are the only way to make sure
that you can leverage this improved understanding.
And it's important to note that absolute world coordinates
do change over time, but that ARCore ensures
anchors will remain stable.
If tracking does fail temporarily,
anchors let us know where to correctly place your asset
once ARCore has recovered.
Let's look at this example under the hood.
After covering the camera, I move
to the other side of the scene.
When I remove my hand, on the right
you can see that ARCore recognizes the new perspective
of the world and figures out where it is, relative
to its past motion.
It then updates the anchor with that information,
letting you know precisely where to put your asset,
so that it looks correct on the camera image.
Further, ARCore has used this extra perspective
to improve its past estimates of motion.
And the anchors let us communicate this more accurate
position back to you.
We even pay extra attention to improving the accuracy of world
geometry around those anchors.
Here we show you how to create an anchor
after a successful hit test--
in this case, of an instant placement or depth point
return.
It's as simple as hit.createAnchor.
To get the pose or position and orientation of that anchor
for rendering, it has a .pose method.
That pose can then be used in your typical rendering loop.
Please check our documentation on anchors
and hit testing for more information.
All right.
So how are we making all of this even better for you?
We have a few new R&D experiments
we're working on to up level all ARCore tracking,
without you even needing to change your APIs.
We talked before about how we need enough motion
to estimate depth.
But what if we could flip that and use depth
to help us estimate motion?
Thanks to recent advances, our depth
is now supercharged with ML and we're
able to estimate it with far less motion and texture
required.
Thanks to this, on the left you can
see that we get accurate, raw depth on the blank wall.
And on the right, we get a full raw depth
image with barely any motion.
We've been working on a new approach
to use this improved depth as an input to our motion tracking.
This will tell us how far away features
might be before we've really started moving, helping
you start your experience even faster.
Let's see what this looks like.
On the left is our earlier example
with very little motion.
On the right, we use this new approach,
combining depth and motion tracking.
As you can see, this new approach
lets us start the AR experience far sooner.
Depth information and the ability
to hit test to place content are available to you
almost right away.
And tracking becomes much more reliable.
This will be a seamless upgrade for your experiences,
and we look forward to rolling it out to you soon.
Now, Luca I will tell you about another exciting development
in the works.
LUCA BALLAN: Thank you, Konstantine.
Now I'm going to talk about how we
use machine learning to improve our motion tracking stack.
Specifically, how we combine IMU
signal with a neural network in order
to provide a continuous AR experience.
As mentioned earlier, in order to get an accurate
estimate of a device pose our motion tracking algorithm
heavily relies on triangulating features across images.
Reliable features, however, might
be missing in some situations.
For instance, textureless environments,
like being in front of a white wall,
as shown in the animation on the right.
Or with fast motions, which generate blurry camera images.
Or in general, when the visual features measure a motion
that is incoherent with the actual camera motion.
For instance, being in an elevator.
The elevator moves up, it moves down,
but the image is always the same.
Or just by having a lot of people
or objects moving in front of the camera
and significantly occluding it.
In all these scenarios, if features
are missing for a fraction of a second,
the IMU will still be able to cover the gap.
However, if these features are missing
for a longer period of time, then tracking
will stop providing poses, making the AR content disappear,
as shown in figure, and consequently interrupting
the AR experience.
We are working on a machine learning solution
to cover the tracking gap when reliable features are missing.
This will allow for a continuous AR experience to all our users.
And the idea is to still rely on the device IMU,
but to use machine learning to intelligently integrate
its measurements over time.
But why do we need machine learning in the first place
here?
Well, let's look at an example.
On the left, we have our standard ARCore processing
a walking navigation scenario.
While on the right, we have the same scenario,
where we artificially remove all the features
at some point in the video--
the black frames.
There we let IMU figure out the motion by itself.
As you can see, the IMU only solution
quickly diverges, resulting in the AR content floating away.
This is actually due to the fact that the IMU only
measures its own acceleration.
To compute its velocity, we need
to integrate these accelerations over time.
Practically, this correspond to summing them up.
Now to compute the device position instead,
we need, again, to perform another integration.
But this time, over the previously computed velocities.
Basically, this corresponds to a double integration
of the original accelerations.
You can see now that any small errors or noise
in the acceleration measurements quickly sum up
and accumulate over time, making the pose diverge.
And this is where machine learning comes into play.
So we design a neural network and train it
on hours and hours of real motion data,
representing many different ARCore use cases.
You can see some of them in the figure.
There, we learn the correlation between IMU measurements
and actual device motion.
Intrinsically, the network learns
how to correct for the double integration problem
and to avoid the divergence.
The underlying idea is that human motion
is relatively regular and not completely random.
We can leverage this information to constrain the motion
estimation problem and this is actually
what the neural network learns.
At tracking time, we take the current IMU measurements
and feed them into the network.
The network, in its turn, returns the current device
position.
So let's look at some results.
When reliable features are available,
ARCore works as normal.
However, when these features are missing,
like during the artificial blackouts in the video, that
is when the machine learning solution steps in
and, in real time, supplies the device poses in place
of the standard ARCore.
As you can see in the video, the AR content is stable, even
during the blackout frames.
And it moves consistently with the user motion.
This feature will soon be available to all Google Maps
Live View users, so we can all enjoy a continuous AR
experience.
In conclusion, we wanted to give you
a glimpse of what is under the hood in ARCore,
as well as the best practices for a reliable motion tracking.
Moreover, we walk you through how
to leverage anchors to maximize the accuracy and the stability
of your AR content.
In the end, we show you our recent improvement
in ARCore, how we use depth to speed up initialization,
and how we use machine learning to make tracking more robust.
Konstantine and I would like to thank you for your time
and we hope you enjoyed this look under the hood.
Happy coding.
KONSTANTINE TSOTSOS: Thank you for joining us.
[MUSIC PLAYING]