- Hi, everyone. Welcome to the ARCore Depth Session.
I’m Adarsh Kowdle, a technical lead manager on ARCore,
and I’m here with Eric Turner, the technical lead for the ARCore Depth API.
ARCore is Google’s platform for building augmented reality experiences.
As a developer, you have access to key capabilities like motion tracking,
environmental understanding, and lighting estimation.
We launched the ARCore in 2017, and it’s now available on over 850 million devices.
Today, we’re going to dive into a specific part
of ARCore capabilities: depth estimation.
We’ll walk you through how scene depth can enable object occlusion,
improved immersion, and novel interactions,
which together can enhance the realism of VR experiences.
We’re going to show you the most of engaging AR apps
built by developers using ARCore.
We’ll also show you code snippets to give you an idea of how easily
ARCore Depth API can be used in your apps.
Finally, we're excited to tell you about new capabilities that let you do even more
with the Depth API. Stay tuned.
Let’s get started. How can depth help you with AR realism?
As an AR app developer, you want to seamlessly blend the virtual
with real for your users.
When the users place this virtual panda in their scene,
they want it to look like it belongs; it should blend in with the real world.
If you’re building an app for users to shop for furniture,
you want them to be confident the armchair they’re about to buy
will fit in their space and looks like what they’re seeing on their phone.
To achieve this on smartphones, the camera must know the world around us,
the size and shape of the real objects in the scene.
This is how most of us understand the world.
In this talk, we’ll show you how you can use ARCore Depth API to add a layer of AR realism
into your apps through occlusion, immersion, and interaction.
Let's start with occlusion.
Occlusion just means accurately rendering a virtual object
behind or in front of things in the real world.
Consider for example, this virtual cat the user's placing in the scene.
If you render this cat without occlusion,
you’ll end up with something which looks like this,
where the virtual cat looks pasted on the screen rather than in the world.
This doesn’t look realistic to the user, but when you use depth of the scene,
you understand how far away the virtual cat is relative
to surroundings like the white cushion,
allowing us to accurately render the cat with occlusion as shown here.
With Depth, you can develop more immersive AR experiences.
You can give users the ability to transform their scene into a winter wonderland.
Using Depth, you can render the snowflakes
to settle on real surfaces like the curved arms and pillows of the couch.
You can give users the ability to add a spooky fog effect
to the real world, where Depth is used to limit the visibility of the scene.
Users can also add a geometry of real lighting
to the scene, where the virtual lights interact,
hide behind, and relight real objects like the pool table here.
But that’s not all. Depth also enables virtual content
to interact with the real world through collision and physics,
which lets users reach out and touch the world through your app.
Users can make virtual characters aware of the real scene
and go over the obstacles like you see here.
In the second example, with the depth of the tree known,
you can see how the colored paintballs are colliding
and splattering on the branches of the tree.
And when depth-based collision is combined with game physics,
the whole experience comes to life.
The virtual objects interact and bounce off of the real world objects
like the exercise bike in this third example, just as they would in reality.
Using these tools, we have seen a number of developers
make their experiences more immersive and realistic using Depth API.
Google and Disney use the Depth API in The Mandalorian AR Experience
to accurately transform the scene with depth-based occlusion.
For instance, you can see how the play space here is restricted
to be within the kitchen walkway.
TikTok has used the Depth API to enhance their AR experiences,
allowing users to place graffiti in the real world,
such as the floor and wall here,
and project videos onto the real-world surfaces
like the couch and other surfaces of the room as you can see here.
Snap built a Depth template into Lens Studio,
so the creator community can add a layer of AR realism
into their Lenses using the ARCore Depth API.
Here are a few examples of Snapchat Lenses using depth.
From occlusion in the Dancing Hotdog Lens, to the immersive underwater experience
where the light guides the turtle around and over real objects like the chair.
And in the Seed Lens,
we see procedural generation of plants that are depth aware.
We believe this is just the start, and the depth templates will allow
the Lens Creator community to build many more creative experiences using Depth.
With these examples, you’ve seen what you could build using Depth API,
but how does it actually work, and how can you start using it right away?
I’m now going to hand it over to Eric, who’s going to explain
how you can use the ARCore Depth API to add a layer of AR realism into your app.
Over to you, Eric.
- Thank you, Adarsh, before I show you some sample code,
I wanted to show you the depth itself.
Depth Maps, shown here, offer a 3D view of the world.
Each pixel is associated with how far the camera is from the scene.
Areas in red are close to the camera, and areas in blue are further away.
We estimate this depth using the motion of the phone,
and that means that only a single camera is needed,
and that allows the Depth API to run on a wide range of devices.
And not all that much motion is required.
As shown on the right, is a phone that just started a new AR session.
The screen is black because no depth has been found yet.
Next, let’s see what happens when we start moving.
Depth pops up, that took just a couple centimeters of motion
and we got robust, accurate depth for the full image.
There are some points to keep in mind
when designing experiences around this depth.
Since we compute depth by comparing different views
of the scene at different points in time,
it works best when the scene stays the same with minimal motion.
And as shown here, the device only needs to move a little bit to generate depth,
but experiences that encourage the user to move more will get better and better results.
This process allows us to estimate depth from zero to eight meters away,
and we’re targeting the best accuracy to be about half a meter to five meters away.
Now let's check out some sample code.
With ARCore, you can retrieve the depth image that matches every camera frame.
As shown in this code snippet, the call to acquire depth image
provides an image object matching the same timestamp
and field of view of the camera ready to use.
Since the depth is from the motion of the camera,
it’s common for these images to only be available
after the user started moving around.
One other thing to keep in mind is that Depth is disabled by default in ARCore.
Depth does consume some processing on the phone,
so this choice saves as many resources as possible.
So if you’re using Depth, make sure to enable the Depth mode as shown here.
You can also see that there’s a check for is Depth mode supported just above.
Not every device supports Depth, so this check is important.
Our goal is to ensure developers are provided a consistent quality of depth,
and we make sure that the current device has enough resources available
to handle robust depth computation with plenty to spare for your ARF.
Fortunately, we’ve streamlined the Depth API to be supported on a wide range of devices.
Since we launched the Depth API last year,
we've been working hard to optimize our algorithms
so it can run on more ARCore-supported devices
for more Android users, and Depth is getting even better.
Another improvement we’ve made since last year
is to selectively use machine learning
to augment our Depth algorithm, increasing its speed,
even with minimal motion from the user.
We also take advantage of any additional hardware the phone might have.
So for example, if the phone has a dedicated depth sensor, such as time of flight,
we take full advantage of it and we automatically merge data from all sources.
Through these updates, we’ll continue to improve Depth to give you
the best possible experience for your apps with no additional change needed from you.
And that’s not all. Today we wanna share new features
we’ve been working on to help you enhance the realism and precision of your apps.
Today at I/O, we are announcing the raw Depth API.
We’ve already shown you examples of how depth images are used
to create immersive AR effects,
letting users transform the look of their scene.
And we’ve heard great feedback from developers,
including an interest in accessing the raw data,
which can be used for more detailed, more accurate 3D sensing.
For example, this point cloud shown on the right,
it was generated using depth from motion.
Today, we are expanding ARCore, so this information is now available to you
through the raw Depth API.
The raw Depth API provides you the 3D points for which we are the most confident,
so you can feed them into your other algorithms.
The full depth image we already showed does a good job of blending depth everywhere,
ensuring every pixel has an estimate, but it doesn’t capture all the finer details.
This version’s great for in-screen effects like occlusion or relighting.
The raw depth image, on the other hand, it might have some missing pixels,
but this is the best, most accurate data for 3D processing.
And we pass this accuracy along to you through a matching confidence image
allowing apps to choose which depth works best for their use case.
Raw Depth is available on all the devices we showed before.
No specialized hardware required.
Just like before, we can access the raw depth image
and the confidence image for every camera frame.
Just call the two matching functions shown in the code snippet on the right.
This raw Depth API is already in use today in a variety of applications.
One example is PHORIA’s ARConnect app, which uses raw depth’s improved accuracy
to generate detailed measurements.
Another example is 3D Live Scanner,
which takes advantage of the richer 3D analysis
of raw Depth to perform object and scene reconstruction.
And improved measurements and physics allow users
to place an Air Rube Goldberg contraption
in their space with Jam3′s raw Depth API demo app.
This experience is open-source and now available.
There's also another feature we're making available today.
Hit Test has been a fundamental interaction for AR apps,
allowing users to place objects at a 3D location in their scene.
But previously this action could only be done when planes were detected,
which limited locations to large, flat surfaces,
like the results shown in green on the floor on the left of this image.
Now we are incorporating Depth to provide the best possible hit results,
giving more results even on non-planar surfaces, like the chair on the right.
These results, in red, even work in areas with low texture
where hit results just weren’t possible before.
The Depth Hit Test takes advantage of both smooth and raw Depth under the hood,
giving you the best data we can from all available sources.
We give you the new Depth-enabled hit test the exact same way as before.
Just call the same hit test function. You’ll simply get more results.
And just make sure to check for depth points in the returned list.
Our Depth Hit Test feature is already available for you to try in the wild
with apps like LifeAR from TeamViewer or our own Depth Lab.
You can find both of these in the Play Store today.
So what’s next? Well, that's up to you.
Please check out all the ARCore Depth API experiences in the I/O sandbox.
We’re providing several open-source examples to start with.
Our Depth Lab is open-source in Unity and has been updated with our latest features
providing a bunch of creative ways to use Depth.
And we have two code labs, including one that teaches you how to retrieve
and apply depth images for occlusion.
And at I/O this year, we have a brand new code lab that walks you through
building an app using raw depth for geometric analysis,
so doing things like detecting objects and finding their bounding box shown here.
Whether it’s to make fun videos or measure the world or play a game,
developers are using the Depth API to make their AR experience more realistic
and more immersive, and so can you.
We’re excited to see what you build with the ARCore Depth API.
Thank you.