[MUSIC PLAYING]
JARED FINDER: Hi.
Welcome to What's New in AR.
I'm Jared Finder, engineering lead for ARCore,
and I'm joined by Eric Lai, product manager for ARCore.
In this session, we're going to talk
about what ARCore is, improvements
we've made over the past year, and some new capabilities
that we are excited to announce.
First, a quick introduction to ARCore.
ARCore is Google's platform for building augmented reality
experiences on mobile devices for both Android and iOS.
We want augmented reality to be available to everyone and work
on existing devices.
Augmented Reality, or AR, is a technology
that enables completely virtual objects
to be placed in the real world.
The virtual objects get drawn on top of the camera image,
and when drawn correctly, they blend in, almost
like they're actually there.
This augmentation of reality allows apps to show information
in a more natural, more human way, such as pointing exactly
down the street to turn on, allowing someone
to remotely help you by adding 3D annotations,
or immersing yourself in a virtual art piece.
There are many different capabilities that enable AR.
We group these all together into the ARCore skill tree.
We've been steadily leveling up foundational capabilities
while adding new ones since ARCore
was first released in 2017, working on ways
to make our technology faster and more
accurate without requiring people
to buy new mobile devices.
ARCore is available on over 1.2 billion active Android devices,
and so you can build cross-platform AR apps.
Select features are available on iOS as well.
We support the iPhone 6s and up.
AR is used by core Google apps, such as Google Maps for walking
navigation, as well as by third parties,
such as TeamViewer for remote AR assistance solutions.
Now, a key request we consistently
hear from app developers using ARCore
is that they want the core capabilities that
let objects feel real to be faster and more reliable.
In the real world, objects don't glitch in and out of existence
when you turn your head.
Users expect the same from virtual objects in AR.
Motion tracking is core to enabling immersion
in AR experiences.
By tracking how the phone moves through the world
and where the floor, tables, and walls are,
an app can place virtual objects on those surfaces
and have them stay anchored in place.
The virtual objects need to always be drawn appropriately,
even as the phone moves around, so that they feel as if they're
actually part of the world.
Over the past year, we've made significant improvements
to motion tracking.
We've reduced the average time it
takes for motion tracking to start up
and a plane to be found by 17%, and we've
reduced the amount of time motion tracking
gets lost by 15%.
This allows experiences to start up faster and be more reliable,
all without any code change needed in an ARCore app.
But we want to do better, and thanks to Google's investment
in machine learning, we can.
Late last year, we started an experiment
where we use ML to drive motion tracking in cases where
the camera image is not enough, forced here
by turning the screen black.
This experiment resulted in faster initialization
and a further reduction in the amount of times motion tracking
gets lost.
You can see in the video on the right
how motion tracking is able to continue appropriately even
when the screen goes black.
We are currently rolling out these ML-driven improvements
to all users of Live View in Google Maps.
Another key part of immersion we've been improving is depth.
Depth enables AR experiences to have more interactions
with the real world.
We use the existing camera on a smartphone
to build a depth map, which captures the shape of the world
out to 8 meters or 26 feet--
about the size of a big room.
Here, you can see a video representation
of the depth map, where the color changes from red to blue
based on how far away that Pixel is.
One thing this enables is occlusion,
where virtual objects appear behind real ones.
This is valuable in all AR experiences.
Here, you can see the same scene with and without occlusion.
Without occlusion, the penguin appears
in front of the potted plant, making it not
look a part of the world.
With occlusion, the penguin is properly
drawn behind the potted plant and immersion is maintained.
This also enables interaction, where virtual objects can now
properly follow the non-plane shapes of surfaces
in the world.
This is from TeamViewer.
See how the line follows the shape of the printer
as if it were being placed there in reality?
Here, too, we've leveraged machine
learning to improve the speed and coverage of depth.
The average time to initial depth is reduced by around 15%.
The average overall coverage of the depth map is now at 95%
and is dramatically improved on surfaces with minimal texture,
as you can see on the empty white wall and on the black TV.
These are places that classical depth would struggle with due
to a lack of visual features.
Depth coverage is now comparable to the amount of coverage
in dedicated depth hardware.
This also helps out motion tracking,
with a further 8% reduction in the time to the first plane
and an increase in the number of planes accurately found
of about 25%.
And on select Pixel phones, we can
use the existing second camera to get the depth even faster.
This dual camera depth further speeds up
the time for initial depth, while also speeding up
motion tracking too.
It only needs to run for a little bit at the startup,
so this has a minimal effect on battery life.
This is currently rolling out for Pixel 4 and Pixel 6.
And I'm excited to announce that we have just
released long range depth, which is available now.
With long range depth, depth can be detected up
to a range of 20 meters or 65 feet.
Here, you can see long range depth map near one
of the Android statues on the Google campus.
With this increased range, depth can now
be used in outdoor situations in addition to the indoor ones.
Occlusion is now available outdoors, even
in direct sunlight.
You can see here how Live View in Google Maps
uses long range depth.
The destination pin is properly occluded
when it is behind the world and gets
revealed, just like a real object,
once it should come into view.
This experience is currently rolling out in Google Maps,
and I look forward to seeing what new experiences you build
when using the depth outdoors.
If you want to learn more about how we leverage ML
for improvements to ARCore, see our talk
"What's Under the Hood in ARCore."
And now, I'll hand it over to Eric
to talk about the new exciting cloud services
that we are launching.
ERIC LAI: Thanks, Jared.
We believe that geospatial, three dimensional, and world
encode information will transform the way people learn,
navigate, play, and experience the world together.
Data shown this way feels more helpful, more understandable,
more human.
And to that end, we have been consistently focused
on building cross-platform cloud services that allow developers
to add augmentations anywhere in the real world,
making the real world into a virtual canvas.
In 2018, we launched Cloud Anchors,
enabling multiple users to simultaneously share
a single AR experience.
In 2020, we followed up with persistent Cloud Anchors, which
persist for up to one year.
Cloud Anchors allow you to build a rich, persistent set of AR
content in the real world that many different users can
see and share.
Cloud Anchors are best use for room scale AR experiences,
where you can be present at the place in person
to host a Cloud Anchor.
Many of you asked for world-scale capabilities.
And today, we are releasing the first set of ARCore geospatial
APIs, enabling world-scale shared AR experiences that
can be authored remotely from your desk
for both Android and iOS.
With the ARCore Geospatial API, AR apps
can now calculate the latitude, longitude, altitude,
and heading of your phone with greater accuracy
than the GPS sensor can provide.
This allows you to place geospatial anchors
for virtual objects.
Geospatial anchors are tied to a specific GPS coordinate.
So they can be placed anywhere in the world
without you having to go there.
This is powered by Google's global localization technique,
combining Visual Positioning Service or VPS, Street View,
and machine learning.
So it works all around the world.
This uses the same technology powering working navigation
of Live View in Google Maps.
Let me explain you how to use the API.
I want to create an experience with a virtual object
in a park near my home right next to the sundial here.
I need to figure out the exact latitude, longitude,
and altitude for the anchor.
This can be done from the comfort of your own desk
using Google Maps.
The specific steps I'll describe in our Developer Page.
Here are the geospatial coordinates of the sundial.
These coordinates can get persisted
into my Apps database.
When configuring the ARCore session,
make sure to enable the geospatial API.
This only needs to be done once before AR is started.
And when a user enters AR, I use the sundial's coordinates
to create a geospatial anchor using the new Earth Create
Anchor API.
And I'm done.
The geospatial API enables location-based virtual objects
for anyone anywhere on Android and iOS.
This can all be done without having
to map out each location, enabling constantly
evolving dynamic virtual objects to be added to the real world.
This is our next step towards building a helpful and useful
AR for everyone.
If you want to learn more about the geospatial API,
check out our talk "VPS and ARCore Geospatial API."
Good AR seamlessly integrates virtual objects
into the real world such that they
are indistinguishable from real objects.
To enable this, we've been leveling up the ARCore Skill
Tree, both adding new skills, as well as making our existing
capabilities faster, more accurate,
and available to everyone.
By combining these capabilities, you
can build compelling new AR experiences.
I hope you're as excited as I am about the future of AR.
If you want to learn more, please check out
our developer resources and samples, as well as our older
talks.
Thank you so much for your time.
And I can't wait to see what you build next.
[MUSIC PLAYING]