(upbeat pop music)
- Welcome to New Capabilities in ARCore.
My name is Rajat Paharia, I’m Product Manager at Google
responsible for ARCore.
I’m joined by Jared Finder, who is the engineering lead for ARCore.
In today's session, we're going to cover ARCore
and some of the key capabilities it provides.
Let's start with a quick refresher on AR.
Augmented reality, also known as AR, is the capability to perceive the world
and then add to or augment it.
It does this by taking live images from the device camera
and then integrating virtual items into the physical world
in a way that makes it look and feel
like they're actually there.
This can be used to create compelling, useful
and delightful experiences in games,
for navigation, for commerce, for social media and more.
ARCore is Google's platform
for building mobile AR experiences for Android.
It launched in 2017, and at the last Google I/O,
we announced that ARCore was certified on 400 million devices.
I'm excited to share that today, ARCore is available
on more than 850 million devices and installed on over 1 billion devices.
Developers are able to build experiences
directly on Android or using Unity or WebXR.
To enable this seamless integration of the virtual and real
we've been building out the ARCore skill tree,
empowering developers with all the foundational capabilities
around realism, perception and asynchronous interaction.
By combining these capabilities in various ways
you can create compelling, useful and delightful
AR experiences for your users.
First, let’s look at the realism branch of the tree,
which includes foundational AR capabilities
around motion tracking, lighting, and depth.
We humans use a number of environmental cues
to figure out whether something is real or not,
including lighting, shadows, occlusion and more.
ARCore's realism capabilities enable you
to put digital content into the physical world
in a way that users can't tell the difference
between what's real and what's not.
Can you tell which mannequin is real, and which isn’t, in this picture?
The mannequin on the left is the virtual one.
Motion tracking enables virtual objects to stay in place
even though the camera's moving around
because if they’re jittering or floating away
they don't feel real.
In the last year, we made ARCore’s motion-tracking
even more robust with an up to 36% decrease in CPU usage
and a dramatic reduction in the number of tracking resets,
in many cases eliminating them completely.
Environmental HDR understands the lighting in the scene
so your virtual objects can look and behave
just like the real ones do
and enables your virtual items to seamlessly blend into the scene.
As you can see, the rocket on the right
looks more like it’s actually in the room
than the one on the left.
Specifically, this includes specular highlights, reflections,
in this case from the brown floor and chairs,
accurate shadows and accurate brightness.
And finally, last year we released our Depth API
which uses a single standard smartphone camera
to provide a depth map that enables experiences
to be more immersive.
Your virtual objects can now actually interact
with the real world, rather than just being stickers
on the screen.
Using the Depth API, you can implement things like particle effects.
Here you can see rain and snow falling on the sofa.
Interactivity, so virtual assets can directly interact
with the real world including collisions and surface interactions.
On the left, you see someone shooting paint
at a tree and the paint is splattering, sticking,
just like you'd expect.
On the right, someone’s shooting virtual objects
down the stairs, they’re bouncing around
just like real items would.
Occlusion makes it so that digital items
can accurately appear behind real-world objects.
On the left, see how things look without occlusion.
The cat appears like a sticker on the screen
overlapping the chair, whereas on the right
you can see that the cat is properly occluded.
Part of its body is behind the chair like a real cat would look.
And finally, lighting effects like the ones you can see here,
that enable you to realistically change
the lighting in the scene.
You can try all these and more in the ARCore Depth Lab app
which you can find in our Sandbox.
Now I'm gonna hand it over to my colleague, Jared,
to take a look at the perception branch.
- Thanks, Rajat. Perception is the set of capabilities
that enable you to detect certain things in the scene
so that you can augment them.
It includes capabilities like instant placement,
augmented images, augmented faces,
ML models that can understand what you're looking at
and the brand new Raw Depth API.
One of the main user experiences in AR
is placing a virtual item, like a chair or a dinosaur,
on a floor or a table.
This requires users to move their phone around
so that ARCore can identify planes in the scene
which can take several seconds and is often a friction point for users.
To address this, we built Instant Placement,
which enables users to place virtual content
without first having to pan and scan to find a plane.
With Instant Placement, we’ve seen a 16% increase
in placement success and a 26% decrease in placement time.
What if you want to attach AR content
to a specific image to bring it to life in full 3D AR?
You can do this using our augmented images API
which tracks the image, even if it’s moving,
and attaches your content to it.
And if you want to create awesome face effects,
we have an Augmented Faces API,
which provides you with a high-quality
468-point 3D face mesh that you can anchor effects to.
This is available on both Android and on iOS
so that you can build cross-platform experiences.
We also have the capability to integrate your own ML models
into AR experiences so that you can understand
what's in the scene, recognize objects and act accordingly.
Using this capability, you could build experiences
like an app that looks at a chess board and shows you the next best move to make
or a photo filter that displays virtual bees
wherever there are flowers in the scene
or a game where every lamp in the world turns into a monster.
To help you with this we just released a new sample
that shows how to integrate your own ML models.
And finally today we're excited to announce
the Raw Depth API
which enables developers to build more accurate measurement,
reconstruction and interaction apps
by giving you access to a more detailed point cloud
representation than our standard depth API.
If you're interested in learning more,
check out our technical session called increase AR realism
and precision with depth.
Finally, we have built a number of capabilities
around asynchronous interaction.
These enable you to build AR experiences where users can interact with places
and with each other across space and time.
The capabilities here include Cloud Anchors and Recording and Playback.
Cloud Anchors enables you or your users
to annotate the world with AR content that stays there
so you can build your own constantly evolving layers
on top of the real world
and create location-based experiences
that can be experienced by anyone and at any time
on both Android and on iOS.
And these annotations can last up to a year
from the last time someone looks at them.
And today we're very excited to announce
the launch of the ARCore Recording & Playback API.
For your developer velocity, you can now record a video
and then play it back through ARCore.
Building an experience in a shopping mall? Now there's no need to go to the mall
every time you want to test a change.
Simply record your visit once
and then test from the comfort of your own desk.
And for users you can now enable new post-capture AR experiences.
Why should your user have to stand in their living room
to see what virtual furniture looks like there?
Now they can record their living room once
and then add virtual furniture to the scene
whenever and wherever they feel most comfortable.
If you're interested in learning more, check out the related technical session
which you can find below.
To enable this seamless integration
of virtual and the real, we’ve been building
out the ARCore skill tree,
empowering developers with all the foundational capabilities
around realism, perception and asynchronous interaction.
By combining these capabilities in various ways
you can create compelling, useful and delightful
AR experiences for your users.
We hope you're as excited as we are
about everything that you can do with ARCore
and we can't wait to see what you build.
If you’re interested in learning more about anything that we covered today
please check out our developer resources.
On behalf of the entire ARCore team, thank you.
(upbeat pop music)