[MUSIC PLAYING]
BILAWAL SIDHU: Welcome to our Tech Talk.
I'm Bilawal Sidhu, a product manager,
and I'm joined today by Simon Lynen, a software engineer.
And we're from the Google Maps team.
Today, we're really excited to introduce you
all to the ARCore Geospatial API, which
will allow you to build global scale, immersive,
location-based AR experiences on Android and iOS.
We've been mapping the world in immaculate detail and scale
over the past 15 years using satellites, aerial, and ground
level imagery to build an unprecedented understanding
of the dynamic physical world.
In totality, we've covered more than 10 million miles
of Street View, a distance that can circle the globe
more than 400 times.
We've also covered more than 36 million square miles
of HD satellite imagery, covering more than 98%
of the entire population.
From this massive corpus of billions of images,
we have fused them together into a 3D world model
that geometrically and semantically understands
the world.
And it is the underpinning foundation for our first party
applications like Google Maps and Earth.
This 3D world model is optimized for visualization, analysis,
and localization-- in other, words maps for humans
and machines alike.
Today, we are here to talk to you about the machine readable
3D model and, in particular, the smartphone camera
in your pocket and how we're enriching it
with Google's understanding of the world to allow you,
the developer community, to create
immersive, world-scale AR experiences.
We're excited to partner with the Google AR teams
to bring this 3D understanding of the world to you
through Google's AR developer platform, ARCore,
which allows Android and iOS developers to build
immersive AR experiences for mobile.
This technology is called Global Localization
and has been stress tested in production,
powering the AR Live View mode inside of Google Maps
and the AR Places Filter in Google Lens.
And today, we're excited to open up the same core
capability to you, our developer community,
to create immersive location based AR experiences
in your own applications.
This capability is available in over 93 countries
across the globe everywhere we have Street View.
With that background in mind, let's get right into it.
We'll start off by discussing different options for building
location-based AR experiences.
Then, we'll provide background on the technical capabilities
on our roadmap as well as walk you
through the fundamentals of using this API
and just how easy it is to get started.
And finally, we'll share what our early partners are already
doing with the Geospatial API to address a range of verticals
and use cases.
Let's take a step back.
If you want to build location-based AR experiences
today, you have a couple of options.
Use built-in sensors like the GPS and compass on your phone.
Use spatial anchors like the ARCore Cloud Anchors API.
And finally, use the Geospatial API, which
is the subject of today's talk.
Let's understand the benefits and trade offs
of using these various options.
You've got GPS, which is a global scale
system, great for building location-based experiences.
But if you've ever dealt with your blue jumping around,
especially in an urban canyon, you
know that the accuracy's a far cry from what
you need to build AR experiences that reliably
augment the camera view.
Even after you do that crazy eight pattern for calibrating
your compass, with Location Services,
you'll typically get 5 to 10 meters of positional accuracy
and 30 to 45 degrees of rotational accuracy.
This means that your camera may not even
be looking in the right direction,
or your AR elements are completely out of frame.
You also have the ability to use spatial anchoring capabilities
like ARCore Cloud Anchors.
In this case, you need to explicitly map
a space or place of interest to attach content to it
and then can reliably localize against it.
Cloud Anchors works really well at near-field distances,
for private spaces, or smaller event-scale experiences where
you can physically show up and map the space
before attaching AR content.
But with larger scale deployments of Cloud Anchors,
discovering content is a challenge and so is placement.
We've heard from you, the developer community,
that you want to guide users to the right location,
to be able to remotely place content without physically
going to those locations, both of which
are hard to do with Cloud Anchors today.
Enter visual positioning system and global localization.
Thanks to Google's decade plus of ground level imagery,
we've already mapped the world in immaculate detail.
By working in conjunction with Location Services,
we can take a good GPS estimate and make
it great, ready for AR.
So with the ARCore Geospatial API,
you are able to remotely attach content to the world and guide
users to it, allowing you to create
immersive, location-based AR experiences almost everywhere
we have Street View.
Let's summarize the differences.
With GPS, you get global scale with lower accuracy
and variable reliability.
With the Cloud Anchors API, you get high accuracy
within a limited scale because you explicitly
need to map a space.
And with the Geospatial API, you get the best of both worlds,
remotely placing content in the world and offering
nearly ubiquitous, accurate localization
and heading without needing to manually map the space.
We hope to surface additional capabilities over time
to help you build truly next gen experiences.
I will now hand it over to Simon to unpack the underlying
technologies powering the Geospatial
API and the capabilities it gives
you to turn the world into your canvas.
SIMON LYNEN: Thanks, Bilawal.
In the following minutes, I'll walk you
through the three pillars of the Geospatial API.
Geo localization, geometry and anchoring, as well as
world understanding.
Localization is the foundation of the Geospatial API.
It is key to bring your AR experiences to the global scale
by allowing you to place virtual content in relation
to the globe.
To provide this capability, the device
makes a server call to the visual positioning service,
VPS, where your geographic coordinates and images
are used to match against high-resolution 3D data
from Google Maps.
This process is called Visual Localization.
And it precisely locates your device
with respect to the environment much more accurately
than what was previously possible with GPS alone.
I want to take a moment to run through how all this works.
Street View images-- which have been captured around the globe
for more than 15 years--
are the foundation of VPS.
Deep neural networks identify and describe
those parts of the images that are
likely to be recognizable over long periods of time.
You can see that the algorithm has learned to focus
on buildings instead of cars.
We then combine these salient points across tens of billions
of images to compute a 3D point cloud of the environment.
This localization model consists of trillions
of points and spans nearly all countries,
with future coverage.
When your device makes a request to the Geospatial API,
a similar process is applied to the image.
A neural network processes the pixels
to find recognizable parts of the environment
and matches them to the localization model.
Computer vision algorithms then compute
the position and orientation of the device that
is sent back to you.
This process makes use of TPUs--
Google's machine learning accelerators--
and it takes less than a second.
Using the response from VPS, the Geospatial API
takes care of merging your local coordinates
with the geographic coordinates so you can work
in a single coordinate system.
As Bilawal mentioned earlier, this launch
offers a very rare combination.
The Geospatial API builds on the visual positioning service,
which is in a very mature state, and we
make the API available for nearly all areas covered
by Google Street View, which is very widely scaled.
One key challenge when building AR experiences
is the accurate placement of content in relation
to real world geometry.
The API we launch today allows for anchor placement
at a given latitude, longitude, and altitude.
We expect to launch terrain anchors later this year,
which will allow you to create anchors
by providing only latitude and longitude
information, while other information from Google Maps
is leveraged to precisely place the content
at the right altitude above ground.
The image on the right shows an example
where the precise altitude information
is key to a delightful user experience.
In the following, I would like to give you
a behind-the-scenes look as a technology
that the team is currently working on.
These capabilities are not launching today,
but we are exploring ways to make
them available for developers in the future.
Their experience has become truly magical
when they seamlessly integrate with the real world--
for instance, through occlusions and physics into action.
We believe that one key enabler is
given by abstract building geometry, as shown
in the image on the right.
Another angle we are exploring is to bring already familiar AR
concepts to a global scale.
For example, ARCore provides the ability
to anchor elements to physical locations
and snap them to plane geometry detected by the device.
For the Geospatial API, we are thinking
about a similar concept which we call Geometric Anchors.
It consists of two parts.
The first part is to capture and persist
intent, such as keeping anchors that were attached
to a wall sticking to the wall.
The second part is about maintaining
anchor poses so that even if our models are refined over time,
anchors remain at the location that you
intended them to be at.
Lastly, we will hope to gradually expose
more of our underlying 3D semantic model, which
will allow you to build even richer experiences.
In particular, in combination with existing Geospatial
offerings, such as Google Maps platforms,
adding semantic understanding to the Geospatial API
will allow you to procedurally place content at places
in the world, such as a specific fountain in a park
or your favorite coffee store.
We are looking into making pieces
of the semantic 3D model available
in the coming quarters.
After this exploration in the future,
let's take a quick moment to look
at what we are launching today.
This code example shows you how easy
it is to bring your AR experiences to a global scale.
To obtain Geospatial data, you need
to enable the Geospatial API by changing the Geospatial
mode in your session configuration to Enabled.
To determine an anchor location for remote anchor creation,
you can for instance sample points from Google Earth
or Google Maps.
You can then start creating anchors
in the global frame using these coordinates.
Another example I want to show is
how you obtain a Geospatial pose in your app.
A Geospatial pose consists of latitude, longitude,
and altitude information, as well as device orientation.
All these quantities are provided
with estimated uncertainties.
So you can adjust your experience accordingly.
You see that only a few lines of additional code
are needed to build global scale AR experiences
with the new Geospatial API.
And with that, I'm heading back to Bilawal
to walk you through some use cases.
BILAWAL SIDHU: Thanks, Simon.
We've seen developer interest for a range of use cases
and verticals.
Today, we'll focus on three sets of applications.
The first vertical is ridesharing and micromobility.
Both demand high accuracy location
for a range of use cases where GPS just doesn't cut it.
Thanks to our new API, partners are removing the friction
from parking dockless e-scooters and e-bikes,
adding pinpoint accuracy so that riders
know exactly when their vehicle is in a valid parking spot.
The result is greater peace of mind for riders
and more properly parked scooters on city streets,
leading to better relationships with the communities
that our partners serve.
Conversely, vehicle discovery gets easier,
especially in dense urban corridors with the increased
location accuracy, which our partners are
using to help customers find vehicles,
thus improving their user experience
and increasing vehicle rental rates.
The next vertical is location-based AR experiences.
Thanks to our API, developers can
focus on creating compelling user experiences that
provide utility and delight without needing
to build and maintain maps of multiple locations.
Telstra and Accenture are creating a companion app
for Marvel Stadium that brings customers to their seats
while showcasing engaging, immersive content
along the way.
DoCoMo and Curiosity are turning Tokyo hotspots
into an immersive world where you
can fend off virtual dragons with your favorite robot
companion.
Finally, we have gaming and self-expression.
With the Geospatial API, you can literally turn the globe
into a canvas for creativity.
In Balloon Pop, you can place balloons
as targets in the real world and try
to pop them using a physics-enabled ball that
reacts to distance in world space.
Nearby users can join the experience
and pop the same balloons via a simple Firebase implementation.
And with Pocket Garden, you can adorn your neighborhood
with a colorful AR community garden,
accurately placing seeds across the real world at scale,
planting vines, puffy succulents, and more.
We've also given Snap early access to this API
to help explore entertaining and educational use cases.
All of these are great examples of what is possible when you
turn the globe into a canvas.
And we're excited to share that Balloon Pop and Pocket
Garden will be open sourced and available to you
to bootstrap your own creativity.
We look forward to seeing what developers build
with the ARCore Geospatial API.
So to summarize, with the introduction of the Geospatial
API, we're providing the foundation
for building global scale AR experiences.
The API is mature and production-ready,
with the underlying technology deployed at Google scale
for more than three years and with a range of partners using
it in their own experiences.
The Geospatial API is available today
on all supported ARCore-enabled devices across Android and iOS.
Get started with our new Geospatial codelab,
or check out our samples for Android, Unity, and iOS,
and the ARCore SDK.
We're excited to see what you create
when the world is your canvas.
[MUSIC PLAYING]