Welcome to our Tech Talk.
I’m Ian Zhang, a product manager for ARCore,
and I’m joined by Nitin Suresh, an engineering manager for ARCore.
Today, we're really excited to introduce you all
to the Recording and Playback API, which will allow you to take AR anywhere.
Before we begin, I wanted to take a moment
to recognize the team that helped bring Recording and Playback to life.
Nitin and I are here presenting on behalf
of a large cross-functional team spanning engineering,
product, business development, UX, quality,
developer relations, tech writing, EngProd,
legal, policy, privacy, and marketing.
We couldn't have launched without the hard work of this entire team.
Today, Nitin and I will first chat through some of the basics of Augmented Reality
and AR app development.
Then we'll talk about the concept of Live AR versus AR Anywhere.
We'll dive deep on what the Recording and Playback API is,
and how to use it, and also where we think you, as a developer,
will find value from both an app development perspective
and a user-facing perspective.
Finally, we'll wrap up with recommendations
on how to get started and where to go to find more AR-related content.
To start off, let's quickly review exactly
what Augmented Reality and Google's ARCore platform are.
AR is the capability to perceive the world,
but the geometry and the semantics about what is in the world and then augmenting it.
If you've ever captured images or video on your device.
then layered virtual items onto the physical world in front of you,
you’ve experienced AR.
AR helps improve our everyday lives and creates moments of genuine delight.
For example, you can get helpful directions
using AR walking navigation in Google Maps Live View.
3D arrows, pins and street names pop up to help localize
and navigate you to your next destination, safely and on time.
You can learn about animals in Google Search,
bringing immersive, 3D animals to life in your living room
just by searching for them on Google.
The experiences you see here, among many others,
were made available on Android devices
by leveraging ARCore, Google's developer platform
for building mobile Augmented Reality experiences.
ARCore launched in 2017 and is available on more than 850 million devices.
So far, ARCore has been installed on over 1 billion devices.
As a developer, you're able to build AR experiences
directly on Android, or using Unity or WebXR.
Recording and Playback, the topic of today's talk,
is available through ARCore and Unity today.
So Taking AR Anywhere with Recording and Playback, the title of this session,
what does taking AR Anywhere mean?
And how is that different from how AR is normally experienced?
Well, until now the vast majority
of Augmented Reality experiences have been live.
They required users to be at a certain place,
at a certain time, with their phone on, an AR app opened and in a special AR mode.
Say you’re at an electronics store,
and you want to see whether this TV in front of you
fits on a table in your office
and matches the aesthetic of your environment.
You have options: You could buy the TV and bring it back
to your office and hope it fits and looks nice.
You could jot down or memorize the dimensions of the TV
then go back to your office and see if it would fit.
Or, using the power of Augmented Reality,
you can place a virtual TV on your real life table
and assess the fit, the color, the lighting,
all without purchasing or memorizing anything.
But you’re probably thinking, in order to experience this in AR,
I'd have to wait until I got to my office then try placing the virtual TV.
That’s a hundred percent right, and that’s what we mean by live experiences.
You have to be in a certain place at a certain time to experience it.
But what if you could remove the live requirement of AR experiences?
What if you could enable AR experiences anywhere at any time?
Well, as of today, with the launch of the ARCore Recording and Playback API,
we're delighted to say you can.
Let's think back to the electronic store example.
Instead of having to go back to the office to place the virtual TV,
imagine pulling up a pre-recorded video
of your office from your native camera gallery,
wherever you happen to be, whenever it's convenient to you,
and trying out any number of virtual TVs,
or, for that matter, any number of virtual objects.
You could do your AR shopping on a train,
commuting to the office, in bed, lounging around.
You could literally be anywhere at any time
and still be able to experience Augmented Reality.
That’s what we mean by taking AR Anywhere,
and that's the power of Recording and Playback.
Now we'll take some time to dive into the APIs a bit more,
explaining what they are, how they work, and what they enable.
Normally, AR experiences require a live camera session feed
with a platform that takes that live session feed
and runs all the algorithms necessary to discover planes, place virtual objects
into the physical world, etc.
The Recording and Playback functionality
helps remove the live camera session requirement.
The Recording API stores a camera's video stream,
as well as IMU data, planes, or any other custom metadata
you, the developer, choose to save in an industry standard MP4 format.
Developers can then feed these recorded videos to ARCore via the Playback API,
which will treat the MP4 just like a live session feed.
To be clear, you can still use a live camera session, as you always have.
But now with this new API, your AR applications
can opt to use a prerecorded MP4 instead of that live session.
Now, what does that look like to a user?
Think back to our electronic store example.
Users can now pull up any video previously taken
from their native gallery and be able to edit in
or playback AR objects, effects and filters.
Important to mention is that while the Recording API
requires ARCore, developers aren’t locked in to using ARCore's Playback API.
In fact, you can use any other playback mechanism
that is AR compatible.
I want to emphasize that one more time because it’s important.
An MP4 recorded using ARCore’s Recording API does not require ARCore or even Android
running on the playback device.
All that's required is a compatible player that can demux an MP4 file,
and manage the 3D telemetry contained inside, like Android’s ExoPlayer.
We chose to build on top of the industry standard MP4 format
for video playback and editing
so that you, as a developer, can reach as many devices
as possible, as seamlessly as possible.
Now we’re going to dive into both the developer
and user facing benefits of using Recording and Playback.
And I'm going to hand it off to Nitin to chat about the developer aspect first.
Thanks, Ian!
I’m excited to talk about the developer benefits of the Recording and Playback API.
The Recording and Playback API removes time and space constraints for developers.
For developers, you can now record a video using ARCore
and then playback using any compatible device.
Building an experience in a shopping mall?
Well, there's no need to go to the mall every time you want to test a change.
Simply record your visit once and then iterate
and develop from the comfort of your own desk.
We have made Recording and Playback
as easy to invoke as possible with just a few lines of code.
ARCore API calls are called as usual.
For recording, you specify the dataset and the recording config during setup.
For playback, you specify the dataset during setup.
To explain further, here is how you would start the recording in ARCore.
As you configure a new ARCore Session, you will set the new recording config
and a file path for the output MP4.
You will ask the ARCore session to start recording
on the recording config you set up, and that’s it.
When you run ARCore, the dataset gets recorded to the output as you desire.
We just saw how to record the dataset.
Let's now see how you can create an ARCore session
and playback previously recorded dataset.
When you want to play back this recorded dataset,
all you have to do is point to the recording part you specified previously.
When you resume the ARCore session,
ARCore will treat this MP4 as if it’s a live session,
running tracking algorithms, depth and generating all the AR frame data
you need to develop your AR application.
In our codelab, which you can find links for the end of this session,
you can get hands-on with the code and see more complex examples.
One of the more powerful examples we talk to is recording an MP4,
and also the UI tap data directly into custom track to access later.
To explain further, here is how you’d record custom data in ARCore.
You would create a data track using the custom UUID
and then set a string mime type on the track.
You would add this track to the recording config you created.
When the ARCore session is in progress, you can record your custom data
on the AR frame as desired.
Similarly for playback, when the ARCore session is in progress
you can play back your custom data on the AR frame as desired.
Recording and Playback shines when it’s difficult to reach
a certain destination to test your AR experience.
As an example, our partner DiDi,
a vehicle-for-hire company, is opening new venues.
Previously, each time DiDi want to test a new feature,
a developer or QA tester had to physically go to the airport,
capture the walking path of the user, navigate the airport,
walk to the baggage carousel, and finally out to the cab ride lane.
With Recording and Playback, DiDi was able to capture the journey once
and send that scene to the development team, where they could test and iterate remotely,
thereby saving significantly on R&D and travel.
Recording and Playback also works well
for reducing iteration time as an AR developer.
Instead of having to record a video
for every single Android device you want to support
for every single scenario you want to test, you can record once
and playback on multiple different devices during your iteration phase.
This type of controlled testing environment is important for developers
like a partner unit which works consistently across many different Android devices.
Recording and Playback also shines when you need to reduce
the manual test burden across AR dev teams.
Instead of having to create custom datasets for every new feature,
our partners like Snap can leverage pre-recorded datasets
by launching new features that incorporate depth
or the latest tracking improvements from ARCore.
Recording and Payback can help save time for many AR app development teams.
And we are excited to hear your feedback.
I will now hand it back to Ian to talk
about how Recording and Playback helps your users directly.
Thanks, Nitin, the Recording and Playback API
removes time and space constraints for you as an AR developer.
But this API also unlocks new possibilities for your end users.
Think of all the times where AR could have brought to light or been helpful,
but the user wasn't able to access the experience
because they weren't at the right place at the right time.
Why should your user have to be standing
in their living room to see what virtual furniture looks like there?
Now they can record their living room once and then add virtual furniture to the scene
whenever they feel like it.
For users, this unlocks a new class of AR capabilities
that we’re calling post-capture AR experiences.
To help illustrate the power of post-capture, here are some examples from our partners.
Nexus Studios just launched a new application
called VoxPlop, available today on Google Play.
Leveraging Recording and Playback, the VoxPlop app allows multiple users
to co-create spatial videos.
Users can drop voice messages associated
with delightful characters into a pre-recorded space
then send this video to their friends.
Their friends can watch and listen as these virtual characters interact
with their physical surroundings, and they can also edit
in their own voice messages with their own characters.
The ability to co-create AR experiences
is a wonderful example of the new possibilities available
with Recording and Playback.
Another one of our partners,
South Korea Telecom, recently leveraged Recording and Playback
to launch a new feature via their Jump AR application
available today on Google Play.
Users can record a given environment, then,
at their leisure add delightful AR characters and effects and share them with friends.
Notice how the user here doesn't need to be
at a certain place or a location with their phone out,
AR app open, and a live camera session on, but they’re still able to experience AR.
By removing that live camera session requirement,
users are able to unlock new capabilities for AR editing,
allowing them to create and access unique AR content at any place and time.
That brings us to the end of our session.
Thank you so much for taking time out of your day to join us.
Before you head out, here’s a few ideas about where you can learn more
and get hands-on with Recording and Playback.
We built Recording and Playback with you, the developer, in mind
and we can’t wait to see what you come up with.
If you have any demos you’d like to raise to the team’s attention,
please tag #ARCore and #RecPlay on Twitter.
If you're interested in hearing about AR core overall
and what’s new beyond Recording and Playback,
check out the session called New Capabilities in ARCore.
If you’re interested in seeing documentation,
sample apps, and demos for Recording and Playback,
check out www.developers.google.com/AR.
And if you'd like to get hands-on, please follow the link
in the video description to check out the codelab,
which will take you step-by-step through the API.
On behalf of the entire team, thank you so much for tuning in.