[MUSIC PLAYING]
MOZART LOUIS: Hi.
My name is Mozart Louis, and I'm a developer engineer--
developer relations engineer advocating for Android camera.
Today we're going to talk about improving
your social experience quality with Android camera.
Now, most of this talk assumes that you use the Android
Camera2 framework within your app
and you know concepts like the camera manager and the camera
device, as this isn't a getting started guide.
And features here will also be available in CameraX sometime
in the future.
So let's get started.
Camera is at the forefront of social.
Camera is such an important part of the mobile device,
and users are always interacting with the camera,
especially in the context of social.
In fact, we have over 60% of images in videos
are taken from apps other than the native camera app.
So you can think of these as social apps
that we know and love today.
Users are also spending an average of over two hours
on social apps daily, and most of the apps contributing
to that time have camera at the forefront.
Because of this and because of how important
camera is to the ecosystem, Android
has doubled down on things that matter most with camera,
especially in the context of quality and performance.
We want to assure users are able to take the highest quality
photos and videos possible and make every single pixel count.
We also want developers to be able to ensure the best
performance in apps and make them fast,
responsive, and good for everyday social and nonsocial
needs.
So we've taken feedback from developers
on exactly what they needed from camera,
and we've landed major changes in Android 13.
Today's agenda, we're going to explore some major updates
within the camera ecosystem, covering topics around capture
quality and performance.
And we'll take a look at how to implement these things
into your app today.
Let's talk about one of the easiest ways
to visually improve the quality of your video,
and that's HDR video capture.
In Android 13, we introduced high dynamic range video
capture, which provides higher contrast and more colors
to encoded video.
Your camera capture will be able to produce
much brighter videos and more saturated
colors compared to SDR.
Normally, high end features like this are made available
within the user's native camera apps, but what we've done
is made this available to any developer that wants
to tap into this great feature.
With HDR comes 10-bit color capture as well,
which enables you to capture a little over 1 billion colors,
instead of just shy of 17 million.
This should make colors more-- this should
make colors in videos smoother and more consistent
and avoid the dreaded banding that
can occur when colors are represented incorrectly.
There are a couple of steps on getting started
and integrating HDR within your app,
and the first step is to check for 10-bit dynamic range
support.
The first thing we'll do here--
the first thing we must do here is use our camera manager class
so that we can query some characteristics.
The characteristic that we're going to query here
is the request available capabilities.
This will return a list of capabilities
that the camera that is given here is able to do.
Now, the capability that we're looking for
is this flag here called request available
capabilities dynamic range 10-bit.
I know it's really long, but that's the actual string there.
Now, great.
Once you've returned that, if that's true,
we know that your device supports 10-bit.
And the next thing we want to do now
is try and get a supported HDR profile.
There's many available.
So in doing this, you do the same thing.
You take the camera manager, and you query a characteristic
based off the camera ID.
That characteristic, in this case,
is request available dynamic range profiles.
This will give you a list of profiles--
HDR profiles that are available within the device--
of dynamic range profiles available in the device.
Now, once you have that, in this case, in this example,
we're requesting for the dynamic range profile of HLG 10.
The reason why we're doing this here, we actually,
in this case, don't need to do this because in Android 13
HLG 10 is the minimum required profile for capturing playback
to provide greater consistency across devices
within the entire ecosystem.
So this means that if you have 10-bit support within your app,
then you must have HLG.
If the device has 10-bit support,
then HLG is definitely supported.
But we add this piece here because you may want to support
a different HDR profile, like HDR 10 or HDR 10+.
So we have 10-bit support.
We have an HDR profile.
So let's actually set up an HDR session.
Here, we're going to, like normally, we
would use in setup an output configuration for every target
surface.
In here, you'll see that we have a new flag
in the configuration.
The flag is called confic.dynamic range profile.
It's here that we can set up our profile
that we checked for that we ensured that we have.
So in this case, HLG 10.
From there, what we'll do is configure our session
as normally we would, as any camera developer would,
and we create a capture session.
And this is setting up our HDR capture.
So we have our session set up with the dynamic range profile.
And now, we need to actually encode the video into a file.
So here, the first thing we want to do
is create our media format instance
with a width, a height, and a format.
And in this case, we're going to use HEVC, which
I'll go into in the meantimes.
Now, we have a lot of flags that we need
to set to do this correctly.
In this format.set integer, we'll call this many times
with a lot of these keys.
And here they are.
You have four things to set up to get HDR properly set up.
You need the key format, the key profile, the key color
transfer, as well as the color standard.
So in this case, in this example,
our color format would be the format surface.
Our key profile will be HEVC profile main 10.
Our color transfer will be HLG, and our color standard
will be BT2020.
Once again, there is multiple options you can mix and match,
depending on what your device supports.
Great, so we have HDR fully set up
in terms of capturing HDR video within your app.
And I encourage everyone to stick around
for the next talk around HDR 10BIT video
to go from end-to-end, to capture, to playback,
and sharing.
Great.
So our next topic today we're going to talk about
is stream use cases, which can help optimize your camera
stream for common scenarios in which you'd
use your camera for.
So why do we need this?
Why is this important?
Well, before Android 13, under the hood,
the camera didn't really know why you want to use it.
So for instance, let's say you wanted to capture an image,
and maybe you want to prioritize image quality.
Well, the camera device didn't really know that ahead of time,
so there was nothing that the device manufacturer
could do to optimize for that particular use case.
How does this benefit the developers, though?
Well, when you're in session creation time,
the camera device will understand
what you're wanting to create this instance for.
So for instance, now device manufacturers
can optimize for the best performance or the best
quality, depending on why you're using this camera stream.
So you can think of an example as if I'm just
trying to take a picture, I want the highest quality,
every pixel that can count.
And so now the manufacturer can say,
OK, we're going to optimize for this use case scenario,
whereas in if you're doing just, let's say, an in-app analysis
and just using the viewfinder, you may not
need the highest quality, and you can then optimize
for performance instead.
So here's a list of some of the predefined keys.
Right now we have default, which is covering all existing
behavior.
We have preview, which is recommended
for viewfinder or in-app image analysis or AR,
augmented reality, things like that.
Still capture, which-- and video recording,
which is pretty self-explanatory.
Video call, which is recommended for long-running camera
sessions, where power drain can become a concern.
You have preview video still.
Now, this one is recommended by Google
for apps that are trying to use the one stream for everything.
So if you're using the one stream to capture images,
using the one stream to capture video and display it
to the viewfinder, use preview video still.
We also have vendor start for Samsung or any other OEM
to be able to have their own particular predefined cases.
All right, so here's a quick demonstration
of what it may look like.
This is a little dramatized.
But if you set a stream, let's say, to a preview,
if your stream use case is preview,
you may notice that there's a jump in frames per second,
and it may make the viewfinder feel a lot more real time
and more realistic because this is optimized for performance.
This increase in performance, though,
may lead to a loss of quality and a less desirable image
and maybe not really great for video,
whereas is view still capture and display
that stream on the viewfinder, you'll
see that it's like almost the highest resolution photo
that the sensor can produce every single pixel,
but the stream is not really expected to maintain
preview-like frame rates, so you may
see it's becoming very laggy, and it may not be something
that you would want to use for the viewfinder.
And so you would use this case for just the image
capture stream.
The first thing we got to do to get this implemented
is check if your stream use case is supported.
Everything must be supported by the device
manufacturer in order for you to use it within your app.
So we use our camera manager, and we
query a camera characteristic.
And in this case, we're going to query the scaler available
stream use cases.
So in this case, we'll go in here.
Well, let's say we created this method,
and we want to use one of these use cases.
We put it into a function like this to check,
is this available.
So once we've checked that the stream use
case we're looking for is available,
we want to now set that up during configuration.
So it's kind of the same situation as before.
Now, we're going to use the example of preview video still
here.
What we'll do is for each output configuration,
we have a new flag called config.stream use case,
and you will set up that use case to the configuration,
and that's pretty much it.
You will then continue as normal,
and that will help your camera stream for the specific use
case that you're using it for be optimized by the device
manufacturer.
Great.
Now let's talk about one of my favorite acronyms, which
is WYSIWYG, which is called What You See Is What You Get.
Now, you may think what is this, and why do we even have this?
This is for preview stabilization.
So now, if you're familiar with camera APIs,
you'll already know that we already have stabilization.
So what does this do?
If you haven't noticed that there are some cases when
you use a multistream where, let's say,
you have one stream directing towards the viewfinder,
and one stream dedicated to the encoder, what may happen
is that what's being seen inside the viewfinder
is not equivalent to what gets captured in the encoder.
So sometimes that will result into a sudden crop
into the image or a sudden--
there's more viewfinder-- there's
more in the captured image than what
was in the viewfinder which is unexpected,
which is not really good for apps
that you want to see what you get.
You saw what you saw, and you didn't
get what you wanted to get, in that case.
So we've implemented preview stabilization,
which helps avoid this issue by applying
the same exact stabilization that
is going to the video encoder to the actual preview, which means
that it can crop in, crop out, whatever it may be doing
to the encoder as well will be applied to the preview, which
will give you the "what you see is what you get"
feeling, which is really important for things
like social apps, where they usually will use just one
stream in this case.
Implementing this is actually really easy.
Like everything else, we need to query a characteristic
of the camera and see if it supports the control available
video stabilization modes.
Once you get this list, you will be
able to then check if we are able to support
control video stabilization mode preview stabilization.
Now, once you get this, if this is true,
you can set it up in your builder with the capture
request, and then you'll have previous stabilization
happening within your app, just like that.
Great.
That's the only thing you need, pretty much
this flag and checking it and setting it within your builder,
and you will be up and running with preview stabilization.
The last thing we'll talk about today is camera extensions.
Device manufacturers can expose special capabilities,
like night mode, bokeh, HDR through third party--
to third party developers through camera extensions.
Developers can use either the Camera2 API or the CameraX API
to enable camera extensions, which are implemented
by the device manufacturer.
For this example, you can see how Snapchat uses the night
extension to brighten up a scene and take a better
image under low light conditions on Pixel devices.
And you can see how I was able to get a nice bokeh
depth of field effect on the Samsung Galaxy
S22 that supports extensions.
Now, we're excited to say that over 160 devices now
support at least three or more extensions,
and device manufacturers are hard at work
implementing and supporting extensions
on more and more devices.
Here's our list of extensions that are currently available.
You have nights, which is useful for low light conditions.
HDR, which is good for high dynamic range still capture.
Auto, which automatically selects
one of these other extensions based off
of the in scene environment and what the device manufacturer
determines at that time.
So you can imagine setting up auto
and let's say it's really low light conditions,
they may then say, OK, we're going to switch to the night
extension under the hood.
We also have bokeh depth of field,
which is a focused subject with a blurry background,
and face retouch.
Like everything else in camera, we
must query and ensure that the camera is
able to support the extension.
So with a method like this, you can
check if a specific extension is enabled
based off of the camera ID.
So let's see an example of creating a session
for the bokeh extension.
The first thing we'll do is check
if the extension is supported with our previous method.
So in this case, we'll get the camera device
ID, which may be the back camera or camera manager.
And let's say in this case, we're
looking for the bokeh extension.
If this is supported, then we'll be
able to continue in this function.
Then we'll set up our camera extension session state
callback.
This is where you can call the OnConfigure
and OnConfigureFailed, if everything works well.
And one of the key differences here
is that we don't create a normal session configuration.
We create a extension session configuration
instead with the extension callback and everything
else available there.
And when you're calling it on your camera device,
what you will use instead is create extension session,
instead of just create session--
create session with your configuration.
Let's recap.
Today, we talked about HDR video,
which can improve the color, brightness,
and contrast of your videos.
The camera stream use cases, which
can optimize the camera for specific use cases.
Preview stabilization, which assures your users
get exactly what they saw in the viewfinder.
And camera extensions, which is access to special capabilities
from the manufacturer of the device.
Now, there's so much more, actually,
that we weren't able to necessarily cover today,
like concurrent camera, jitter reduction, updates to CameraX.
But we encourage you to follow the links here
to be able to view all of these extra updates.
And the camera team is looking for as much feedback
as you can give us for the next version of Android,
so we'd love for you to just keep in contact with us
and let us know what we're doing that we
can improve the platform and camera
experience within your app.
Thank you.
[APPLAUSE]
[MUSIC PLAYING]