[MUSIC PLAYING]
FRANCOIS GOLDFAIN: Hi, my name is Francois.
And I am the Engineering Director for Android Media.
KARTHIC VEERA: Hi, I'm Karthic, the Product
Manager for Android Media.
In this doc, we'll discuss two main topics.
Premium media experiences, like Spatial Audio and HDR
on Android devices.
FRANCOIS GOLDFAIN: And the latest Jetpack Media3 release,
which helps developers build these premium experiences.
KARTHIC VEERA: Let's start things
off by introducing some of the new premium media experiences.
Spatial Audio is one of the premium Android features
we launched in Android 13.
It creates a more immersive listening experience
by placing sound around you in a virtual space,
so you feel like you're in the middle of the action.
We started by adding support for movie and TV content
in Android 13 and plan to bring in more use cases, including
music, in the future.
Device implementations have to be carefully tuned
to ensure the spatial audio experience is
more engaging and preferred compared
to the traditional stereo experience.
This requires a spatializer to render an immersive experience
while respecting the creator's intent,
head tracking support with low latency
between the mobile device and headset, and flexible APIs
for developers to detect capabilities
and control the spatial audio experience.
Android is committed to delivering a high quality
Spatial Audio experience to our users.
We have been working closely with our ecosystem partners
to ensure the technology integrates well
and in particular, the platform can support important formats,
both current and future, the Bluetooth and audio stacks
are optimized, and the spatializer is tuned correctly.
And users can have access to the premium content they love.
You can experience this quality on Netflix.
By working closely on spatializer tuning
and head-tracking latency, we have
ensured users can enjoy immersive content that
meets Netflix standards.
Check out our developer site to learn more
on how to implement Spatial Audio in your app.
If you're using a recent version of ExoPlayer,
all you have to do is start streaming Multichannel Audio.
Another audio feature that helps enable a high quality
music listening experience is lossless audio.
Lossless audio refers to audio that
has been compressed with little to no loss in quality.
This is becoming increasingly popular with content providers.
So in Android 14, we are adding better support
for playback of lossless and hi-res audio formats
over wired USB headsets.
Delivering pristine quality audio USB can be a challenge.
For example, the audio framework may narrow down the bit depth
or perform resampling, which can introduce distortions
and reduce audio quality.
The audio framework also lacks standard APIs
to support lossless formats, like MQA or DSD.
To get around this lack of support,
you have to directly control the USB device
and bypass the audio framework.
This in turn creates more challenges,
like security issues, needing additional user permissions,
and the inability to play ringtones or alarms over USB.
To address these problems, we have added a few new APIs
in Android 14, starting with the AudioMixer Attributes Class.
This class represents the format in which the AudioMixer should
output audio and the behavior it will use to mix audio data.
You can pick between two mixer behaviors, DEFAULT, meaning
that the platform will behave as normal and mix
audio from different sources; and BIT_PERFECT,
meaning the audio data will be sent directly
without any additional processing.
You can use this new BIT_PERFECT behavior to stream high quality
audio, such as DSD over PCM on Android.
The Audio Manager API has also been updated to better support
these use cases.
With getSupportedMixerAttributes,
you can query a connected USB device for its supported mixer
attributes.
With the add-on PreferredMixerAt tributesChangedListener,
you can attach a listener to react
to changes in the PreferredMixerAttributes.
And with set PreferredMixerAttributes,
you can configure specific mixer attributes
for the Audio Manager to use.
Note that for lossless playback, your audio format object
must match what you set with SetPreferredMixerAttributes.
Moving on to premium video experiences,
HDR technology provides a more realistic and delightful
viewing experience by enabling more vibrant colors and greater
range of brightness.
HDR is becoming increasingly popular.
Many premium Android devices support HDR technology
for rendering and capture.
In the Android 13 release, we standardized
support for HDR video.
And in Android 14, we are bringing HDR support to images.
We are excited to announce Ultra HDR, a new technology
to enable HDR images on Android.
With Ultra HDR, users can capture, view, edit, and share
their photos in HDR quality on supported devices.
One of the biggest challenges with introducing new formats
is the potential for AppCompat issues.
Ultra HD is fully backward compatible with JPEG.
This means that images render as standard JPEG images in SDR
on devices and apps that do not support Ultra HDR.
It is important to note that HDR photo
capture, available on cameras today,
is different from Ultra HDR feature coming in Android 14.
HDR in the context of cameras typically
refers to some sort of exposure bracketing to capture
a high dynamic scene.
The captured image is then mapped
to an 8-bit standard dynamic range for saving to a file.
This means the image is only rendered
in standard dynamic range when being viewed.
In Android 14, the Ultra HDR feature
enables the captured image to be saved
in its original 10-bit high dynamic range.
So when the file is viewed, it can be rendered in HDR quality
on HDR screens.
Since many premium Android devices already
have HDR screens, you can deliver these visual quality
improvements to your users as soon as Android 14 rolls out.
Ultra HDR includes a 8-bit SDR JPEG
and extends it with a gain map layer
to update the image to HDR.
This means the image can still be rendered in 8-bit SDR
on SDR screens or when the viewing conditions
are more appropriate for SDR.
The device camera and decode pipeline
must be enhanced for 10-bit capture and HDR rendering.
We are working closely with our Android OEM and SOC partners
like Qualcomm to optimize and take
advantage of hardware capabilities
to support Ultra HDR experiences in stunning visual quality.
We want to ensure Ultra HDR is widely adopted
so users can enjoy their photos in HDR quality
across their devices.
In Android, we expect Ultra HDR to be
adopted as a default format in camera apps
and through APIs for in-app cameras
on premium devices running Android 14.
The platform will natively support rendering of images
in HDR on supported devices.
We are also working with Chrome and key industry players
to make Ultra HDR available on other form factors.
We are also excited to announce Google Photos support
for Ultra HDR.
Ultra HDR was developed in close partnership with Photos
to ensure it met the high quality
standards of Google Photos.
Photos users will be able to view, edit,
share with family and friends, and reminisce on their memories
in Ultra HDR quality.
We believe Photo support for Ultra HDR will
encourage industry adoption and improve the quality of photos
that users consume and share every day.
To learn more about capturing images in HDR, watch the How to
Build High Quality Camera Experiences session.
Unlike HDR video, migration to surface view
is not a requirement.
And HDR images can be rendered directly in Image view
alongside SDR assets.
Apps can opt in to render images in HDR
with a simple one line change to the manifest file,
or by setting the Color mode in the Activities
on Create Method.
You can also use standard Android APIs, like the Bitmap
class, with Ultra HDR images.
We encourage developers to use these APIs to add HDR image
support to your app.
[MUSIC PLAYING]
FRANCOIS GOLDFAIN: All these premium platform capabilities
are exciting.
But developers are often challenged
by how difficult it is to build media experiences that
will delight their users.
We understand the importance of having great developer APIs
to mitigate this complexity.
And I am excited to share the latest updates on our Jetpack
Media3 Support Library.
First, let me acknowledge some of the major trends we've
been seeing in media usage.
Watching long form video content in full screen
remains one of the most popular use cases on mobile.
But user experiences with short form video content
have become hugely popular these past few years
and have spread to all types of applications, from social media
apps to news apps to apps that previously only delivered
long form content.
We see user experiences that include an unlimited feed
of bit-sized videos.
Furthermore, short form content is now
created not only by professionals but also
by any user directly from their mobile device.
Apps strive to give users a full suite of tools
to capture photos and videos, edit them
with effects and audio tracks, and then share them
to other users, often over weak or unreliable
cellular networks.
Building such media experiences with quality
has become increasingly complex, especially
as newer capabilities, such as HDR, are introduced.
But they are needed to meet growing user expectations.
To address this complexity, we have
been working on Jetpack Media3, a support
library that is now the home for all high level media APIs.
We introduced Media3 in alpha at the Android Developer Summit
in 2021.
And after a few beta releases, we launched it as stable 1.0
earlier this year.
Media3 APIs are designed to be simple yet highly customizable,
reliable, performant, and optimized for each device
capabilities.
This abstracts away much of the complexity
that comes with device and OS fragmentation.
Media3 includes ExoPlayer, our best in class player
which is now directly integrated with Media's session, making
it much easier to expose playback
controls to other surfaces, such as Android TV
or connected Wear OS devices.
All new ExoPlayer enhancements will
be part of Media3 moving forward, including optimization
for short form content and support
for use cases beyond playback, such as transcoding and video
editing.
Now that ExoPlayer is part of Media3,
we strongly recommend all developers
that currently use ExoPlayer to switch over to Media3.
Migration is easy.
For developers who are already running the latest
version of ExoPlayer, it's just a matter
of changing the package names.
And we even provide a script to automatically take
care of that.
Apart from the package name, the implementation
is strictly identical.
One benefit of Media3 is that includes a stable API
surface to provide you with a strong guarantee of APIs that
won't change in subsequent releases
and make future upgrades even smoother.
We are currently in a transition period, during which we're
releasing ExoPlayer both as part of Media3
and through the standard ExoPlayer GitHub.
But we plan to stop doing this later this year.
So please, switch over to Media3 to take full advantage
of new and upcoming features.
As mentioned earlier, Media3 covers use cases
beyond playback.
This includes the Transformer APIs,
which is a solution for media transcoding and editing.
These operations are normally complex to implement,
especially with great performance and quality.
Yet they are critical for building an app experience
with user-generated content.
Transformer currently supports transcoding from one media
format to another using optimal parameters
for the local device, handling of special formats,
such as slow motion videos and HDR videos
with support for tone mapping from HDR
to SDR, single asset video editing with an effects
framework.
We're also working on adding support for previewing effects
in ExoPlayer so that Media3 can then
be used as an end-to-end video editing toolkit.
Transformer API is full of patterns similar to ExoPlayer.
They are very simple to use for built-in functionalities,
and yet they can also support advanced behaviors
by letting developers inject customized modules instead
of using defaults.
Let's take a look at a few examples.
First up, a simple transcoding example.
Begin by building a transformation request
with a MIME type of the desired video output.
In this example, we just want to transcode a video clip
to the H264 video format.
Then build the Transformer, passing in the request
and the listener.
The listener has callbacks where you
can track progress and handle possible error cases.
Finally, start the transformation,
which happens in the background thread.
In this second example, we want to tone-map an HDR video
to SDR.
We just need to change the transformation request
accordingly.
And everything else is the same.
Here's an example of video editing.
Transformer includes many built-in transformations.
So in this example, we've just changed the transformation
request to be a 180-degree rotation, and also
a horizontal flip of the video.
Lastly, here's an example of a custom effect
where we'll use Transformation matrix to zoom
into the video over time.
Start by subclassing metrics transformation.
Inside, we'll instantiate a matrix.
And then map the provided presentation time
onto a scale matrix.
This video effect can then be passed to the transformer,
similar to this transformation request
in the previous examples.
For even more advanced use cases,
you can also plug in a GL texture processor
into Transformer as a custom effect
and use custom OpenGL code.
Please check out the Transformer Demo App
on the Media3 GitHub for more examples and to try Transformer
out for yourself.
We are actively working on building more capabilities
into Transformer, including multiasset editing.
So please stay tuned for more updates.
All right, so let's recap.
Media is a critical area on mobile.
And user expectations grow every year,
not only to consume, but also to create and share
their own content.
We are investing in both bringing premium media
capabilities to Android and in helping
you take advantage of them with simple Jetpack solutions.
KARTHIC VEERA: So we can't wait to see
the delightful media experiences you're going to build.
Thank you for your time.
FRANCOIS GOLDFAIN: Thank you.
[MUSIC PLAYING]