LAGOS MOLNAR: In today's world, video is everywhere.
You are an app developer, and you want to wow your users.
But how do you create the detailed, fine-tuned media
magic that you have envisioned?
I am Lajos, from the Android Video Framework Team,
and I'm going to talk about how to create great, even
impressive multimedia experiences on Android.
Sure, Android has great high-level media APIs.
But if you want to do something other than playing or capturing
video, or you need a feature that is not
supported by MediaPlayer or MediaRecorder,
you will need to get down and dirty
with the nuts and bolts of Android
Media-- the low-level APIs.
They consist of the MediaCodec that
is used to both decode-- compress audio and video--
and to encode raw audio and video.
MediaExtractor parses media container files
into audio and video packets.
Conversely, MediaMuxer packages audio and video packets
into media container.
MediaCrypto and MediaDRM are used
to decrypt protected media content
and to manage the digital rights.
You can do everything that MediaPlayer and MediaRecorder
does using the low level APIs.
I know it, because we did it.
For example, to build a media player,
you need two MediaCodec objects and a MediaExtractor
to parse the individual audio and video packets out
of the data source.
To play back secure content, you will need a MediaDRM object
to manage the secure session.
And you will need to provide a MediaCrypto
object to the video decoder.
You can implement MediaRecorder using
two MediaCodecs and a MediaMuxer.
In fact, you can build a wide variety
of multimedia applications.
And, might I add, quite cool ones.
Then you combine these media blocks
with others, such as textures and texture shaders.
For example, this is how you could do video editing as it is
done in Auto Awesome Movie in Google+.
Decode the two source videos onto GL Textures.
Composite them using GLES Shaders
onto the input surface of a video encoder.
Viola.
That was easy.
So how does MediaCodec work?
MediaCodec, at its core, is a signal processor
that produces output data from input data.
It is processing data asynchronously
and uses a set of input and output buffers.
At a simplistic level, the client requests an empty input
buffer, fills it up with data, and sends it
to the codec for processing.
The codec uses up the data and transforms it
into one of its empty output buffers.
Finally, the client requests a filled output buffer,
consumes its contents, and sends it back to the codec.
The real operation of a codec is a little more complex
as each of these operations is happening in parallel.
Also, codecs that produce raw video buffers
can be connected to a surface.
Let's go over the states of MediaCodec.
When you create a codec, it starts
in the uninitialized state.
First, you need to configure it.
When you call Start, the codec moves to the flushed state
where it holds all the buffers.
As soon as you dequeue an input buffer,
the codec moves to the running state.
It spends most of its useful life here processing
input buffers, generating output.
If you queue an input buffer with the End of Stream flag,
the codec moves to the End of Stream state.
Here it no longer accepts further input buffers,
but still generates output until the End of Stream buffer
is reached.
You can move back to the flushed states
from any of the executing states by calling Flush.
Or call Stop to go back to the configure state.
If you call Reset from any state,
the codec moves back to the uninitialized state.
On rare occasions, the codec may encounter an error
and move to the error state.
This is communicated using an invalid return
value from a queueing operation or sometimes via an exception.
Call Reset to make the codec usable again.
After you finish using a codec, you
have to release it by calling Release.
Boom.
By default, codecs use [? buffers ?]
for both input and output data.
However, video codecs can be configured
to use hardware accelerated surface buffers for raw video.
This is the preferred mode to operate hardware
accelerated video codecs as it allows
them to work without memcpy.
In either mode, MediaCodec uses buffer indices
to refer to buffers.
You get the index of an empty input buffer
by calling Dequeue Input Buffer.
You can then fill the buffer with data,
apply a time stamp or flags, and sand in to the codec
by using Queue Input Buffer.
To get a filled output buffer, call Dequeue Output Buffer
to get its index.
Note how to time stamp is propagated from the input
buffer.
If using [? buffers, ?] you can then process the data
and release the buffer by calling Release Output Buffer.
A couple of cautions.
Although buffers are referred to by indices in a buffer array,
not all indices may be valid so there is really
no indication of the number of buffers used.
All output buffers are read only.
You should only access a buffer by your holding
onto it with Dequeue and Queue, or Release Buffer calls.
Though buffers are handled asynchronously,
release each buffer promptly.
Doing otherwise may stall some codecs.
As mentioned before, you can set up a video encoder
to process hardware accelerated surface input.
You do this by calling Create Input Surface after Configure.
Pass this surface to the producer
of the video frames which will now directly talk to the codec.
Similarly to the normal operation,
the producer will request and empty input buffer,
fill it with data, apply your time stamp,
and send it back to the codec for processing.
When you want to stop the encoding session,
call Signal End of Input Stream to apply the end of stream flag
to effectively the last frame received.
Likewise, you can configure video decoders
to use hardware accelerated output buffers
by specifying an output surface in Configure.
Use the familiar Dequeue Output Buffer
to get the index of a filled output buffer.
However, in contrast to [? buffers, ?] you cannot
inspect the contents of surface output buffers.
You can only decide whether or not
to send a buffer on to the output surface.
To discard it, call Release Output Buffer with render
set to False.
Or sets render to True to display
the buffer on the output surface.
In the L Developer Preview, the added support to optionally
set the time stamp of a surface buffer about to be rendered.
If set, some buffer consumers will wait for a time stamp
to pass before using that buffer.
We have recently added a few features
to make MediaCodec more useful.
One of these, as demonstrated briefly before,
is the ability to precisely schedule
the display of video frames.
Up until now, A/V sync has been difficult to achieve.
With the addition of surface time stamp
support in MediaCodec and high-resolution time stamp
support an AudioTrack, this is now straightforward.
Scheduling of video frames only works on surface view.
Use the precise audio time stamp to calculate the exact system
time when you want the video frame to appear on the display.
Then set the time stamp by calling Release Output Buffer
about two VSYNC periods prior.
The frame will be shown at the first VSYNC
after the requested time stamp.
Another recent improvement to MediaCodec
is the support for adaptive playback.
Adapted playback is an optional feature of video decoders
that enables seamless change in resolution during playback.
Whereas the client can start to feed the decoder,
input video frames of a new resolution, and the resolution
of the output buffers change automatically
and without a significant gap.
It is only supported if the codec is configured
to use hardware accelerated surface buffers.
If a codec does not support adaptive playback,
you can still change the resolution.
First, mark the last frame of the old resolution
with an End of Stream flag.
Then wait until all frames are decoded.
Then stop the codec, or simply flush it
if using API level 19 or higher.
Then configure it for the new resolution.
Finally, start the codec again.
Wow.
And I used to think that was convenient.
Before you can rely on the adaptive playback features,
you need to verify that it is supported
by the codec you are using by calling [INAUDIBLE] Feature
Supported API.
You enable adaptive playback during codec configuration
using two special keys, max width and max height.
These for a hint for the maximum resolution that
has to be supported by the codec.
If the configuration succeeds, the codec
is expected to switch resolutions smaller
than the hinge in a seamless fashion.
Switching to larger resolutions is still possible dynamically,
but it may not be seamless.
Also, if you try to switch to a resolution that is not
supported by the codec, it will enter its error state.
A word of advice, to conserve memory,
use the smallest resolution hinge
required for your use case.
To change the video resolution while the codec is running,
simply cue a SYNC frame using Queue Input Buffer.
A SYNC frame is a special key frame
that also contains the configuration change
parameters.
For H264, this means that the SPS and PPS
are supplied together with the IDR frame in a single buffer.
Woof.
That was a lot of information.
We have published a streaming video player example
on the developer website that demonstrates
the use of the low level APIs as well as adaptive playback.
There's also video by [? Ollie ?]
that describes this sample in detail.
Be sure to check them out.
And for those of you who want more,
the low-level media APIs are also
available in the Android NDK.
These CAPIs mimic the Java ones and are
available for both 32 and 64-bit native binaries.
Sweet.
With that, I must say farewell.
I leave you with the challenge of building something
fantastic for your users using the Media Building Blocks.
For more information, visit the Android developer website
for API reference, guides, and sample applications.
Thank you for your time.