[MUSIC PLAYING]
DOM TURNER: My name is Dom.
I work in the Android DevRel team,
and I'm here today to talk to you about low latency
audio on Android.
So why is audio latency important?
Well, there are a whole raft of use cases
where the audio latency is directly
proportional to the user experience.
So this can be in games where you tap on the screen
and you hear a sound, particularly in rhythm games
like Guitar Hero-style games where
you're tapping in response to some rhythmic event
and you need to have audible feedback as quickly
as possible-- like, the longer the delay between tapping
on the screen and hearing a sound,
the worse the user experience.
Also, in DJing apps, you are tapping on the screen
and manipulating audio, and you expect that audio manipulation
to happen in real time.
Karaoke, where you've got an input, which is your voice.
And generally, you're listening to a backing track and also
your own voice.
So if the delay between you singing and hearing
your own voice is too long, then it sounds awful.
Also, in VR, we have objects which
make sound in a virtual world.
And if that sound doesn't follow you around
as your head moves in this environment,
then it kind of distorts the reality.
And lastly, of course, there's a whole load
of apps for live performance-- synthesizers,
drum machines, looping apps.
Anything which you press a key and make a sound,
you need low latency audio.
So with this in mind, we built a library
to help developers build these kind of apps.
It's called Oboe, and it's available on GitHub now.
We just launched Version 1, so it's production-ready,
able to be included in your apps today.
And the way it works is, under the hood,
it uses the AAudio API on API 27 and above,
which is the new high-performance, low latency
audio introduced in Oreo.
And on older devices, it uses OpenSL ES.
So basically, the idea is provide
a simple, easy-to-use API which works across the widest
range of devices.
So rather than me talk about this API,
I thought it'd be fun to try and build a musical instrument
in 17 minutes and 23 seconds.
So before I start that, I'm just going
to explain the architecture so it kind of
makes sense when I'm in the code.
So we have an app, and I'm going to build the audio engine
part of this app.
This audio engine is going to be responsible for creating
an audio stream.
That's provided by the Oboe library.
And we're going to be passing audio frames of data
into this audio stream.
Ultimately, this stream is responsible for putting data
out of the audio device.
In this case, it will be the 3 and 1/2 mil jack
on this Pixel XL phone.
Now, every time the audio device needs more information,
it's going to give us a callback.
So we get this callback loop of, hey, I need more audio data,
and our audio engine is going to be passing frames of audio data
into the audio stream.
For some form of control, we're going
to monitor tap events on the screen.
So when you tap down, the sound will come on.
When you lift up, sound will go off.
This worked about 50% of the time in rehearsal,
so let's see what happens.
OK.
First problem, I need to log in.
Can you see my screen?
Fantastic.
OK.
So I'm just going to run the app.
So I've kind of got a very simple shell of an app.
It doesn't really do anything at the moment,
but it just has a few little shortcuts
that make it possible for me to do this
in a very short amount of time.
So I will just run the app on this Pixel XL.
And hopefully, you'll be able to see that it does nothing.
So here we go.
Here's the app.
And when I tap on the screen, nothing happens.
No sound comes out.
It's non-functional.
I just want you to know that there's
no smoke and mirrors there.
It is genuinely live.
[LAUGHTER]
Thank you, Glenn.
OK.
So we're in our main activity here,
and we're going to talk to the Oboe library via JNI.
So I have a couple of native methods
here which I'm going to use, and I'll implement those methods
in a second.
So we're going to create an audio engine,
and we'll start by calling start engine.
So let's just jump into our JNI code.
So this is in native-lib.cpp here.
So I'm going to define an audio engine up here--
just call it engine.
And then, I'm going to call a method
on my engine called start.
Now, I've already created the header and implementation
files--
just the blank files for this audio engine class.
So I'll go ahead now and write the class.
So AudioEngine, and I'm going to have one method called start.
OK.
Now, I can use Option-Enter to generate
the definition for this.
I'm in my implementation, and I'm in the start method.
Before I can start using the Oboe library,
I need to do two things.
Number one, I need to include the Oboe header.
There we go.
And the other thing I need to do is--
this kind of just makes it easier for me-- which
is to use the Oboe namespace.
And this just avoids me having to prefix all of the Oboe
objects with the word Oboe.
So in our start method, we want to create an audio stream.
To do that, we use an audio stream builder.
[? Call ?] [? up ?] [? b. ?] And the builder allows us to set
properties on the stream.
So that's things like the format of the stream.
Now, when I set the format, there
are two choices I can choose from--
either 16-bit integers or floats.
I will use floats.
I can also set the number of channels.
So that would be two for stereo or one for mono.
And I can also set properties which
inform the stream of my latency requirements.
So the most important one here is set performance mode.
And there are a number of options, but the one I want
is, obviously, the low latency one.
The second thing I can do is set the sharing mode on the stream.
So let's just set that.
So I'm going to set this to an exclusive sharing mode.
And what that means is that I'm requesting
that the audio device give me an exclusive stream.
That means that my app's audio won't
be mixed with any other audio on the system.
So if the device supports it, I can
avoid having my audio mixed with anything else
before it goes through the speaker or headphones.
And by doing this, I can cut a few milliseconds of latency
off the output.
So that's all I need to do to open the stream.
So I can go ahead now and call open stream.
This takes a reference to an audio stream pointer.
Again, I can use Option-Enter to create
a new field called stream.
So back in our header, it's done this automatically for me.
OK.
Once the stream is open, there's one final step I need to take,
which is to set the buffer size on the stream.
So I can do this by doing set buffer size in frames.
Now, to get the smallest possible buffer size,
we have to interrogate the audio device for the minimum amount
of data that it will read in one operation-- so a discrete chunk
of audio data.
And we call this a burst.
So we want to obtain the burst size from the audio device.
We can do this using stream get frames per burst.
And that's the minimum possible buffer size
that we can set our stream to have.
But we actually don't recommend that you
use this absolute minimum.
We recommend that you use double this because it provides
a good protection against underruns,
and it's a good trade-off with latency and underrun.
OK.
So that's all I need to do to create a low-latency stream.
So now, I can go ahead and start the stream, which
will do nothing because we haven't
found any way of putting data into the stream.
So to get data into the stream, we use a callback.
So I'm back in the builder, and I'll
use the set callback method.
This wants to take an audio stream callback object.
And to make this simple, I'll just
use my this object, which means that my audio engine needs
to implement this interface.
So we'll have it do this.
Now, I can use Control-O to show me the methods
that I can override in this interface.
The one I want is onAudioReady.
This is the method that will be called
every time the audio device needs more data.
So inside here, let's just take a quick look
at what this method signature is.
So onAudioReady is called.
It tells me the stream that wants more data.
And it also gives me a container array.
So this container array, which is of type void star
because it can be either 16-bit integers or floating point
samples, is something that I'm going
to write my audio data into.
So I write into it, and then that's
passed out to the audio device.
Lastly, we have num frames, which tells me
the number of audio frames which need to be populated
in this container array.
So I need an audio source.
And I'm going to cheat a little bit here.
I have created an oscillator in advance.
And let's just take a quick look at it.
And it's just going to generate a square wave here.
So that's a periodic signal varying between two points
to create a square wave.
So let's now create an oscillator.
Let me just do oscillator.
It's a templated object, so I need to tell it what type.
And I'll just include the oscillator header.
So now that I have an oscillator,
I can do osc render--
OK.
So Android Studio is complaining about this method signature.
That's why it's got a red underline here.
I'm just going to go ahead and build,
and that normally sorts it out.
Yeah, there we go.
OK.
Ignore the errors.
OK.
So on my oscillator, I have a render audio method,
which is going to put the audio frames from the oscillator
into this audio data array.
So first thing I need to do is cast it to an array of floats--
audio data-- and pass in the number of frames.
So the last thing I need to do in onAudioReady
is return a result. And this can be either to continue,
where the callbacks will continue, or it can be stop
and the callbacks will stop.
So in my case, I'm going to continue.
Right.
The final thing I want to do is set some things up
on my oscillator.
So I'm going to set the amplitude, which
is kind of the volume.
I'm also going to set the frequency.
Set that at about 80 Hertz--
nice bass frequency.
And I also need to set the sample rate, which
tells the oscillator how frequently these samples should
be produced.
And I can get that from the stream--
get sample rate there.
OK.
I know you're all desperate to hear a sound.
There is one very final thing I need to do here,
which is I need to respond to tap events.
So I'll just go back into MainActivity,
and I'm going to override the onTouchEvent so that,
if the event is down, then I'm going
to call this native method tap.
Make that true-- else.
Otherwise, if the event is up--
I'm lifting my finger off the screen--
then I'll pass in False.
OK.
Let's just have a look at this tap method.
This needs implementing.
So I'll just pass in this true or false value.
Create the new method.
And I'll just call osc setWaveOn,
and that's going to pass that in there.
Now, moment of truth.
OK.
I'm going to run this.
And in theory, you should hear a sound.
So when I tap on the screen, we should
hear an 80-hertz square wave.
The pressure.
[VIBRATING SOUND] There we go.
[APPLAUSE]
[VIBRATING SOUNDS]
So you can see it's the lowest possible latency here,
and it's actually pretty responsive.
So we have a musical instrument.
Admittedly, it's not the best musical instrument
in the world.
A little bit to be desired on the sound and control front.
So what I actually thought would be nice
is if we could add a little bit more control.
So for the last four minutes and 30 seconds,
I'm going to tie the motion sensor
to the pitch of the oscillator.
So to do this, I'm going to cheat, uncomment
some code that I wrote earlier.
And what this does is register a listener,
which will listen to the rotation vector sensor.
So to listen to these events, I need
to implement the sensor event listener interface, implement
the methods onSensorChanged.
So what I want to do is set the frequency
based on the event values of the x-axis.
And I also need to scale this value.
So I want to have it from, let's say, 20 Hertz--
the limit of human hearing--
and let's go up to like--
this should give me up to around 100 Hertz.
Yep.
That looks good.
Again, I just need to implement this frequency.
OK.
So we'll just go osc setFrequency.
And there we go.
OK.
So we're good to go on that.
Brief interlude-- very brief, in fact.
Has anyone heard of the Amen break?
OK.
One person.
Yes.
So the Amen break comes from a song in the '60s by the Winston
Brothers called "Amen Brother," and it's
four bars of the most incredible drum solo.
And it's the most sampled loop in history,
but no one's heard of it-- evidently-- apart from one guy.
So I thought it would be nice if I
could play my new musical instrument over this loop.
So here's the loop.
I need to run the app.
Let's give it a go.
Let's just make sure it's here.
OK.
So with a bit of luck.
[MUSIC PLAYING]
Here we go.
[VIBRATING NOISES OVER DRUMBEAT]
Right.
[APPLAUSE]
OK.
So that's about it from me.
If we can just go back to slides?
And yeah.
So the library is available on GitHub.
There's documentation, code labs, all sorts
of good stuff on there.
We'd love you to start using it, start
building amazing audio experiences on Android.
And I'll be around afterwards if you have any questions.
So thank you all very much.
[APPLAUSE]
[MUSIC PLAYING]