PETER BRINKMANN: Hi, I'm Peter Brinkmann, an engineer on the
speech infrastructure team in New York.
And today I'd like to tell you about 20% project, the
Patchfield Audio Architecture for Android.
In a nutshell, the idea is to allow you to implement audio
functionality, such as synthesizers and effects, with
minimum fuss and to patch them together in a Patchfield as
you would in a recording studio.
The core of Patchfield is a service that manages the
signal processing graph.
And it also acts as a central hub for audio.
So the service manages all your units, such as
synthesizers or effects.
And it provides all kinds of functionality
for connecting them.
Now you can run the service locally
in your app or remotely.
When it's running locally, then essentially it gives you
a powerful way of organizing audio within your app.
And it gives you also a couple of other nice features, such
as support for formatted processors.
When it's running remotely, you get an added feature.
Namely, you get support for interactive audio routing,
meaning that, say, you might have one that just acts as a
synthesizer.
You may have another app that acts as an effects
[INAUDIBLE].
You can connect them, and then audio signals will flow from
one app to another, and you get a lot of
flexibility that way.
Now in terms of APIs, there are two APIs
for you to work with.
There's a client API in Java.
And the purpose of that Java API is to let you query and
manipulate the signal processing graph.
So you can get a list of the audio modules that are
currently available.
You can connect audio modules.
You can disconnect them.
You can enable or disable them.
And you can do a number of other things as well.
And the second API is the audio module API itself.
That one works with Java as well as C. And that's what you
use for implementing new audio functionality, such as, well,
synthesizers and effects.
All right, let's bring up a little control app for
Patchfield.
So right now you see we have an on/off button for enabling
or disabling audio processing.
And right off the bat, we get two audio modules.
We get one for the microphones of the device, so we see one
audio module that has two outputs, one for each of the
built-in microphones.
And we see another audio module with two inputs.
And those are the inputs for the left speaker and for the
right speaker.
And for example, I might say connect one of the microphones
to one of the speakers.
And if I were to turn audio processing on right now, we
would get horrible feedback.
So let's do that.
Instead, let's launch another app.
So this one is a very simple app that just loads a wave
file and gets ready to play it.
And when we go back to our control app, we see that a new
audio module has appeared.
So now we have an audio module for our wave file.
And let's bring up yet another little app.
So here this app provides a low pass filter.
And once again, if I go back to the control app, there's as
a new module right here that represents
our low pass filter.
And now we can connect them.
I send the audio from the wave file into the low pass filter.
And I send the output of the low pass
filter to the speakers.
And now if I turn on audio processing, we
hear the wave file.
Now we can go to our low pass filter.
And if we play with the cut off frequency, you can tell
that the higher frequencies go away.
So the low pass filter acts as expected.
So that was the core functionality of Patchfield in
a nutshell.
All right, let's take a look at the code.
The Patchfield service implements an interface called
iPatchfield Patchfield service.
And this interface consists of three types of methods.
You have methods for manipulating the state of the
Patchfield service.
You have methods for querying the state of the service.
And you have methods for internal use.
So when you're manipulating the state of the service, for
instance, you can turn on or off audio processing.
That's what the on/off button in the control app was for
that we saw a minute ago.
You can activate and deactivate modules.
You can connect ports of molecules or you can
disconnect ports.
So essentially, you can make the changes
that you would expect.
And then you have matching methods that tell you, for
instance, whether two ports are connected.
You can get a list of the audio modules that are
currently installed.
And you have a number of other methods as well.
Now if you're implementing a Patchfield client and you
would like to keep up to date on changes in the state of the
Patchfield service, then you can implement this
IPatchfieldClient interface.
Now if you register yourself with the Patchfield service,
you will get call backs whenever something happens.
So you might get a call back when a module is created or
when a module is to deleted, when ports are connected or
disconnected, and so on.
So this was essentially bookkeeping.
The interesting part really is the audio module API, because
that's where you implement audio functionality.
So the foundation of it is a class called audio molecule.
And it's an abstract task that has four methods that you will
need to implement.
The most important off those methods
is one called configure.
Where is it?
Here.
And the purpose of the configure method is to connect
your audio module object to the corresponding data
structures in the Patchfield service.
And let's see how this works in practice.
So let's look at the low pass module that we saw in action
in our little demo.
So here's a module called low pass module.
And it extends the audio module class that we just saw.
And sure enough, it implements the configure method that I
just mentioned.
So now this configure method internally just takes the
handle that it received before and the number of channels
that it wants and sends it to a native method called
configure native components.
So let's look at the implementation
of that native method.
That's in a file called lowpass.c.
It's right here.
And this method allocates space for a data structure
that will hold all the information that characterizes
our low pass filter, such as optic
coefficients and filter values.
And then it goes ahead and
initializes the data structure.
And now from the point of view of the API, the most important
call is this one right here.
There's as a function called am_configure.
And so that's one establishes the connection between your
audio module and the Patchfield service.
So this configure method takes the handle that
you received early.
So you receive the handle in the configure method.
And you just hand it over to this function call right here.
Then it takes a process function that you implement
and that provides the audio processing
capability that you want.
And then you give it the data structure
that we just created.
Let's take a look at our audio processing function.
It's right here.
So this one initially just retrieves the data structure
that we located earlier.
And then it extracts the filter update coefficient from
this data structure.
And there's one crucial point in here.
When it retrieves this update coefficient using a GCC
built-in called sink_fetch_and_or.
And the point here is to access this coefficient in an
atomic way.
Because we need to worry about threat safety here.
This coefficient may be changed by a different threat.
When you move the slider in the UI, then the UI thread
will change this coefficient.
But we're in the audio thread here.
We need to make sure we're properly synchronized, and
this GCC built-in takes care of that.
So now once we've retrieved our third [INAUDIBLE] update
coefficient in a threat-safe fashion, we can do the actual
signal processing.
And that's happening right here.
So we iterate over our buffers.
We read our input samples.
We perform our signal processing magic.
And we write our output buffers.
So in particular, if you've studied signal processing,
then you will recognize this line right here.
This is just the update formula although of a simple
RC low pass filter.
And that's it as far as the code is concerned.
All right, so that's it for now.
You can find the source code at GitHub.
And there's a forum at Google Groups and don't hesitate to
get in touch if you have any questions or comments.
Happy hacking.