[MUSIC PLAYING]
CHET HAASE: Hi, welcome to our talk on Android graphics.
I'm Chet Haase.
SUMIR KATARIA: I'm Sumir Kataria.
CHET HAASE: And we are on the Android Graphics Team.
And the Android Graphics Team basically
writes the software that gets the pixels to the screen, which
involves the drawing operations and figuring out
what the color values are.
And it also involves a lot of work
on high-performance rendering.
Speaking of which--
SUMIR KATARIA: Let's talk about low-latency graphics.
So when you're using a pen or stylus with your Android
device, you might notice that as you draw a stroke,
the rendering of pixels sometimes
lags behind where your stylus is on the screen.
The reason this happens is because Android utilizes
multi-buffered rendering.
Multi-buffered rendering starts with input,
which goes to the app.
This is written to a buffer, and this buffer is passed around
to another part of the OS.
This part of the OS then actually displays it
on the screen and renders out the pixels.
This can take a little bit of time.
There is another way to do this, and it's called
front-buffered rendering.
Front-buffered rendering has the app writing to the same buffer
that the display is reading from,
and this takes much less time.
This is the ideal way to render for stylus input.
It is possible on Android using OpenGL
on almost all versions of Android.
But historically, we haven't given platform support for it
in our APIs until Android 13, where we added it
to hardware buffers.
Today, we're also introducing a Jetpack low-latency library
to implement this in a backwards compatible way
for stylus-friendly surfaces.
Please check out the talk Adding Stylus Support for your Android
App for more information, but I'll
describe the high-level architecture here.
Think of it as two different layers,
one on top of the other.
The top layer is front buffered and is called the active stroke
layer.
The bottom layer, where the rest of your app's UI
is normally drawn, is multi-buffered,
and it's called the committed content layer.
So starting with the stylus input--
this all happens on the front-buffered layer--
we draw a stroke, still rendering it as fast as we can.
And then when we lift up, it is committed
to the content layer behind it.
So you get the best of both worlds.
You got a fast response to the stroke being drawn,
and you get multi-buffered rendering behind it.
CHET HAASE: All right, let's talk
about something completely different, Blurs.
So you could do Blurs before, but it wasn't necessarily
obvious how to do that.
We're making that a lot easier as of Android 12.
But before we talk about Blurs, let's
talk about how drawing happens on Android.
In general, most drawing operations,
even when it's just a UI element on the screen,
go through a series of primitive operations
at low-level APIs like Canvas and Paint
and Drawable, stuff like that.
So a typical operation looks something
like this, where we have a drawLine command.
And that has really three portions to it.
One is the operation that's happening.
In this case, we're drawing a line.
The other is the placement of that thing.
Where are we going to start and end?
What is the size of this object?
And then finally, what are the attributes?
How are we going to draw this line?
What is it actually going to look like?
The attributes are collected into the Paint object,
which stores the information about how each object is drawn.
There are simple attributes, like what color
am I going to draw the line, or how wide is that line going
to be, or do I want to fill this rectangle versus draw
an outline of the rectangle versus both?
There are also more complex and interesting effects
like color filter that changes the color of each pixel
as it goes through the rendering system,
or Shader, which gives you things
like linear or radial gradients or bitmaps.
And that tells the system, when I
draw this line or this rectangle,
I'm going to sample those pixel colors from some bitmap
off-screen, and then I'm going to basically draw
that bitmap out into the area.
But each of these operations is going to be very manual.
You set one of these attributes on the Paint object.
And then in a custom view, you're
going to call one of these Canvas commands
to draw a line with the Paint that holds this information.
But what if you want all of those attributes applied
to the overall view?
Well, that's why we introduce the RenderEffect
API in Android 12.
This allows you to collect all of that information
together so that you don't have to have a custom view
and go operation by operation through.
You're just telling the view, these are the attributes
that I want to apply.
So to do this, you'd call setRenderEffect on a view,
and then you don't need to do all of these things manually.
There are several effects that we build into this API
to begin with, things like drawing from a bitmap,
or color filter stuff that I mentioned before,
or blending modes.
And there's Blur.
So let's talk about Blur.
So first of all, I wrote this simple demo over on the right.
As I drag the seekBar back and forth,
you can see the Blur radius is changing for both x and y.
And it tells us the area that we're
going to use to get the information for each
of the resulting pixels.
I should point out, don't write an app like that.
You don't want a seekBar and make the user pick some blurry
background.
But this was me playing with the effect
to figure out how things worked and what the visuals are going
to be like for a UI effect that we will see a little bit later
where I pop out information on top of a blurred background.
So how does this work?
As the seekBar gets dragged around,
we get this listener called.
And there, we call updateEffect with the current value
of the progress.
And updateEffect then takes that value
and says, well, if it's 0, then we're not going
to set a render effect at all.
And instead, we're going to set render effect to null.
But if it's greater than 0, it's going
to be a value, in this case, from 1 to 50.
Then we're going to create a Blur effect
with the radius that results from that value,
and then we're going to set the render effect on the container.
It's going to blur everything inside of it.
SUMIR KATARIA: OK, let's talk about SurfaceView
and TextureView, which are two different ways
of embedding content into your app's view hierarchy.
So SurfaceView has been around since the beginning of Android.
And the core concept with the SurfaceView
is that it cuts a hole through your app's view hierarchy down
to a surface.
And that surface is assigned a hardware overlay.
You can think of this as a second window that
is more power efficient and supports high-dynamic range
rendering.
So here's visually what would be happening.
You have your app surface.
We literally cut a hole.
We put the surface view surface underneath it
so it can render through it.
TextureView, on the other hand, has been available
since API 14.
It's meant for deep integration into the view hierarchy.
So content is drawn to an offscreen buffer
and copied back into the view hierarchy.
And you should note that this is like an extra operation,
or it takes a little bit of extra time,
so it's less efficient than a SurfaceView.
So this gets to our recommendation.
We think SurfaceView is the way to go for most cases.
Media content can be shown in HDR.
DRM playback is only possible using a SurfaceView.
Games are already using SurfaceView.
And we recommend that they keep doing so because they control
their own content generation.
TextureView does have some specific use cases that it's
useful, for example, if you want to use view-level primitives
like rotation, alpha, RenderEffects, clipping, et
cetera.
Also if you need to sandwich something
between two different views, TextureView
is the only way to do it.
As I said, SurfaceView cuts a hole,
so it's not as possible to do it there.
So in short, use the SurfaceView.
CHET HAASE: All right, now, let's talk about AGSL--
Android Graphics Shading Language,
which is basically Pixel Shaders as of the Android 13 release.
So I mentioned Shaders before.
And those have been in the API for many years,
but those are fixed-function Shaders.
You can pass in parameters for those kinds of Shaders,
the linear and radial gradient stuff.
Those are essentially Shaders which
are altering the value of each pixel along the way.
But you can't really have a lot of flexibility there.
In Android 13, we give you full-on Pixel Shaders
which allows you to write your own code
and pass in functionality and logic that
is going to dynamically determine what happens
to each of those pixel values.
So first of all, let's talk about what Pixel Shaders are.
I should note that the correct term for anybody
that wants to correct me in this video is fragment shader.
That is what people actually talk about.
They're kind of the same thing, except fragment is really
about the stuff that is visual, that
is going to be altered on the screen, as opposed
to pixels, which are every element on the screen.
They're kind of synonymous.
Pixel is a more common word, so I'm using it here
so that it's not as confusing.
Anyway, a Pixel Shader is the code
that is going to run on every single pixel of an image
to determine the final color value that shows up.
All right, so here, we see some simple code
on the screen that shows a Shader that
is going to create a particular visual when
it is run on every single pixel of a rendering surface.
So you can think about this Shader basically running
on every pixel that you are seeing,
resulting in this, which is a lot more
interesting than the code that was on the screen before.
So if you go to Skia.org, you can
see other examples of that Shader,
as well, as some other ones.
So here, you can see me clicking through the Shaders
on the left.
And you can see the code that represents
that Shader on the right, and then the visuals in the middle.
Some pretty interesting stuff going on here, very graphically
interesting, dynamic, custom content, which is great.
But what I want to focus on today
is, how would you use Shaders in more of a dynamic UI situation,
as opposed to just full-on custom graphics?
That is also possible.
So let's see how that works.
First of all, what is AGSL?
As I said, it's Android Graphics Shading Language.
It is basically SKSL.
Full-on, it is the Skia shading language.
We renamed it for Android so it'd be a little less
confusing, name-wise.
And it's very similar to GLSL ES1.0.
So if you have written Shaders in OpenGL,
it's going to look really, really similar to you.
The difference between GLSL and SKSL, or AGSL,
is GLSL integrates that Shader into the GPU pipeline.
And that basically says, here is what that pixel or fragment
value is going to be.
In Skia, it is going to integrate that Shader
into the rest of the Shaders that
are operating on that fragment.
It's doing things like clip bounds checking
and anti-aliasing computations.
All of that gets compiled into one Shader
that runs all of this code.
So you can think of this as being injecting your Shader
code into the overall Skia rendering pipeline.
One of the interesting things I think about Shaders
is that, gee, it seems like a lot of operations
are happening on every pixel.
Maybe I shouldn't use Shaders because that sure
seems like a lot of work.
But the truth is, we've been doing this
since we enabled hardware acceleration way
back in Android Honeycomb.
This is how GPUs work.
They run code to calculate the color values
for every single pixel.
The only difference here is we're
allowing you to add a little bit of code
and customize what that Shader is doing for each pixel,
as opposed to just letting the system compile
its standard stuff, given the other attributes that you
passed in for the drawing operations.
So the way that you use AGSL is basically
create a RuntimeShader.
You pass in a String, which is the code
that you want it to use.
It will compile that String, and then it'll
cache that in the system.
And then you set the Shader on a Paint object
if you're going to draw manually the drawing
operations, like drawing a line with a Shader or whatever.
Or if you want to apply it to an entire view,
as I was saying before, that's what RenderEffect is for.
So you can create a RenderEffect that uses a Shader as inputs.
And then apply that to a View or a RenderNode,
and it will apply to all operations that happen
on that View or RenderNode.
Let's talk about what happens for a typical Shader running
on every pixel of an image.
So first of all, we have an input color.
This is the color that would be used
if we didn't have this extra Shader logic kicking in.
That goes in through our Shader program.
And then out the other side comes the color
that we have computed.
So if we take a look at the Shader code here,
we have the coordinates.
That is the xy location of this pixel in the overall rendering
surface.
Here, I'm creating a static red color.
And I'm going to mix that, just do
a simple linear interpolation and use
half red and half whatever the original color was.
So if we look at that, it looks something like this.
We may have had an image coming in,
and now we're going to run the Shader logic on it,
and we're going to tint every single pixel red along the way.
So if we go back to the demo that we
saw before where I was just blurring the background,
the reason that I was doing that was I
wanted to pop something to the foreground to draw the user's
attention to that picture, much like a photographer
draws the user's attention to whatever is in focus,
and then everything else just fades into the background.
So the other thing I wanted to do with this demo
was to have a label that is applying additional Shader
logic on top of the Blur that's happening in the background.
So let's take a look at how that happens.
So you can see that, where it says "Ocean view, Crete."
That is some Shader logic that's doing both a Blur
as well as a frosted effect on top of it.
So we can take a look at the way that works.
So first of all, let's look at the containment hierarchy.
So we have the overall container that
is the picture gallery that has all
of those views inside of it.
That's the thing that gets blurred by the RenderEffect
that we saw earlier.
On top of that, we have a pop up that's coming up in the front,
and that is an ImageView.
And then on top of the ImageView is a TextView.
The TextView, all that's doing is drawing the text
onto a transparent background.
The actual Shader logic is running
on the entire ImageView, although it's only
applying to the pixels inside of that label area
down at the bottom.
So first of all, this is how we actually create and pass
in values for the Shader.
First of all, we're going to set the height and width
so it knows how big the rendering surface area is.
And then that allows it to calculate
what the label area is, where it's actually going
to apply the Shader logic.
And it's going to create the RuntimeShaderEffect
and tell it the name of the variable where
it's going to pass in that input color, which is inputShader.
And then it sets the RenderEffect on the view.
All right, here's where we actually create
the RuntimeShader logic itself.
We're passing in a String that contains all the code.
You can see the variables that we have up at the top
called "uniforms" for the width and the height of the area,
as well as the input color that's coming in.
We have the coordinates that are telling us the xy
location of that input value.
We're going to calculate that value up front.
We're just going to do an eval on the coords.
And that says, here's what the input value is.
If we are not in the label area, we're just
going to return that value.
We don't want to do any shading when it's on top of the label.
So then we just return current value.
Otherwise, if we're in the label area,
then we go on to the next step.
So there's two parts to that.
One is, if we're inside of the label area
and not on the edges--
things get a little bit funny when you blur on the edges--
but if we're inside the label area,
then we're going to do a simple box blur.
This is not as good as the RenderEffect blur,
nor is it as fast.
But it shows you just a way to get a simple blur
for the pixels that I want under that frosted area.
It's simply going to combine the 25 pixels that
surrounds the pixel where we're actually
calculating a value for right now and then average those.
And then I want a frosted effect on top of that.
So now, I blurred those background pixels
under the label.
And then I want to apply a simple gradient that's
going to be opaque white at the upper left down
to mostly transparent white down at the lower right.
And that is going to basically expose
more and more of the visual, as you see here
at the top of the slide.
So to do that, we have this lightenFactor,
where we calculate what's going to happen
in this linear gradient.
And then again, we do a mix, which
is just a linear interpolation of whatever
the original pixel value was with this lightning factor
that we have applied to it.
And that is it.
SUMIR KATARIA: Thank you.
CHET HAASE: Thanks.
[MUSIC PLAYING]