CAREN CHANG: Hey everybody.
So my name is Caren, and I'm a developer
on the Android relations team.
And today we're here to talk about CameraX.
So I'm really excited that you all
have showed up because that shows how enthusiastic you
are about the CameraX library that we just
announced in Google I/O.
So if you recall, back in May, the CameraX team
announced its student library for the alpha version
of CameraX.
And the idea behind developing this library
was that everybody knows it's been
really frustrating to build camera apps, right?
Camera2 API is really hard to use.
Especially with a camera stack there's
been a lot of fragmentation on Android devices.
So figuring out how to battle the different inconsistencies
to make sure your app performs consistently
across all kinds of devices has always been a challenge.
So those are the two main problems
that the team was trying to solve when they came out
with this new library.
So we see a lot of excitement and eager from developers
trying to incorporate this new library into their apps.
So we're really excited to tell you
more about what we've been working on for the past couple
months.
So specifically the team has been
focusing on three major areas.
So the first one is adding new capabilities and new APIs
so that you can build out even more features for your camera
app.
Second one is we know everybody really
cares about testing and making sure that you have stability,
and that each time you ship your app the same consistency
is there in your app.
So we worked on enhancing the stability of the library.
And the third thing is we also heard
there was a lot excitement over extension functions.
So extension functions, if you recall,
are these special capabilities that you can now
tap into as third party camera developers.
So features such as night mode or HDR
that previously were only available
for first party native camera apps
are now available for developers.
So we worked a lot on providing more compatibility
for these extension functions.
So jumping into the new capabilities
that we've been working on, in terms of new APIs we really
wanted to listen to the developer community
to figure out, hey, what is it that you guys want to focus on
and what would help you most in developing
features for your camera apps?
So some of the main features that we are able to implement
include tap to focus, zoom control, device rotation
info so you can better handle the different configurations
on your apps, flash ability based
on lens-- so querying for whether a specific camera
lens on the phone has flash ability-- and a lot more.
So the thing about adding these new APIs when the team is
building out these new APIs, the thing they wanted to focus on
was, one, easy to use so it's not
as hard to use as some of the Camera2 APIs
that you might be more familiar with.
And two, we really wanted to focus on making sure
that, as these APIs are builded out,
they work for all the devices.
So low end devices, high end devices,
we want to make sure that when you're building out
these features they weren't the same
across all kinds of devices and they're
compatible with all kinds of devices.
The second thing that the team really focused on
is really making sure that, every time a new version
of a library is shipped, we wanted
to make sure that the same kind of stability is still there.
So you might have heard about the test lab
that we have where we have 52 different kinds of devices
representing over 200 million active devices out there
in the wild.
So this is really us trying to make sure that every time you
ship a new version app you don't have
to go through the pain of testing
the feature on all kinds of different phones.
So we want to ensure that you can have
confidence in this library.
So we added stuff for a unit test, integration test,
functional test on all kinds of devices, from low end devices
all the way to high end devices, so that you can really
be sure that when you build out these features
it's going to be OK.
And lastly, one of the things that a lot of developers
were excited for were extension functions.
So extension functions lets you take advantage of capabilities
such as night mode or HDR mode that previously
were only available if you were native camera app developers.
So these will now be open to third party developers,
as well.
So we wanted to work on expanding
the footprint for these and adding
additional manufacturers.
So I think when we announced this at Google I/O
we didn't have as much capability for this,
and now we've added a lot more compatibility.
So devices such as Samsung phones and LG phones
also now have capability for allowing
you to add extension functions.
And specifically, it's really exciting
that, starting in Android 10, all Motorola phones
will be open to enabling extension
functions for developers.
And besides working on just adding capability and adding
more testing, the CameraX team has also
been working really closely with the Lens Go team
to kind of get a sense of how the library would
perform in the wild.
So for those of you that aren't too familiar with it,
Lens Go is an app that lets users
point their camera at a specific visual
and do some visual analysis to kind of translate
things in real time.
So this is a really good opportunity
to integrate the CameraX library into the Lens Go app
to see how it would perform.
And specifically it was a really good way
to test how the CameraX library would
work because Lens Go was built for low end devices
specifically.
And millions of users are using Lens Go every month
across hundreds of devices, so getting
a sense of how the CameraX library performs in Lens Go
was a really good way to kind of get
a sense of how it would work for developers once we released
the final version of CameraX.
So CameraX was used in production,
and the CameraX team worked very closely with the Lens team
to resolve different kinds of issues.
So what this means was because we're
able to work closely with the Lens team,
we are able to figure out what kind of incompatibility
there was, and what kind of fixes
that we needed to make sure that when
you start using the library you already
have all these fixes in there.
So you don't have to worry about these inconsistencies yourself.
And some of the biggest benefits that the Lens
team saw from integrating CameraX, one,
it was a smaller APK size.
So instead of them having to maintain their own camera set
they were able to just use the CameraX library.
So this allowed them to take out a lot of code
that they previously had to maintain and update themselves,
and also device compatibility.
So due to the extensive testing that we
were able to do in our testing labs,
they were able to ship features faster because they didn't have
to spend as much time testing these new features every time
they put out a new version of their app.
And now to talk a little bit more
about how you can integrate CameraX into your library
and some of the APIs I brought up,
here's Xi from the CameraX team.
[APPLAUSE]
XI ZHANG: So thank you, Karen.
My name is Xi, and I'm an engineer
working on the CameraX team.
So today I'm going to walk you through some of our new
[INAUDIBLE].
But first, for those of you who are new to CameraX
I would like to recap the basics.
So the basic element in CameraX is called use case.
So before creating CameraX, we took a look
at how the camera API is used, then
built the use case around the most common scenarios.
So currently there are three use cases, the preview, image
analysis, and image capture.
So the previews case allows you to have
a view finder in your app that shows a live camera feed.
The image analysis use case gives you
the axis camera frames.
You can get the bytes and run your algorithm
against it, which enables features like object
detection or augmented reality.
And the capture use case simply allows you to take the picture
and saves it to disk.
So the use cases are straightforward to set up.
There are three steps in setting up a use case, configuration,
binding, and interaction.
So let's take image capture, for example.
So suppose you want the app to take pictures.
The first tab is to create a image capture
object with the config.
So in the config, you can specify parameters
like the resolution of the picture.
So there is a challenge here.
Users love Android for its device diversity.
However, it can be challenging to us
developers because the parameters focused on one
device may not work on another.
But in this case, CameraX will handle this diversity for you.
For example, if the device doesn't
support the resolution you set, it
will simply fall back to the second best thing.
So the configuration will always be successful.
The second step is binding.
There are different life cycles in the Android camera app,
the life cycle of activities, the life cycle
of camera and capture session, and many more.
So by binding the use case to a life cycle owner,
CameraX will take control of all the life cycles
so you don't have to manage the [INAUDIBLE] machines yourself.
The cameras open when it's needed and released once done.
The last but not the least, interaction.
So once you have the use case set up, you can start using it.
So in this case, if you call take pictures,
it will snap your picture.
With those three lines of code, you
will have an image capture pipeline.
So now that we've covered the basics,
let's talk about the new things.
So the API I'm going to talk about
are camera control and camera info.
Those are high level APIs that allows
you to control the camera directly independent of the use
cases.
So for example, if you have a preview use case and image
capture use case, when you update
the state have the camera, like the zoom, the flashlight,
or the other status, it will issue update
for all the use cases.
So this is where you can use camera control.
Next I'm going to show you a couple of examples
of how to build popular features with CameraX.
So the examples are tap to focus, pinch to zoom,
and zoom slider.
So the first example is tap to focus.
CameraX already support autofocus,
but sometimes you want a user to have the manual control,
like allowing the user to set the focus by tapping
on the viewfinder.
So there are two challenges if you want
to do this with the Camera2.
The first one is that you have to figure out
the transformation between the UI coordinates
and the camera sensor coordinates,
and also you have to specify the size of the focus area.
Let's see how to do this with CameraX.
So this code fits nicely in the viewfinder's
untouched callback.
So assume that you're using TextView for your viewfinder.
So you can transform the coordinates
by calling texture view metering point factory.
It takes in a texture view and the UI coordinates,
and it returns the metering point, which
is the sensor coordinates.
So this is only for TextView.
For other views we have other factories.
So once you get the meter and point,
you can create an action with this.
So this action is called FocusMeteringAction,
which means you want both focus and metering
happening around this point.
And then you give this action to camera control,
and CameraX will take it from there
and handle the rest of the work.
And that's for tab focus.
It's really simple.
So the second example is pinch to zoom.
So this is the feature where user pinches
on the viewfinder and the camera zooms in and out
with the user's fingers.
Again, to do this with Camera2 you
have to figure out the transformation and crop region.
Let's see how to do this with CameraX.
So there are two values we need, the base value
and the delta value.
So the base value is the current zoom ratio,
and the delta value is how much it's been
changed with the user's pinch.
So to get the delta value, first create a scale gesture
detector.
So this is the Android object for converting a touch
event into a scale factor.
And the scale factor here is our delta value.
Then we can use camera info to get the base value.
So camera info is an API for getting the current status
of the camera like the zoom ratio, flash availability,
or sensor rotation degrees.
So once we have the two values, we
can combine them and set it back by calling size and ratio
on camera control.
So CameraX will figure out the crop region
and send a request to the camera, and that's it.
That's all you have to do for pinch to zoom.
So the next future is also zoom, but slightly different.
Say instead of pinch to zoom you want to have a zoom slider.
This changes a lot of things.
You can no longer use the API from the previous slide
because the user experience will be inconsistent.
Let me show example.
Suppose you have a camera that can zoom from 1 to 10.
So this is the zoom ratio.
When you zoom from 1 to 2, the field view will shrink by 50%.
When you zoom from 9 to 10, although it's
the same distance on the slider, the field of view only
shrink by 10%, so that's a bad user experience.
So we have this API called setZoomPercentage
which is designed specifically for this case.
You can call this API directly with a slider value,
and it'll do the transformation internally
and the zoom will be linear.
So that for zoom slider, just one liner.
So we've talked about a couple of APIs here today.
Note that not all of them are available right now,
but they will be in a release coming next month.
So there are other things we are building,
like previously the previous use case only support text review
but now we're adding support for surface view, as well.
So I cannot now cover all of them in this talk.
If you're interested, please check out the resources online.
So on the dev side, we have a sample app
and we have code labs.
And we also have blog post videos
like this one and documents.
So there should be enough to get you started.
Today I also have good news to announce.
So in the coming December, we are
going to launch the beta version of CameraX.
So this is-- [APPLAUSE] so this is truly a big step forward.
So being in beta means two things.
First, the API will be more stable.
You will see less API change once it's in beta.
And second, the code quality will
be held to a higher standard, which
means it's be less buggy and more tested.
So overall it means it's ready for production.
We already have internal adoption,
like the Google Lens Go.
So if you're also considering migrating to CameraX,
this is a good time.
So that's about it.
I hope it's helpful.
So if you're interested, please check out our booth here
at the Android Dev Summit.
We have engineers there to answer your questions,
and we have some fun demos there.
And if you're watching this online,
you should go to our Google Group
so you can post your questions and leave
your feedback in that group.
We have people following that group, as well.
With that, I thank you all for coming.
[APPLAUSE]