[MUSIC PLAYING]
FRANCESCO ROMANO: Hi, everyone.
I'm Francesco from the Android Developer Relations team,
and welcome you to the session "Best Practices
for Video Apps on Foldables and Large Screens."
Nowadays, people use your apps on all kinds of devices.
They expect your apps to work seamlessly on different screen
sizes and form factors.
There are currently hundreds of millions
of large-screen Android devices in use,
and you can support all of them with a single code base
by building responsive layouts that
adapt to fit phones, tablets, foldables, and laptops.
However, when dealing with video applications,
there are some additional steps to take in order
to improve the user experience, such as handling posture
changes in foldable devices and making sure
that the UI correctly scales incoming video streams
across different window sizes.
In this session, we are going to explore some of the best
development practices and we will have the chance
to learn from two special guests that
will share their journey in updating
their apps for large screens and foldables.
Let's start with posture changes.
Foldable devices offer users the possibility
to do more with their phones, including innovations
like the tabletop mode, where the phone
is placed on a surface with the hinge
in the horizontal position and the folding screen
is in a partially open state.
This mode is convenient when you want
to use the phone hands-free, and it's
great for watching media, making video calls,
taking pictures, and even gaming.
A great example that you can see is the optimization
that the Google Duo team did on their app
to work well on tablets and foldable devices.
In this session, we will see how to build a very simple layout
that contains a single ExoPlayer instance that automatically
resizes during posture changes with this nice animation.
A simple implementation will be, for example, an activity
where the root element is a MotionLayout that
contains three child views--
the video player, the control panel, and a guideline.
You may already notice some new features
that have been added in constraintlayout to help
manage foldable devices.
Let's see them in detail.
There is a new mechanism to inject runtime values using
a shared value and a new element called a ReactiveGuide,
which is an invisible UI component that positions itself
automatically when a shared value changes.
Note the attribute layout_constraintGuide_end.
This is the value you want to change to move the guideline.
Since the ReactiveGuide guide is horizontal,
this attribute represents the distance
between the guideline itself and the bottom of the part.
With that layout defined, it's now
time to your app fold-aware.
Jetpack WindowManager comes to help with this task.
After it is initialized, the library
allows you to listen for layout changes
by collecting the flow exposed by windowInfoRepository.
Every time you get a new layout information,
you can query the displayFeatures
and check the device postures.
A device can be used in a tabletop mode
if the folding feature state is half open
and the orientation is horizontal.
When this is the case, you can calculate the relative position
of the fold and move the reactive guide
to that position, while moving it to 0 otherwise.
Remember that in this case, 0 means 0 distance
from the bottom of the screen.
Now let's see some real-world use cases of successful layouts
for foldables.
I'm going to leave the word to Will Chan
from Zoom to learn how they took advantage
of the new foldable form factors to improve the UX of their app.
WILL CHAN: Hi, I'm Will.
And today, I want to give you an overview
of what Zoom has done to bring our app to foldable devices.
At Zoom, our goal is to make video communications
frictionless, and we're always looking for ways
to better connect people.
Over the last year, we've noticed more people
using larger screen devices on Android,
so we wanted to make sure these users have the best
experience possible when in a Zoom meeting.
One of the opportunities we saw for improving
the experience on larger screen devices was foldables.
Designing for foldable devices enabled
us to scale our mobile UI to address our user's needs,
no matter what size device they use.
The foldable form factor itself allows
for interesting new applications.
Many of these new foldable devices
can sit on a table in a tabletop mode that
allows for hands-free interactions, which
can be especially useful in video communication.
For example, users can still be seen in a Zoom meeting
while having the freedom to walk around a room.
For this experience, when a user folds a device,
we adjust the Zoom meeting UI so that all the controls are
on the bottom half of the screen,
while the video of other participants and self-view
are displayed on top.
This creates an intuitive video conferencing experience
and provides users with a pixel-perfect UI.
This way, we prevent the crease line on the screen
from obstructing faces of video participants.
To accomplish this, we use WindowManager
and constraintlayout.
WindowManager allows us to easily switch interfaces
when the folding state of the device changes.
And by adopting constraintlayout,
we reduce the complexity of our layout logic,
while allowing us greater flexibility
to design more responsive UIs. We also saw better rendering
performance, which helps when there are more complex layouts.
We are pleased to offer users with foldable devices
a new, convenient way to use Zoom,
and we look forward to integrating more Android
libraries to help deliver great video communications
experiences on Android.
Back to you, Francesco.
FRANCESCO ROMANO: Thanks a lot, Will.
It's always exciting to see products
that care so much about optimizing the user
experience in every scenario.
Now let's move to the second part of this session.
I already mentioned that another important point
is to adapt video streams to different aspect ratios.
So streams maybe are coming from your camera
or from someone else's, from a file, or from the web.
Maybe a user is sharing their screen with you,
but their device has a different orientation or aspect
ratio than yours.
If your app does not handle those cases,
it may result in images distorted
or with the wrong rotation.
Just like we did it before for posture changes,
let's consider the simplest scenario with a single video
stream coming from your camera and being
previewed on a TextureView.
And let's explore some best practices
to avoid ending up destroying Leonardo's masterpiece.
What you just saw can happen because by default,
the preview will be scaled to fit the whole surface.
This behavior make those scaling problems
since the x and the y-axis may be scaled by different factors.
Also by default, the TextureView does not
handle display rotations, so it's
up to the client application to correctly scale and rotate
the video stream with a multi-step transformation.
In particular, we will need to scale back the preview
to its original size, applying the right scaling,
and then handling the rotations with a few adjustments.
Let's start with rotations.
Camera sensors and camera previews
aren't always in the same orientation on Android devices.
Camera apps generally assume a fixed relationship
between the orientation of the device
and the aspect ratio in the camera preview.
But these assumptions are being challenged
by new form factors, such as foldable devices, and display
modes, such as multi-window and multi-display.
Here we can see a code snippet that
computes the rotation required in degrees to transport
the camera sensor output orientation to the device's
current orientation.
As you can see, we want the rotation
to be clockwise or counterclockwise,
depending if the camera is facing the front or the back
of the screen.
And then we basically just calculate the difference
between the two orientations.
We do some math to make sure that value is always
between 0 and 360.
Then we need to set the size of the TextureView
to be identical to the preview size chosen.
This is important to avoid the default buffer
size to be overwritten.
Note that at this step, the preview
can still be stretched out.
Finally, we calculate the scale factors
for the x and the y-axis to bring the preview back
to the original size.
And then we apply the desired scale.
In this case, I'm using the minimum value
so the picture will probably look smaller
than the viewfinder with black bars at the borders.
Keep in mind that this code shows the general procedure,
but does not cover all the edge cases.
For an extensive documentation on handling camera views,
I'd recommend checking out links at the end of this video.
So this was a simple scenario, but there
are applications out there that handle multiple video streams
like a champ.
Let's spend some time with the LumaTouch team
to listen to their experience in bringing their app to Android
and Chrome OS devices.
CHRIS DEMIRIS: Hi, I'm Chris Demiris, cofounder and head
of engineering at LumaTouch.
We're really excited to bring the powerful, multitrack video
editing experience of LumaFusion to Android and Chrome OS.
We're especially happy it'll be available in so many schools,
where Chromebooks have become an integral part of the learning
and creative experience.
We partnered with an amazing engineering team at Solbeg,
and they're here to talk about their experience,
including some of the challenges they overcame
and the tools and resources they used to make
this development a success.
ILYA CHECHENEV: We have been entrusted by LumaTouch
to build a great video editing application.
LumaFusion.
And we have faced a number of interesting development
challenges and learned a lot while creating the application.
The main thing we did to asses visibility
was to create an initial proof-of-concept video editing
app on Android, which is now available
as an open-source project that you can check out today.
The demo app reads four different video clips;
applies scaling, rotation, and transparency effects;
and ensures smooth playback and correct export.
VIKTOR LAVRENTYEVl: One of the most complicated challenges
we faced was initializing four 4K media
codecs at the same time.
We tried to use a software decoding solution,
but performance was not adequate.
So we switched to Android's MediaCodec API and ExoPlayer,
which showed a good performance.
We started experimenting with different codecs initialization
params and found a balance between performance
and stability.
The input files in the demo cover a variety of media files
and effects, from 720p videos to heavy 4K videos
and 60 frames per second.
ILYA CHECHENEV: This will be a great resource for developers
looking to get started with multitrack editing and effects
on Android and Chrome OS.
ANDREY LAPTSIONAK: We were pleasantly surprised
with the fact that adopting the application
for different device families has not taken too much time.
Despite the fact that there are a lot of devices on the market,
the app platform is relatively consistent across device
configurations, which helped speed up for our development.
ILYA CHECHENEV: We were able to overcome the performance
variations issue because of the ability
to configure the default LumaFusion setup to work well
for a given device.
For example, we can limit the ability
to work with 4K videos on devices with lower performance,
thus allowing it to run as users expect
on a wide range of devices.
VIKTOR LAVRENTYEVl: We optimized encoding
for both low-level and high-end device
to achieve the best performance.
There are two code paths.
First, if the device is able to initialize all the required
codecs with the highest performance setting,
we do that.
And if an initialization fails in the first approach,
we implemented a fallback mechanism
to calculate initialization parameters that work.
ANDREY LAPTSIONAK: We were initially
concerned about the performance of multitrack editing
on the wide range of devices.
But we discovered that even some mid-range devices
are able to process five 4K media at the same time.
ILYA CHECHENEV: Another thing that was difficult to manage
was seeking within a user timeline,
especially with multiple high-resolution tracks
and advanced effects.
We developed an adaptive algorithm
that finds an optimal balance between close and exact seek.
ANDREY LAPTSIONAK: There was work
we had to do for different screens.
By using scalable graphics for UI elements
and by having a flexible design for UI layouts,
we were able to accommodate different screen sizes,
including the special layouts for foldable versions.
ILYA CHECHENEV: LumaTouch has an outstanding UX experience,
and we are working to make sure we
deliver the best large screen possible to Chrome OS users.
We are proud of what we have done
and grateful for the opportunity to work
on such a challenging project.
FRANCESCO ROMANO: Very nice.
It's great to see so many apps coming to Android and Chrome OS
with a tailored UI.
In the description, you can find the links
to join the LumaFusion beta and to check out the source
code of the demo app.
Time is running out, and we need to recap
what you learned today.
You can find additional information on posture changes,
adaptive layouts, and much, much more on the official page,
developer.androi d.com/largescreens.
I can't wait to see what you will
build, and thanks for watching.
[MUSIC PLAYING]