[MUSIC PLAYING]
EMILIE ROBERTS: Hello, my name is Emilie Roberts,
a developer advocate working on Chrome OS and Android apps.
Today, I would like to talk about how
to make sure different input methods
work for your apps across all Android form factors, phones,
foldables, tablets, Chromebooks, Chrome
boxes with external displays, Chrome bases
with built-in monitors, and even Android TVs.
For all of these devices, input is a crucial part
of every user's app experience.
Many Android devices have non-touch input built in,
like Chromebook keyboards, and others offer them
as standard options, like this keyboard and stylus
that came free with my new foldable last month.
Before we get focused on the technical details about how
to support input devices, I would
like to zoom out for a moment and underline the motivation
behind this.
Namely, you want your users to be
able to interact with your app in the easiest,
most intuitive way possible.
If you buy a new device that comes with a keyboard,
but your favorite app does not support standard key presses,
that experience will be frustrating.
This talk is entitled, Input For All.
The "All" here is really about the users, all users.
I invite you to spend some time thinking about folks that
might be interacting with your app
and not using a phone's touchscreen.
This includes things like a keyboard and stylus,
but it also means users with accessibility needs.
How do you make your app delightful
to use for the greatest number of users?
For each kind of input device, I would
think about app functionality in two big buckets, standard
and enhanced.
Standard use cases include things like, selection,
text input, long-press, and right-click.
This is the kind of functionality users
just expect.
Luckily, the technical implementation
to handle these cases is straightforward,
and in some cases, automatic.
To provide more unique and enhanced support for your app
requires a bit more thought.
These cases are not always just nice to have,
but can really be required.
For example, a mobile game that does not
support game controllers could be perceived badly.
Likewise, a text editor that doesn't support standard copy
and paste shortcuts is not going to be
successful with keyboard users.
Where I think apps can really stand out
is when they think beyond the basics
and add unique and enabling functionality to their apps.
For example, Edjing added support for keyboard DJ-ing
and trackpad scratching.
They also provide MIDI DJ controller
support to allow users to turn their phones or Chromebooks
instantly into a DJ station.
Cubasis just released a Chrome OS optimized version
of their Android app.
It takes advantage of large screens,
as well as the ease of hooking up MIDI controllers
to Chromebooks to make this top notch, digital audio
workstation powerful and useful.
Drawing apps are likely to be interested in making Bluetooth
and USB drawing tablets work well,
as well as leveraging Chrome OS's low latency stylus
API to their drawing and painting apps.
I don't think I can say it too often.
Your users want unique, delightful, intuitive
experiences in your apps, and it's up to you to provide them.
Keyboards come built into many Chromebooks
or are a daily part of the user experience for detachables,
tablets, foldables, and TVs.
The good news is that unless you are building your own on screen
keyboard or writing your own text input views from scratch,
most basic keyboard input should work automatically.
One exception to this rule is the Enter key.
In a standard Edit Text view, the Enter key
will create a new line.
If your app has a chat functionality, a feedback form,
a newsletter sign up, or anything where text is sent,
the default new line behavior is not what you want.
Supporting the expected send functionality is easy.
Here's the code that looks for the Enter key being pressed.
Override, onKeyUp, and look for KEYCODE_ENTER.
Be sure to return true if your app has handled the key event,
and pass the event up to super if not.
If your app does media playback, include media key support
as well.
To do this, replace KEYCODE_ENTER
in the previous code with the media
key codes you want to support, as shown here
for MEDIA_NEXT and MEDIA_PREV.
Don't forget to include space for playing and pausing
with the space bar.
For games, you'd want to include the arrow keys
and WASD support as well.
This is just as easy.
Find the correct key code in the Android documentation
and watch for those keys.
Note, generally with keyboard support,
you want to look for onKeyUp.
This way, you don't have to worry
about the repeated onKeyDown events that will
be sent if a key is held down.
However, if you want to make sure you squeeze out
every millisecond of response, you can watch for onKeyDown
and handle repeated keys yourself.
Check out ChromeOS.dev for more information.
What about beyond the basics for keyboard?
The main thing to think about here
is control-based shortcuts.
Obvious ones, like copy and paste, or undo and redo,
apply to many apps.
Some apps go farther and put advanced capability
at the user's fingertips, like how
Edjing made it possible to scratch with the trackpad
just by holding down the Control key.
Think about what your app could do with excellent keyboard
support.
Here are some code showing Control-Z for undo.
It is similar to the onKeyUp and onKeyDown code from before,
but dispatchKeyShortcutEvent is used
to indicate a meta key combination.
Note, this will be triggered if Alt, Control, or Shift is
pressed.
If you only wanted to respond to a specific meta
key combination, use the flags Meta Alt On or methods
like isShiftPressed or isControlPressed.
Here, I'm doing the undo action only when Control-Z is pressed,
and the redo action only if Control-Shift-Z is used.
Alt is ignored.
Like with the keyboard, most mouse and trackpad input
should work without any extra code.
It's still worth testing out all of your apps
functionality with the mouse to see if there are any gaps.
One of the most common oversights is right-clicks.
A common paradigm with touch is to look
for long-presses on the screen to perform
context-based actions.
However, long clicking with a mouse is not intuitive.
To support right-clicks, you have a couple of options.
Here's the code for adding a contextmenu.
First, register your view or views for a contextmenu.
This will automatically handle both long-presses
and right-clicks.
Then override onCreateContextMenu and inflate
the correct menu layout.
The same menu can be used for multiple views.
And finally, set up onContextItemSelected
to indicate what action to take when a specific menu
item is selected.
For other right-click behavior beyond contextmenus,
you can set an onContextClickListener
for your views.
Just use it to call the same method
you are already using for long-presses.
When a user is using a mouse or trackpad,
they expect the UI to respond in a certain way.
Seeing visual feedback when a mouse cursor
is hovering over a view that can be clicked--
maybe the pointer icon changes or some other visual indication
happens--
feels intuitive.
Here are some code that looks for when the pointer is
hovering over a view.
setOnHoverListener to the view.
Look for the HOVER_ENTER event, and then take some action.
Here, I am changing the pointer icon to a hand
and coloring the background blue.
Don't forget to reset the icon and background color
when HOVER_EXIT occurs.
The Android APIs also provide for advanced pointer handling
and make it possible to add gestures,
pinch and zoom support, and more.
We don't have time to investigate all that today.
But take a look at developer.Android.com to get
started.
Another common mouse and trackpad enhancement
that can be critical for some games
and add functionality to some apps is pointer capture.
This allows your app to capture the mouse cursor so that it
does not appear on the screen.
Your app can then receive relative pointer events
without the cursor running into the edge of the screen.
A good example of this are first-person point of view
games, like "Minecraft--
Education Edition."
To enable pointercapture, you can call requestPointerCapture
and release it again with releasePointerCapture.
To actually use the received pointer data,
add a CapturedPointerListener to your code,
and use the relative change in pointer position
to do something cool.
And that's it.
If you have added good pointer support to your app,
styli should just work as expected for most use cases.
There are a couple of enhancements
that you should be aware of.
Some devices have support for stylus tilt and pressure.
This can add some really nice control and functionality
to your painting or drawing app.
To see this in action, check out an app,
like Concepts on a supported Chromebook or Android device.
Visit the documentation for Axis Pressure
and Axis Tilt on developer.Android.com
for implementation details.
Speaking of concepts, they are one of a number of drawing apps
that have implemented low-latency stylus
support on Chrome OS.
This library allows you to get the lowest latency display
response, and also offers configurable stroke prediction
that can give you a pen on paper like feel.
Check out the low-latency library and demo app on GitHub.
We'd love your feedback.
So, let us know what you think.
If you are optimizing your applications
for more advanced stylus support,
you will be pleased to know that we've recently
partnered with Microsoft to bring host machine
stylus support to the Android emulator.
This will let you test precise tilt and pressure
control through the Android emulator
on supported host machines.
GUY MERIN: Hello from sunny Seattle.
My name is Guy Merin, and I'm part of the Microsoft Surface
Duo Developer Experience Team.
Today, I would like to share with you some recent updates
we've contributed to the Android community, which enables
advanced input development and testing,
such as the multi-touch, insight, and stylus support.
This is an ongoing collaboration with Google,
and we have made a few public contributions
to the emulator program.
Let's see those in action.
Question we often get from developers
is, how can I test my app using a pen when
I'm running on an emulator?
What you see here is the Microsoft Duo 2 emulator
running on a Surface Studio, with two apps
running side-by-side.
A sample app on the right pane and Microsoft OneNote
on the left pane.
The sample lets you test pen pressure sensitivity,
pen orientation, the eraser tip, and other pen buttons.
On the left pane, we have the Microsoft OneNote app.
Using the emulator, I can draw on the OneNote Canvas,
and take notes or erase.
We're also excited to announce that on touch-enabled PCs,
the Android emulator now supports multiple touchpoints,
which lets you test gestures and interactions with your app that
require more than one finger, such as pinch, zoom,
and other touch interactions.
Let's see this in action on another emulator.
The built in Android studio foldable emulator,
which is running Google Maps.
Here, just with using two fingers,
I can zoom in and out of the map.
But this emulator updates go beyond just two fingers,
and they can support all the way up to 10 touchpoints,
if your hardware supports that.
All the changes you saw here are not specific to the Surface Duo
emulator and can work with other foldable emulators as well.
At Microsoft, we've been using this emulator update
in the development and optimization
of our own applications.
For example, when we test pen interactions and multiple form
factors, such as the Surface Duo, large screen,
and other foldables.
Thank you and back to you, Emilie.
EMILIE ROBERTS: If you have a game,
you need to have gamepad support.
Adding it is as easy as adding basic keyboard support.
Look for the corresponding key codes
and decide if you want to act onKeyUp or onKeyDown.
One handy thing is that the key codes for game controller
directional pads and keyboard arrow keys are the same.
Just look for the key code dpad events,
and your app will handle both.
Gamepad buttons have their own key codes as well.
So, you can look for those, like I'm
doing here for the X button.
When it comes to specialized hardware,
support will be specific to the device and your use case.
MIDI support is especially interesting for music
and creativity tools.
It allows for a wide expressive range
of input signals from pressure sensitive piano keyboards
to devices with many different input triggers, like sliders
and knobs, pads and more.
There are some handy open source samples
in the Android media samples library that can
help you get started with MIDI.
Check out the MIDI synth project,
and in particular, the synth engine module in that project.
Note, from Android 29 on, you can also
use the indicate MIDI API.
Let's recap the main ideas here.
Large screen Android devices are here and growing in popularity.
Good input support on Android has always been important,
but is even more critical for foldables, tablets,
and Chrome OS.
Lastly, spend some time thinking about your app's input
handling, and how you could increase engagement and unlock
new functionality and app experiences.
With that, I'd like to thank you for investing
in making really awesome Android apps, and of course,
adding incredible input handling.
Enjoy the Android Dev Summit, and be
sure to check out our other talks for large screens.
[MUSIC PLAYING]