- Hello everyone.
I'm Jian from Stony Brook University.
Today I'm gonna introduce a paper, Ultra-Low-Power Mode
for Screenless Mobile Interaction.
As we all know, power consumption is a major concern
for mobile phone users and manufacturers.
Some people might easily getting panicked
why their phone battery's low.
For instance, while you're walking on the street
and trying to reach your friend.
However, your cellphone just have less than 5% battery.
You might freak out and try to reach your friend
and then notify them, "Hey, I'm out of battery."
I believe some of you guys might encounter such similar
situations, right?
So far, I have some sort of awful experience
which is really painful.
I show you some example of really a panic case
is that this very popular app called 'Die With Me'.
The whole purpose of this app is to allow panic users
to text that their phone battery is gonna die.
So you can see the message, the screenshot on the screen.
So they're really stressed and, yeah.
So with the preliminary study, a survey of 35 participants
by asking them what kind of tasks would you want
to perform when the battery level is low?
You can see that 82% of users, they would like
to still perform text entry related apps
such as Gmail, WhatsApp,
messenger apps, stuff like that.
However, for using phone camera or playing online video,
few people do that.
Like 11% and 6%.
Then we change the question by saying if you are using
a system X that drastically saves power when battery goes
low, then what's your answer?
So then we can see the result that almost all users
they prefer using text entry related apps and, surprisingly,
for using phone camera and playing online video, people,
their desire of using those apps also increased.
They both increased to 40% or 34%.
So in this paper, we're gonna mainly focus
on those three use cases.
So that to extend the hours, the time of playing this
or round in these three use cases.
So, therefore, our goal is the system X
which we call it Ultra-Low-Power-Mode, is short for ULPM
for it is designed for critically low battery status.
As we know, the screen is always one of the most
power-consuming component.
So we did a preliminary study by changing the screen status
either by just totally turning it off
or by changing the brightness.
So the Y-axis represent the hours of time playing the music.
So the higher, the better.
You can see that the first case, while we turn off the
screen totally off, you can see that it almost lasts
for nine hours.
That is because the screen is turned off.
So that the screen is not consuming power,
and all the other apps is not rounding actively anymore.
So the next two cases, by changing the brightness
either by 0% brightness or 100% brightness,
it only lasts for like three to four hours.
Why?
The reason is that the screen is still turning on,
so the screen is still consuming significant power.
Please notice here that there is zero brightness,
it still have something displayed on the screen.
It is the lowest brightness that user can reach by setting
the android or IOS phone.
So,
another very interesting phenomenon is that
by changing or by switching the screen turning
on and off for every two second frequently,
actually it doesn't save power.
It even drains power more quickly than the case of remaining
100% brightness.
Well the reason is that while you just keep changing,
turn the screen turning on and off, the UI, the app
has to destroy this UI and reconstruct the UI back and forth
which induces more calculation and power consumption.
So here's our goal.
We're trying to minimize the power consumption
by the screen.
In order to minimize the power consumption by screen,
ultra-low power mode supports screenless interaction.
What does screenless mean is that you can still interact
with your cellphone by turning off your screen.
So as we mentioned, we mainly focused on those three
use cases like using camera, we're using the text entry
cases, we're playing online video.
Why screenless?
So that all the screen just being blank,
just being turned off but you're still able
to interact with it.
So here is the short demo video about how it is work.
So the monitor in back is showing the real time
power consumption for each case.
The left side is showing the screen on case.
The right side's showing the screenless case.
You can see the big difference between those two
in terms of power consumption.
And the next is by try taking video using camera,
and this one is for listening to online video.
Okay, in order to achieve this goal to do screenless
interaction, we first need to solve the challenge
how to keep the phone interactive once screen turned off.
So, I give you a concrete example.
While you turn off the screen ...
So a user is editing a email on a Gmail app
while the user press the power button on the cellphone.
There are actually several procedures ongoing in the system.
So first, this app UI gonna move to background
of the UI stack, which is the module mainly maintain
all the UI stack, open UIs.
So, at this point, UI gonna be moved to background
and then the opening system is gonna destroy UI
so that this app UI gonna be uninteractive and is not
updating its UI anymore.
So what do we do?
In our case, we just by setting some specific app as
non-stop setters, for example in this case Gmail,
so that this app still updating its UI
so that it is also remain interactive
and keep refreshing its UI.
But please notice here, this does not waste much power
because you can see the remaining app still being destroyed.
We only setting specific app being active.
However, after solving that, the typing events still failed
to dispatch like a user typing on a screen.
It doesn't respond on the Gmail app.
Why is it?
While the typing on that, the hardware composer gonna
block out this typing event so that it won't be dispatched
successfully on this Gmail.
The hardware composer is module for rendering the final
image on the screen and maintain all the screen status.
As it detects a screen being turned off, it's gonna block
out so that nothing gonna happen on that.
So what we do is by modifying the hardware composer to make
it active, to input events regardless of the screen status
whether either it is on or off, it gonna let go and then
typing event gonna be dispatched successfully.
So, after solving the problem, there's another challenging.
How to interact screenlessly?
As we said, there are three main use cases
that we're focusing on;
Text entry, using phone camera, and playing online video.
As we can see that text entry's the most complicated case
which required very frequent visual feedback
and interactions.
So in this half, I'm gonna mostly focus this text entry case
screenlessly how we do that.
So the key idea of doing the text entry screenlessly is
by leveraging the motor memory as known as the
muscle memory, and customizing the keyboard and then enable
all your feedback.
Before going through this solution, let's first look
at what is the problem of typing screenlessly.
While you're in the screenless condition user,
you're more easily typing the wrong place
because there are no immediate visual feedback
because everything's black, and user easily
mistouching other UI parts.
For example, like a delete button which I don't want to.
And, it's also very hard to type the right word
in the right way,
and while you type the right word, it's also very hard
to self correct it.
So what we do, we design the ULPM keyboard
to tailor the UI.
You can see that we tailored the specific functionalities
so that to avoid mistouching, and we've placed more blanks
on the screen so that avoiding the mistouching
on another part.
So you can see here.
Another point is that in the meanwhile, we use gestures
to replace those functionality keys which have been removed.
Another point that we leverage motor memory.
Please notice here that all the keys, all the alphabetical
keys, they still remain in their original position
so that the user, because the user can memorize the keyboard
position already.
We also tune the auto-correction model since while screen
turn off in the screenless condition,
user more easily typing the wrong words.
So we modified the keyboard decoder and we also add audio
feedback to read out what users have typed.
We did a study that this consumed negligible power.
So we designed gestures for screenless typing for replacing
those remote functionality keys.
So like screen right, a swipe right is committing word
or just committing a space button, and swiping left is
by deleting word by word as the video is showing.
So, we implement the system by customizing android 7 system
to add ULPM feature.
We implement it both on Nexus 5 and Nexus 6P
for both the user study and power evaluation.
So here's a short video of showing how typing screenlessly.
- [Phone] Force is with you.
- So the evaluation.
With the power evaluations for text entry,
you can see the Y-axis represent the power consumption,
so the lower the better.
So we can see that instead of 100% brightness,
even for 50% brightness and 0% brightness,
screenless interaction on ULPMs sure outperform
saving drastically power up to 60%.
Similarly, for the case of using camera or playing online
video, ULBM screenless condition
also saved power, up to 40% of power.
Then we go to through user study.
The first user study trying to answer question,
can users type when the screen is turned off?
So we set up the user study by asking them to transcribe
certain sentences, but, you know, while screenless
condition, user couldn't know what are they gonna type.
So we placed a monitor to remind them we're gonna just need
to type or transcribe.
So here's a result of the typing speed.
You can see that ULPM in screenless condition,
user in average they can achieve over 30 words per minute,
which is a surprising speed regarding of the screenless
condition and the error rate is below 10%.
So we conduct a second user study to answer the question
can users center an object in the video capture once screen
is turned off?
We set up the user study by asking them to capture 20 second
of video by holding a smartphone to target at this QR code
to make it as centered as possible in that video.
Then we analyzed those 20 second video to show their pixel
deviation from the center, for on the object that you
captured and to the center of your screen.
So here's the result of showing the pixel from the center.
So the lower is the better.
So we analyzed showing that the average difference in offset
is aproximately .9% of the image size,
which means that ULPM's screenless mode has minor effect
on capturing video.
So the conclusion, we design a ultra-low power mode system
for screenless interaction and we make app interactive when
the phone has screen turned off, and we're leveraging motor
memory to support screenless text entry.
The results show ULPM drastically saved power consumption.
Thank you for your attention.
(audience clapping)
- [Facilitator] So there are two microphones in the room.
There's one over on this side and one in the middle.
If you have a question, please step up to the mic.
Go ahead.
- [Gregory] Hi, Gregory Abot.
Good motivation for your work.
- Thanks
In looking at what you did with this screenless
text entry, I wondered if you considered two options that
kind of immediately suggest themselves; using audio,
so using the microphone instead of typing,
and what about just putting a physical overlay on the phone
of the keyboard?
Did you look at those options?
- Yeah, that's very good point.
Just put some 3D printing or some very transparent,
some assisted technology to remind users.
That could be a very good inspirations.
So we considered that before, but I think in order to make
it apply to more cases that user don't need to bring
something to assist them, we first didn't consider
that much, but I think that can be regarded as a future
work of that.
Yes, that's a good point.
- [Facilitator] Great, so maybe the next question over here.
- [Leif] Hi, Leif Zaro from CODE University, Berlin.
I wonder, so some of the things you're doing are probably
similar to user interfaces for the blind.
Have you considered those as well?
- Exactly.
We had a continuous work by applying this to the blind user
or visual impaired users because they're really cautious
about their power consumption because they're always
carrying power bank.
The second is that they're really conscious about,
they might be shoulder surfing
because while they're typing they're
afraid that user, some other guy they might pass by
and see what they've typed.
So, they really want that system actually.
So we are doing that.
Yeah.
- [Leif] Okay, but if you turn it around and sort of say
what of the current user interfaces for the blind could you
maybe reappropriate for what you're doing?
Have you thought about that?
- You mean how to redesign that to make it appropriate
for disabilities?
- [Leif] No, how to redesign systems
meant for the blind to be power saving.
So--
- I got it.
To use them in the context of you don't have a screen,
which is basically what blind people have, right?
- Yeah.
Yeah, so how to coach them to better utilizing the phone
through saving power, that's a very good direction.
So far we didn't conceive that specifically
'cause first they have a lot of misunderstanding
'cause they feel like while you're turning off the screen
or just making the,
they most of the time they even just randomly
slide the brightness.
So that which doesn't save out,
or even full brightness.
100% brightness which is really drastically disaster.
So yes, I think that's good point that we should come up
with this mad design to coach them to better utilizing that.
- [Leif] Okay, thanks.
- Thanks.
- [Facilitator] We have time for one more question.
Andy?
Oh, unless it's a really quick question.
- [Audience Member] Really really quick one.
- [Facilitator] Okay.
- [Andy] Have you considered OLED displays?
- Yeah.
- [Andy] Like they have an interesting property.
- Yeah, that's good point.
So actually AMOLED is totally different technology
which is our next project trying to partially display
that keyboard because partial display gonna save power
as well, but for LCD it doesn't work.
It doesn't save power at all.
That's good point.
Thanks, Andy.
Okay.
- [Facilitator] Um, I think it is a short question?
You have a--
- [Audience Member] Yeah just, so some of the techniques
you use to turn the screen down, have you thought about
making them available as an API so other developers
could add those capabilities?
- Yeah, I think it's possible.
Since this is more low level hacking, so we think that it's
better to collaborate with some manufacturers to let them
to apply this technology to make it more available
to others.
Yes.
Thanks.
- [Facilitator] Great, let's thank our speaker once more.