Hi everyone. I’m a Hyunchul Lim
and a Ph.D. student in Information Science at Cornell.
Today, I would like to introduce our paper, HandyTrak.
This research is about recognizing the holding hand
on a commodity smartphone using body silhouette images.
The co-authors are, David Lin, Jessica Tweneboah, and Professor Cheng Zhang.
Okay, let’s start with the motivation for this study.
Understanding which hand people hold their smartphones with
is increasingly important in mobile user interface design
This is because a user’s hand mode has a direct impact on their mobile user experience.
For example, when holding smartphones,
the thumbs cannot be easily stretched to reach certain parts of a mobile screen.
As shown in the video,
users make an effort to touch uncomfortable areas
by tilting the device or frequently changing their grip
which significantly affects their touch performance and mobile user experience.
So, the commercial companies have developed techniques
like one-handed mode to make it easier for users to access the further part of the screen.
However, in this case, the reduced user interface would need to be manually readjusted
with a user’s explicit inputs according to a user’s hand modes, Left-, Right, and Both hands.
This is because the comfortable phone screen area for a user’s thumbs is different depending on their hand mode.
To address these issues, many researchers have identified more ideal solutions
that automatically detect how a smartphone is being held
and then provide adaptive user interfaces tailored to their hand mode.
For example, some researchers have developed automatic handedness detection
right after a user’s several touch interactions or only at unlocking moments.
Also, other researchers have proposed methods to continuously track how people are holding their smartphones
by using external sensors or additional devices.
So, this means that previous methods have not been able to track hand mode
at all times without additional hardware.
So, we present HandyTrak. It can continuously recognize the holding hand
whenever the user is holding the phone without any additional interaction or hardware.
Just hold the phone.
HandyTrak will provide holding hand information anytime and any moment.
So, HanyTrak is to allow designers and developers of mobile user interfaces
to acquire holding hand information for adaptive mobile user interfaces.
For example, HanyTrak can be integrated into FaceID and
then provide adaptive one-handed mode right away according to their hand mode.
Also, HandyTrak can be used on the camera application
to provide an adaptive virtual button that continuously adapts to
the fingers of the holding hand when taking selfies.
Imagine that.
It would be especially helpful when taking selfies at odd angles.
So, here, I would like to introduce the key idea of our system.
Handytrak uses the body silhouette images
captured by the front-facing camera in a smartphone.
These body silhouette images, shown in the figure as blue,
differ depending on whether the phone is held in a user's left, right, or both hands.
These differences are highly informative when training a deep learning model
to classify which hand the user is holding the phone with.
Based on this idea, we designed HandyTrak.
The raw images of the user's body captured by front-facing camera
are pre-processed to produce segmented body images.
Then, the segmented images are used as input
to a deep learning model to perform classification
to determine which hand the user is using.
We evaluated HandyTrak under three common mobile scenarios:
standing, resting against table, and sitting on a chair and
two different ways how people hold the phone: in the middle and in a skewed position.
We also evaluated HandyTrak based on five different camera configurations
since different manufacturers have different camera positions.
Under the study design,
participants were asked to perform three scenarios:
looking at the phone for face ID,
opening and browsing through an application,
and taking a selfie.
Overall, HandyTrak achieved a classification accuracy on hand mode
, left, right, or both hands with an average of about 89% under two hand positions,
three scenarios, and five camera configurations
which means that HandyTrak works in a variety of situations
and could be applied to various types of smartphone models.
To be more specific, HandyTrak has better performance
when the phone is in a skewed position
rather than in the middle position since
the angle of the camera in a skewed pose makes bigger differences on body images.
Now, we will discuss the opportunities and challenges
of deploying HandyTrak in real-world scenarios.
We found that the accuracy improves as the window size gets bigger.
The average accuracy with the window size 10 was 87.2%.
This increased to 92.3% when the window size was 120,
which means that HandyTrak can achieve high performance
if applications are less sensitive to detecting the holding hand.
The second one is battery and privacy issues.
Using a front-facing camera continuously will take a lot of resources
and cause privacy issues.
However, we believe that these problems can be addressed
by applying our system at selective times.
For example, our system could be used to activate only for certain time periods
depending on the purpose of its use in applications.
HandyTrak showed that it still works well
during unlocking moments, app browsing, and taking a selfie, respectively.
We admitted that requiring the user to provide training data
is one limitation of HandyTrak.
Our system needs at least three minutes training data
to acquire the current accuracy.
HandyTrak could not perform well for a user-independent study.
Since the different participants have different hand poses and body shapes,
it is hard to expect a model with such a small training set
to generalize well on a new user’s data.
So, we believe that if provided with a huge amount of data
on different hand positions and body shapes from other users,
it is possible to develop a user-independent model.
For example, we can generate huge synthetic training sets
with various hairstyles, body shapes and backgrounds
to improve user-independent model performance.
Due to the time limit,
I only briefly mentioned each part.
You can find detailed information about the study in the paper.
Thank you.