Hello Everyone.
I am Yueting Weng from Tsinghua University.
Today, I am happy to share you with this paper: FaceSight: Enabling Hand-to-Face Gesture Interaction
on AR Glasses With a Downward-Facing Camera Vision.
This is done in collaboration with Chun, Yingtian, Yuhang, Yukang, and professor YuanChun.
FaceSight fixes a wide-angle infrared camera onto the nose bridge of an AR glasses, which
looks downward to provide unique sensing capabilities of users' lower face and all hand behaviors
on and around the face.
In this work, we focus on sensing hand-to-face gesture to input AR glasses.
Researchers have shown that the hand-to-face interaction, not only inherits the benefits
of haptic, eyes-free and always-available on-body input, but also embodies semantic
gestures related to different facial parts that are intuitive and easy to learn.
Also, AR glasses are head-worn device that involve contact with face, which is able to
sense gestures performed on the face with embedded sensor.
These benefits together bring great potentials for performing hand-to-face interaction on
the AR glasses.
Our method is to deploy an IR camera on the AR glasses.
This configuration brings three benefits.
First, we can set the luminous intensity and range of the infrared lighting source, to
illuminate only the face and hand while keeping the background almost dark.
This not only simplifies the algorithm but also reduces the privacy concerns of capturing
the surrounding environment.
Second, our target can be captured in close and high-resolution images so that it is possible
to detect rich and subtle hand-to-face gestures.
Third, the form factor is compact, which is crucial for the design of wearable devices.
We first explore the design space of hand-to-face Interaction with the help of FaceSight: The
dimension contains: 1)Touch Location 2) Tapping and Swiping.
3)Symbolic Hand Gesture 4)Nose Deformation as Input, The nose deformation can be used
as continuous input signal to enable novel nose-based input techniques.
For example, one can control a scroll-bar by pushing the nose or using different pushing
pressures to trigger discrete functions.
5)Hand Identity, use the right hand or left hand. and 6)Number of Touching Fingers.
In total, we propose and evaluate twenty-one hand-to-face gestures in this work.
We implement a real-time pipeline.
Given a gray-scale input image, we first segment the hand, facial skin according to the brightness
variation features.
Then, we run a geometric rule-based contact detection method to detect face touching and
recognize touch location.
After that, we train different CNN models associated with different touch locations
to classify the hand-to-face gestures.
Finally, we further run corresponding methods to determine the required interaction parameters,
such as estimating the nose deformation by tracking the geometric features of nose contour
in continuous frames or locate the fingertip for swiping inputs.
We totally collect fourteen thousand four hundred touch samples, around one hundred
and ninety thousand images from ten users in our evaluation.
Our method achieves around eighty-three percent in recognizing all gestures simultaneously
in this dataset.
Based on the results, we develop example AR applications to demonstrate the usages.
Including “Video Player”.
Users can gently touch nose wing for fast forward or backward, tap nose tip to pause
or resume, or rudely push nose wing for playing next or previous video.
Users can also swipe on the cheek to scroll a page, or perform cover mouth gesture to
automatically open voice assistant, which will close while user releasing the hand.
Note that the huge object in the left half part is the camera, the obvious blocking issue
happens only in this screen recording mode, but not in the real case.
We let ten participants use our system and collect their subjective feedback.
The results showed that our interaction techniques are highly accepted by users, and the form
factor rarely narrows real-world view because of camera physical protrusion.
FaceSight also has the potential to apply on the another platform, such as VR or smart
eyeglasses, to control IOT devices.
To avoid Midas touch problem, we can design a mode-switch gesture to let user specify
the interaction mode, or repeat an action twice to confirm the intention.
For the limitation part, we need to test our algorithm at more complex environment and
research more lightweight algorithm so that we can deploy our method on the AR glasses.