Hi everyone, my name is Tengxiang Zhang from Institute of computing technology Chinese
academy of sciences
I'm gonna present
ThermalRing: gesture and tap inputs enabled by a thermal imaging smart ring
This is a collaboration work between the Chinese academy of sciences, lenovo, Huawei
and Tsinghua University
The trend of IoT brings a plethora of smart devices, each with its own interaction interfaces. people can use the designated controller
use a smart phone as universal controller
voice command or using in-air gestures
however
controllers are not spontaneous
voice commands or in-air gestures can be intrusive and tiring
so
It still needs an input solution that is versatile
spontaneous, subtle, and private
We propose ThermalRing, which
recognizes
drawing gestures on flat surfaces
can also recognize thin and passive thermaltags for quick access purpose
ThermalRing is Identity-anonymous, illumination-invariant,
power efficient finger-worn vision based input technique
it is versatile for it's vision-based, and spontaneous and subtle for it is worn on a finger
private since thermal
images does not reveal identity
In this paper we use ML90640 as thermal camera
which has a diameter of eight mm
and a height of six mm
it costs around forty us dollars and it only consumes 20 mA at 3 volts
even though the prototype we used in paper communicates with PC via cable connected serial port
we have a new version which uses bluetooth to transmit the thermal image
which is open source on
Github
all thermal images go though the same group processing flow
the raw temperature data is the first scaled
then filtered to smooth the image then we apply otsu threshold
only keep the largest contour
which is assumed to be the interacting hand
let's look at some background and realated work
thermal camera works in the long wavelengths infrared range
oh
images
energy from heat body
RGB and NIR cameras, however, images reflected energy from the object
This difference makes thermal imaging more robust to illumination changes
more privacy preserving and power efficient
thermal imaging is used by some work for interaction purpose
cameras
to detect
join gestures
babe
I use them
ah
camera on top of the table
through detect the traces left on the table top
some more reflection
surrounding reflective services
directly images the hand itself
located on the finger, leverage the dexterity of
fingers
In this paper
we will talk about three example domains and the first example domain
drawing gesture sensing
users can use this asymmetry bimanual interaction
for its naturalness, easiness and for the natural
interaction area framed by the auxiliary hand
six steps sensing flow
first locate fingertip and the detect finger lift
for calibrations then we estimate x y coordinates and apply a Kalman filter
I
extract features using bag of words model of
the fingertip movements angle and amplitude
which is fed to SVM model for training and prediction the detailed
algorithm can be found in the paper
we also conducted user study
to evaluate our technique we used nine graffiti gestures, 8 represents
smart devices and 1 the cancel operation
to pair with smart devices. we recruited
six participants
conducted three sessions
the trials of each gesture per session
the ring was taken down during the rest between sessions
to mimic a more realistic usage scenario
total of three thousand two hundred and forty trials and
gestures and three hundred and sixty four each gesture
the average within-user accuracy is nine eight point two percent between-user accuracy is 85.7%
we can see there is some higher confusion between the gestures
similar to each other
I
subjective results show that users are
feels the technique comfortable and convenient
do not mind to rotate the ring
I
in purpose but they do have a issue with input speed
that's because we only uses the eight frames per second
we believe a camera with a higher frame rate can
enable a fast
drawing gesture recognition
the second example domain is a click and slide gesture sensing
which can be used to control
detailed sensing algorithm can be found in
our paper
We recruited eight participants
there are three sessions
sixteen clicks and eighth slides operations
toh
black
768 click gestures and 192 slide gestures
the overall click accuracy is 94.9%
almost all slides
operations have been successfully completed
users feel that they can locate 4 bottons and 2 sliders by referring to the auxiliary hand
users do not have issue with locating the UI elements
they feel the technique to be precise
not tiring
the next example
we show how thermal ring can be used to recognize thermal tags
and passive tags made of high heat reflective materials
in a DIY manner. users can use a knife and cut through a copper tape
a
ThermalTag works by reflecting the heat from the hands
during the imaging process
We design the interaction process to be touch , lift and hold
to ensure
the accuracy of the detection
users first touch the camera on the tag
the camera
lift at around
three hundred meters and hold
there
for a period
the pilot study to determine the optimal thermaltag
size
to be the 20mm square
We conducted a user study to scan
six different thermal tags which is play, stop
help, up, down, and search
8 participants
there are two sessions with the sixth box procession
twenty trials but blocks so there's
one thousand nine hundred and twenty scans and three hundred and twenty for each tag and the average
within-user accuracies 95% between-user accuracies
ninety point one percent
I
every
I'm
and on completing the scan is only three point five seconds
users
do not think
the technique is physically or mentally demanding and they are happy with the scan speed
there are
more application scenarios for example users can control a smart curtain from table top
a smart speaker for my chair
Users can also applied a thermaltag on a white board to navigate the slides
or attach one on the door so that when he pushes the door open the light automatically
I
here we discussing limitations of future work. ThermalRing is not evaluate
with blocked images in the paper also we only conduct validation
spacing direction
and the more users scenarios are required for final product
how to input requires a mode switch mechanism
so thermalring can be used to complete the task different in their nature
a single a double taps on the table can be a good mode switch gesture
also important to add a visual or haptic feedback
certain interactions appearance
used to interact
inside of pocket, thanks to its illumination-invariant
nature
enables discreet and one-handed interaction
future work
I
helpful to use 3D printed cases to build standard UI elements
so that is easier for users to scan
Thermalrings can also be used to recognize in-air gestures
object against heat body for example a tape on the palm
detect the current context
the last image shows a hand that is holding a pen and writing