- Thank you for the introduction.
My name is Huy Viet Le
I am from the University of Stuttgart
and I will present this work on behalf of my co-authors
Sven Mayer and Niels Henze.
Smartphones offer a wide range of functions
which you can access through a touch-screen.
On a technical basis, touch-screens relate touches
into two-dimensional co-ordinates,
which are mapped to the user interface.
This concept enables an intuitive interaction but also
limits the input vocabulary
since users prefer to hold their devices
in a single-handed grip.
So when we hold a smart phone as you can see in this image,
the input consists of a single co-ordinate by the thumb,
while all other fingers on the back of the device
are not used at all.
This loose hand interaction compared to input devices which
are designed to be used with multiple fingers
such as a computer mouse or a keyboard.
So previous work extended input space
to the backside of the device
to enable all other fingers on the backside to interact.
This concept is called back of the device interaction and
basically attaches a second device or an external touch pad
to the backside of a smart Phone.
This works well for many exciting use cases, but inputs of
different fingers are still treated equally.
More importantly, the full finger surface
and the hand surface is touching
the backside of the device during interaction.
This makes it challenging to infer a meaningful
to decoordinate and to recognize
any form of intended interactions.
To avoid this problem, previous work limited size of the
touch panel to the upper half of the device
or assumed uncommon grips
so that only the finger tip is touching the backside.
But by doing so, back of device interaction is limited to
a small area of the phone, so that use cases like
grip-sensing, would not be possible at all.
We addressed this challenge and present InfiniTouch
a smartphone prototype, which enables touch input
on the whole device surface.
In addition, our prototype uses deep learning to estimate
the 3D position of the finger tips based
on the raw capacitive measurements.
With this, each finger can now be responsible
for a different function
while inputs of all other fingers
and other parts of the hand are automatically ignored.
This opens a wide range of new use cases
which I will show in the following.
So one sample use case that we implemented
are finger specific gestures.
So, in this example, sliding down with both the index
and the middle finger would select all files
in our file manager application.
In contrast, sliding down with only the index finger
would copy a selection to the clipboard
while the middle finger is responsible for pasting files
from the clipboard.
The concept of finger specific gestures
could be applied to any other application.
For example, as shortcuts to frequently used functions
or an application in which we want to avoid occlusion
of the touch-screen through fingers or UI components
such as buttons.
Another use case that we implemented
uses deflection state of the middle finger
as an action modifier.
In this example, users can draw on a painting application
when the middle finger on the backside is stretched.
But when bending the middle finger, the eraser is activated
without the need to select it on the user interface.
The concept of finger flexion
works similar to how modifier keys on keyboards work
and augment the touch input on the front screen.
This concept introduces a wide range of new input dimensions
and could be applied to other applications, such as
remote switching, 3D navigation or auto-write shortcuts.
So our smartphone prototype is based on
two LG Nexus 5 devices for back and front touch
as well as an Arduino for capacitive touch-sensing
on the edges.
Instead of simply attaching both Nexus 5 devices
back-to-back, we separated the touchscreens
from all other remaining processing units
to avoid a bulky device
which would affect how users usually hold a smartphone.
So, on the right side we can see the handheld device
which is what users will hold in their hand
and what they use to interact with.
The handheld device is based on a 3D printed frame
which holds two touchscreens
one on the back, one on the front
as well as the copper plates
for capacitive touch-sensing on the sides.
On the left, we have the hardware container which contains
all hardware to operate the handheld device,
such as the Nexus 5 circuit boards
the batteries, and the Arduino.
So both parts include self-designed printed circuit boards
which connect the handheld device with the hardware
container using a flexible flat cable.
The PCPs basically act as an extension adapter
to separate the remaining touch-sensing components
from the hardware.
We share the PCP schemes
the 3D models for the frame and container
as well as our source code in the data-set on our website.
The link is also in the paper.
Our device themselves communicate over wi-fi network
which is operated by the front device.
So the backside and the edges send our capacitive
measurements to the front side
which merges and provides them to the application layer
so that each end of application can use these measurements.
The capacitive measurements are represented
by a capacitive image
which is available every 50 milliseconds.
This is an example of such a capacitive image
so this image basically represents low resolution
finger prints and represents the electrical capacitance
caused by the touches on the device surface.
For the edges, each copper plate is represented
by one pixel in this image.
To access these values on a Nexus 5 device, we modified the
Android kernel and pulled the data
from the debugging interface.
Based on these capacitive images
we will train a deep-learning model
to estimate the 3D finger position
of finger tip positions on the device surface.
To train a model we conducted a study
to collect capacitive images
and the respective 3D finger positions.
We used a motion-capture system
to record the position of the finger tips
and our device with a sub-millimeter accuracy.
We attached reflective models from each joint of the hand
then the rigid body of four markers on the top side of
our device, to enable motion tracking.
In that way, we covered a wide range of finger positions
by instructing participants how to move their fingers
based on three variables.
The participants started by holding the device
in one of five common grips
which represented in previous work by Lea Dunn
We then instructed them to move a specific finger
in a given pattern.
The patterns included free movements of the fingers
swiping gestures
or placing their fingers in different locations
and holding them there for a short time.
Each condition lasts 30 seconds so that each participant
was recorded for 40 minutes during the study.
In total, we recruited 20 participants for our study.
This video clip shows some of the free finger movements
that participants performed during the study.
The visualization on the bottom left side
shows the resulting capacitive images
from finger movements on the device surface.
Here on the right, we can see the captured
3D finger positions as provided by our Optitrex system.
The axis that you can see in the background basically
represents the orientation and position of our device.
So after the study, we pre-processed our dataset to assign
the 3D finger positions to each capacitive image.
In the first step, we labeled all recorded markers
using the semi-automatic labeling approach
as provided by the Optitrex software.
We also removed all incomplete frames which did not include
our markers or the rigid body because of occlusion problems.
In the second step, we then changed from the
marker positions to a local co-ordinate system
so that basically each marker represents the 3D distance
to the upper left corner of our device.
In the last step, we matched the capacitive images to the
3D motion data based on the millisecond timestamps.
In total we had over 9 million samples which we stored in a
HDF5 file to use it for model training.
So we trained a Convolutional Neural Network which estimates
the position of the finger tips based on the
capacitive images.
So for model development and evaluation
we split our data circuit to three sets
which is common practice for deep learning.
Thereby, the training set consists of
14 participants and was used to train our CNN.
In the grid search, we then optimized the hyperparameters
of our CNN, such as the number of layers
or the number of neurons to achieve the lowest error
on the test set
which consists of four participants.
The validation set consists of two participants
and was never touched before in this process.
We used this set once in the end to validate the model
which performed the best on our tests done.
In terms of optimization, we developed our own loss function
to consider the error in the axis
which is perpendicular to the touchscreen.
So basically because the touchscreen cannot track movements
above or below it, we lowered the weight of the Z axis
on the loss function
since a less accurate estimation
can be easily compensated by checking
the blob availability in the capacitive image
during runtime.
Therefore, we used the traditional
Root Main Squared Error Function for the X and Y axis and
a Root Main Squared Logarithmic Error for the Z axis.
Our CNN architecture consists of four convolution layers
and two fully connected layers
with 256 and 128 neurons each.
Thereby as input we expect a 28 by 32 capacitive image
as provided by our device and the output consists
of 15 neurons which basically represents
the X, Y and Z axis of all five fingers holding the device.
We used the validation set to calculate
the mean absolute error of our model.
Thereby, the error's defined as the offset between the
estimated position of the model
and the ground truth position
as provided by our Optitrex system in centimeters.
So the average error for the X and Y axis
is 0.85 centimeters
while the Z axis error is 0.53 centimeters.
So as a comparison, the average diameter of the human index
fingertip is 1.622 centimeters
while previous work also showed that
traditional touch interaction has a systematic offset
of 0.4 centimeters.
With these errors, our model already enables
to perform precise interaction for use cases
such as finger specific gestures
or finger placements as action modifiers on the back side.
So we deployed the model on our prototype using
TensorFlow Mobile, on the front unit.
Thereby one inference takes 24.7 milliseconds on average
which enables the model to run in the background and
estimated finger positions for each frame.
We exported our model without any modifications
but techniques such as quantization or pruning
could further reduce the runtime
for a small loss of accuracy here.
So instead of using the model for each frame
we could also assign the estimated finger positions
to the closest blob in the capacitive image
and then just track the blob.
This would reduce the workload
since the model inference is only needed
when new blobs are appearing in the capacitive image.
So we used OpenCV to detect the blob controllers
and matched them with the closest estimation point.
Our simple tracking algorithm
takes 3.4 milliseconds on average
instead of the 24.7 milliseconds for model inference.
So we showed that the finger positions can be estimated
accurately for different use cases.
Future work would improve our results
by using other touch-sensing techniques
to get a higher resolution for the touch image.
So in contrast to the space saving capacitive touchscreen
infra-red sensing on frustrated total internal reflection
would provide images with a higher resolution
which could be used for model training.
Also internal activations in our use cases could be
minimized by post-processing the estimated positions.
For the finger specific gestures example, we could use a
threshold for a minimum gesture length or the confidence
score of gesture recognizes to compensate
for wrong estimations.
For the action modifiers use case
we could also display the action state of the front screen
so users are aware of what will be done next.
Lastly, we separated the touch sensing components
from the processing units
to keep the handheld device as small as possible.
So smartphone manufacturers could
for example then use preparatory sensors
such as a flexible PCB to integrate everything
into a small device and therefore
avoid the hardware container.
So to sum up this talk, we presented InfiniTouch,
a smartphone prototype which enables finger interaction
on the whole device surface.
And our contribution consists of
a fully touch sensitive smartphone
which runs Android and has the form factor
of a standard smartphone.
But in addition, we trained a deep learning model
which estimated 3D finger position
with a mean absolute error of 0.74 centimeters.
We released the PCP schemes, the source code
and our dataset on our project website
and future work could use our dataset
which by the way
contains the marker positions of all 20 joints
to reconstruct the whole hand posture.
This would enable a wide range of use cases
such as transferring the whole hand into virtual reality
or predict actions based on the hand kinematics.
So thank you for your attention
and I am open to taking questions now.
(loud applause)
- [Moderator] So we have microphones back there
and over there if you have questions.
Is that Gregory coming up?
- [Gregory] Yes it is, interesting talk.
As I was listening to it I am thinking
this is such a straight forward idea.
Why haven't companies pursued this?
So I did a little research while you were talking
and there's actually a 2016 patent application
by Microsoft that does exactly this.
I'm wondering why you think we don't see this
in commercial devices right now
given your experience?
- Yes so, one problem there could be is the usability
so basically the whole device gets touch sensitive now
and what could happen is users will hold in their hand
and accidentally perform input.
Also what is not really investigated now
is how do actually fingers move on a device
and how can we increase the usability level which makes it
suitable for the mass market.
So I think it's basically the usability aspect there.
- [Moderator] OK, Dan?
- [Dan] Hi, Dan Ashbrook, University of Copenhagen.
So given that I'm curious, do you think there is anything--
any usability that you can do without actually
identifying all the fingers
because you've got these fairly specific hand poses
you've trained on
doesn't necessarily work with multiple hands, I assume
so if we say, OK we are not gonna train on user's fingers
can we just do something by sensing
the capacitive footprint in general?
- So you mean whether we can interact with it without
finger identification
but instead of just using the gestures?
- [Dan] Right.
- Sure, we can use it for a simple shortcuts for example
so we can train it just to recognize
to identify gestures independent
from which finger performed the gesture
and can simply use it for launching application
or accessing the clipboard as I showed in this talk.
- [JaeMin] Hi, JaeMin for Seoul National University
nice talk.
I think our commitment to smarter should be integrated
which mobile phone server
but do you think about power consumption of running
this model on mobile devices?
- Okay, so our prototype basically contains
of three separate devices
so two Nexus 5 devices and an Arduino
which are all operated by different batteries there.
So I think if this prototype would be developed
by a smartphone manufacturer
everything would be added like a flexible PCB
some additional sensor which would sense detaches
on the back side for example
and thereby depends on which sensor
they use for the battery consumption.
In our case, we have the same battery runtime or little bit
lower than a normal Nexus 5 device would have.