- Hello, everyone.
My name is Alix Goguey.
I'm a post doctorate fellow working with Carl Gutwin
at the University of Saskatchewan.
Today I'm gonna present you a project
that started during my PhD and was led my Damien Masson,
a master's student at the University of Lille,
supervised by Sylvain Malacria and Gery Casiez.
One of the main topic that drove the research
on touch modality for the past decade
is the increase of the input vocabulary.
Compared to desktop computers,
touch devices embed significantly less power of expressivity
with every actions undertook by a user as to confirm
a 2D location, similar to the user's mouse.
As a result, touch versions of software
tend to offer a subset of all the possibilities
provided by the desktop counterparts.
For example, on Photoshop on desktop
offer more than 600 commands whereas
on smartphone only a few dozens are available.
To augment the input space, researchers added new dimensions
by leveraging new information such as pressure,
or orientation of the finger,
or body parts,
and also identifying which fingers cause which touch
more commonly known as finger identification.
Finger identification has seen rising interest
and given the increased input vocabulary it provides.
By executing similar physical actions,
but with different fingers,
the user can put different commands to the system.
As a simple example, we could map
different commands to different fingers.
For instance, in a vectorial drawing application,
drawing a rectangle using the index finger
and drawing ellipsis with the little finger.
Identifying finger on a touch surface
is not a trivial problem to tackle.
A lot of prototypes have been developed over the years
but the current solutions either needs all fingers
to be in contact with the screen, need clean hand shape,
are expensive, cumbersome, not realtime,
or not easy to reproduce.
No technology efficiently supports
finger identification yet.
More importantly, the reliability
of the prototypes is seldom reported.
We only found five works that
report their recondition rates.
However, their prototypes recognize
a subset of all the possible records.
Although we believe that the future generations
of multi touch devices will embed a non-invasive,
mature, and reliable finger identification technology using,
for example, capacitive sensing or 3D biometric sensors,
researchers need a robust environment to explore the use
of this information to propose useful applications today.
We therefore propose WhichFingers, a reliable,
open sourced, low cost, easily reproducible,
multi modal device enabling realtime finger identifications.
It chooses a Piezo base vibration sensor,
does not require calibration, and works not only
on touch to face but also on keyboards,
and supports cross devices uses.
The WhichFinger prototypes consist
of five mini sense hand-read vibration sensors
attached to each finger using elastic strings.
Our underlying assumption is that the finger that contacts
the interface produces a higher response on its sensors
compared to the other fingers, and thus can be identified.
The five sensors are plugged into a micro controller
through a custom design shield.
We developed a device prototype using Arduino boards
that reads at the voltage of each sensor
and sends the raw values to a host computer
at 1000 hertz using raw HID over USB.
In total, it costs less than $35.
The device is coupled with a low interaction monitor
that detects interaction events,
such as touch events or key press.
To identify which finger has been used,
we use a simple algorithm that examines
which vibration sensor created the highest voltage
right before the event occurred.
The value of the vibration sensors are sampled at 1000 hertz
which is usually much higher than the frequency
of detecting low level events, which is around 100 hertz.
The algorithm retrieves the max value within
an empirically defined time window of 30 milliseconds.
This time window can be customized
in function of the hardware used.
The device is inherently realtime
since finger identification is available
as soon as a software event is detected.
If a second contact occurs with its
time window overlapping the first one,
the two time windows are merged into a unique one
and the algorithm retrieves the two max values.
Those contacts, which are less than 30 milliseconds apart,
are considered simultaneous by the host application
and thus fingers need to be disambiguated using a heuristic
like the x y position on a touchscreen
or the key location on a keyboard.
To be noted, finger identification is available
at most at the second touch frame.
We first developed a wired version
using Arduino Leonardo board and continued the development
and issued another version using Arduino MKR1000,
which works both in a wired configuration or over WIFI.
The wired version will be available
for demonstration tonight at the demo reception.
In the following, I will present the results,
were obtained using the Arduino Leonardo board.
WhichFingers is able to identify every combination of finger
from single contact to five finger chords.
In both our experiments, we tested
single fingers configuration and two finger chord,
those configurations being the most used.
We video recorded all the trials
and manually labeled each contact points of the gesture
with the finger used by the participants.
We first conducted a control experiment
to evaluate the ruggedness of WhichFingers.
We asked 15 participants to contact
and perform slides on the touch pad
as well as typing keys on two different keyboards
with two different travel distances.
For all conditions, we imposed which combination
of dominant hand fingers to use,
either one finger or two finger chords.
For the touch pad parts, participants
had to perform two different types of operations,
a tap, which corresponds to tapping the touch pad,
and the slides, which corresponds to performing
a quick unidirectional gesture in any direction.
There was no requirement for the direction of the movements
so that we have no prior knowledge
when doing the recognition.
Participants could touch the touch pad at any location.
The five individual fingers as well as
the ten possible two finger chords were tested.
For the keyboard part of the experiment,
participants had to perform two different type of operations
on the keyboard, a type, which corresponds to typing a key
starting with the finger not in contact with the keyboard,
and a push, which corresponds to pressing a key
with the finger first positioned on it.
The participants could press on any key.
In this one, the five fingers were tested.
The overall recognition rate for the touch pad was
more than 96%, more than 99% for single fingers
and more than 94% for two finger chords.
After analyzing the video of the trials,
we found that the lower recognition rate for chords
is mainly due to two chords,
the middle finger little chord and the thumb little chord.
That corresponds to hand posture
that are difficult to articulate
and thus we hypothesize that users tend to tense their hand
in order to hold those difficult postures
which increases the sensitivity
of all the vibration sensors.
The overall recognition rate for the keyboards was more than
98%, more than 99 for the low travel distance keys,
and more than 97% for high travel distance keys.
We then conducted a second experiment
to evaluate the ruggedness of WhichFingers
in a non-constrained scenario using tasks
representative of daily activities on a desktop computer.
We asked 12 participants to complete four tasks that make
use of a track pad and our low travel distance keyboard.
We tested a pointing task where participants
had to click on singular targets,
a scrolling task where participants had to use
a two finger scroll to find a target item on a list,
a docking task where participants used two finger gesture
to translate, scale, and rotate a geometric form,
and a text entry and formatting task using the keyboard only
where participants had to reproduce formatted text.
In this one, participants were prompt
an Apple pages software window with partial text
alongside the target text to achieve in a disabled window.
The keyboard shortcut were modified
so that all were issued with the left hand,
which was the hand equipped with the device,
and the shortcut were reminded
at the bottom of the disabled window.
The overall recognition rate for the touch pad
was more than 94%, more than 98% for single fingers,
and around 92% for two finger chords.
For the keyboard, the overall recognition rates
was more than 99% and more than 99% for the single fingers
and about 98% for two finger chords.
Overall, the results of our two experiments first highlight
that WhichFingers is an accurate, low latency solution
to identify fingers in various conditions.
More generally these results indicate that fingers
enter in a sequence, so one after another,
can be detected in a robust way.
However, we identify different causes of error
that can be addressed in future work.
In some situation, the wires collided with each other
which produced signal on neighboring sensor
that is then wrongly interpreted by our monitor.
Our prototype could be improved by using less rigid wire.
We also designed our hardware and software
to make it easy to replicate and thus intentionally
use a simple signal processing algorithm that can be used
as the backbone on which heuristics tailor
for particular interaction context can be added.
For instance, in an interaction context
that involves scrolling gesture, finger could be
disambiguated using the x position of the contacts.
The left most contacts would be associated
with the left most finger, and the second contact
would be associated with the remaining finger.
For rotational gesture, the y position could be used.
The down most contact would be associated
with the left most finger if we are using the right hand
or the right most finger if we are using the left hand.
Finally, even though when informally using
our WIFI prototype we found similar performances,
rigorous testing of the prototype
still needs to be carried out.
I'll let you with some example that we implemented
so our device can be used system-wide,
running a background application
that can override the OS routines.
For example, we overrode the OS pointing routed to implement
a switch between relative and absolute pointing.
We also implemented the scrolling equivalent
in which an index finger scrolling gesture triggers
relative scroll of a document and middle ring scrolling
gesture triggers an absolute scroll of the document.
In our last example,
we overrode the Windows management system
in which an application window occupies
the left, right part, or the entire screen depending
on which chord is used while clicking on the menu bar.
To summarize, we presented WhichFingers,
a reliable, open source, low cost,
easily reproducible, multi modal device
that enables realtime finger identification.
All the material needed to replicate the device,
as well as the paper and the video,
can be found at this address.
Thank you for your attention.
(audience applauding)
- All right, thank you, we have time for one.