Unknown: Hello Ubicomp I'm glad to present my work ViFin:
Harness Passive Vibration to Continuous Micro Finger Writing
with a Commodity Smartwatch. So this is a joint work with my
university, University of Virginia and University of
Washington and the company VibInt AI limited. So, finger
writing is a very popular way we use to achieve device input
nowadays. And two years ago in SenSys, the researcher they put
sensors on the fingertip to achieve the finger gesture
recognition. And last year in Ubicomp, there are two papers,
they put sensors on the fingers, or they put cameras to check the
finger movements so that they can recognize the continuous
finger movement. But these are customer device and they cannot
be achieved with commodity smartwatch. It will be more
practical if we just use this commodity smartwatches to
achieve this continuous finger writing. And yes there is a
work, like ViBand, they use this commodity smartwatch to
recognize gestures, but this system cannot recognize
continuous micro finger writing. And our contribution in this
work of ViFin is we can achieve the continuously micro finger
writing, just with a commodity smartwatch. So how does it work?
Our observations here is when we move our finger, the vibration
will propagate through the hand to the smartwatch. And as we can
see when we write 123 car here the vibration captured by the
smartwatch IMU. And we write continuous 123. Here, we write
"c a r"car here we write cat here. And interesting in this
red box, we can see "ca" in the car and "ca" inthe cat the
vibrationare very similar. So based on this observation, we
believe that just read the IMU in the commodity smartwatch, we
can recognize these micro continuous finger writing. And
what's the challenges here? Firstly, how can we recognize
continuous finger writing? and secondary people write
differently? People have different handshape. So can this
system works across different users? And third, Can this
system be robust, which means it can recognize the finger writing
in different writing speed, different positions, and
writewhile they're walking? So firstly, so if we write a letter
A and the letter B, so we can just use a slight window to
cover this vibration, like a four sub window. But the problem
is, how can we label that? It can label AABB ABBB, or, AAAB.
And our idea is we can just consider all the possibility
Wesum upall the probability, but it's too much computing. So
instead of the measure of exhaustion, we use the dynamic
programming the full bandwidth algorithm so it can calculate
more efficiently. And then we use the GRU network to train the
system. And it can, for example, it can get a result AABB and
witha simple trimming, you can cancel these repeated letters
and become "a b". What if write Hello. So, there is a problem.
We don't want that "helo". Inspired byrecurrent neural net
neuro aligner we know we need to add another class, labelit null.
So, in this way we can recognize hello and this is the neural
network architecture. The second challenge is people can write
differently like in one in this way or in this way. And people
have different handshape so here we use transfer learning to fine
tune the new users and also we have a calibration and update
scheme to make sure the system work robust and we have
spellcheck to know like which letters you are now cannot be
recognized well, so we can keep update and return the system.
And we also achieve some trainingfree gestures based on
the gyroscopeangles. So we can know, can have some gestures to
wake up the system deleted textor enter the text. And here
is the figures of the gyroscopeangles. So we can know
we can recognize twist the angle and move left, right up and
down.
So we achieve this system as a standalone application program
on commodity smartwatch and we connect this smart smartwatch to
the MadGaze smart glasses. So on the smarter screen we can see
the text output. And we just use the accelerometer and gyroscope,
the build-in IMU sensor. And we collect 15 participants data
with numbers and letters. And we have validation set here and we
test on a standard public transcription. We asked the
users to write two hours on this transcription. And the accuracy
the average accuracy is to 90%. For more accuracy, evaluation
details, and the real time evaluation we tested in the real
world for the real users, as well as the user study. Please
go to our paper details. In summary, the contribution of
this work is we achieve a micro finger writing and it works
continuously. And we have comprehensive evaluation in this
system. Lastly, let's see the demo. Thank you.