- Behalf of my co-authors, Valentin Schwind,
Philipp Gottlich, and Niels Henze.
So this work is about our system called PredicTouch,
which reduces touchscreen latency
using neural networks and inertial measurement units.
So touchscreens combine input and output
in a single interface,
which makes interaction more intuitive,
and like all other input devices,
touchscreens also have a certain latency.
However, in comparison to a mouse or a keyboard,
latency is more visually apparent on a touchscreen.
So for example, in this image,
you can see that latency results in a gap
between the stylus and the writing.
So in drawing a line or dragging an object
across the touch screen, latency results
in a clearly visible gap between the stylus
and the registered position.
So this gap is visualized as dashed lines in this picture.
So latency is nearly unavoidable on today's mobile devices,
and this is because the touch input
has to be passed to several components
and layers before the output can be visualized
on a touchscreen.
For instance, touch input needs to be registered
by the touch controller.
It needs to be passed through the operating system
and needs to be processed by the application
before the response is rendered on display.
And previous work investigated touch latency
and its effect on the user.
So Ng and colleagues found that latency
on commercial devices typically range about 50
to 200 milliseconds, and they also found
that even a small amount of latency,
even below 10 milliseconds, is easy to perceive
and clearly visible.
Jota and colleagues confirmed these results
and further found that latency
decreases the user's performance.
Ritter et al. showed the same and also found
that latency above 170 milliseconds
is regarded as unacceptable, especially in tasks
such as drawing, or dragging objects around.
So how do we reduce touchscreen latency?
Previous work presented different approaches,
including software-based approaches
and hardware-based approaches to reduce latency.
So the hardware-based approaches involve additional
and novel hardware, which is faster at capturing
and faster at visualizing response.
These prototypes were mainly built
to investigate the effect of touch latency on the user.
So amongst others, Leigh and colleagues
developed a multi-touch sensor
with a sampling rate of four kilohertz,
and used a high-speed projector to visualize the response.
Similarly, Ng and colleagues use a high-speed camera
to track the fingers, and a projector to visualize response.
Cattan and colleagues used a motion capture system
and optical markers, which you can see on the right image,
to capture the finger movements,
and used a high-speed display to visualize the touch.
So the advantages of this approach
is that latency can be reduced down
to about one millisecond, however, the disadvantage here
is that additional and also bulky hardware
is required to reduce latency.
And therefore, they are only suitable
for stationary setups but not for mobile devices
where users need to carry around these bulky devices
so as to reduce latency.
In contrast, software-based approaches
do not require any additional hardware.
In general, these approaches reduce the latency
by estimating the future position of the finger
to overcome the latency gap.
So Cattan and colleagues used a linear extrapolation
based on the previous movements of the finger
to estimate its future position.
So the extrapolation reduces the gap
between the registered position and the actual position,
so this is shown as a red line in this image here.
And Henze and colleagues improved this approach
and trained neural networks
to estimate the future touch position.
So while these approaches work
without any additional hardware,
they include prediction errors.
So the prediction error is the gap
between the estimated position
and the actual position of the touch
in the time of the estimation.
So that means that wrong estimations
that are rendered on a display are perceived as jitter
and can even decrease the user's performance and accuracy.
So while software-based approaches can be used
on mobile devices, they also induce jitter,
since the bridge prediction is not always correct.
And this is where our system comes into play.
So we present a hybrid approach
that uses neural networks similar to the approach
by Henze and colleagues, and recombine it
with inertial measurement units
to reduce the prediction error.
So these inertial measurement units,
in short, IMU, consists of an accelerometer
and gyroscope, and a magnetometer,
which we used to catch the micro-movements
of the hand while it performs input on a touchscreen.
And these micro-movements give the neural networks
more information to predict the future touch position.
So we captured the micro-movements of the hand
on different positions by placing the IMU
on the wrist, on the finger, and also on the stylus.
And we choose these positions because they can be replaced
by (mumbles) devices such as a smart watch,
a smart ring, or smart pens that include the IMU
on the top of the pen.
So therefore, our approach can be implemented,
on nowadays, mobile devices without requiring the use
to carry any additional hardware
just for reducing the touch latency.
So we first built a prototype of this approach,
and then conducted our data collection study
to collect the (mumbles) data to train the neural network.
So our system consists of three IMUs
attached to the stylus, the finger, and on the wrist,
as we can see on the right of this picture here,
and these IMUs are in turn are connected to the Arduino.
The Arduino reads the sensor values
and streams them to this Nexus 7 tablet,
which connects the IMU using (mumbles) the OTG cable.
The Nexus 7 locks all the input
made on a touchscreen, the touch input,
and also the IMU sensor values
while the user is drawing on the touchscreen.
So we conducted a study to collect the (mumbles) data
for the neural network, and we asked participants
to draw, to write, and to perform a Fitts' law dragging task
similar to the approach by Henze and colleagues.
So in the drawing task, we instructed participants
to draw images of their last vacation,
while in the writing task, they had to copy a text
which we gave them.
And in the Fitts' law dragging task,
they had to drag a tile into a target shape
which we can see on the bottom right of this image here.
And participants performed these tasks
using the finger once and also the stylus once.
For the study, we recruited 18 participants,
whereas 50% of them are female.
So the collected data set
consists of about one million touch samples
from the touchscreen, and about 28 million samples
collected from the IMUs.
For each touch sample that we collected
from a touchscreen, we matched the corresponding IMU samples
by using the timestamps as the matching criteria.
So using the collected data set,
we trained neural networks
to predict the future finger position
based on the previous hand movements and touch positions.
We actually trained multiple neural networks
with different IMU combinations
to find the one model with the lowest prediction error.
As input, we used the 10 latest direction vectors
of the touch samples, which we also made position
and rotation invariant.
And for each IMU that we used in a model,
we included the 10 latest samples of the IMU.
And as output, we have the X and Y position
of the future touch positions in steps of 16 milliseconds.
So we choose the steps since this corresponds
to the sampling rate of a 60 hertz touchscreen.
As the architecture for the neural networks,
we used three hidden layers with 500, 400, and 300 neurons,
and used the adaptive gradient algorithm,
in short, AdaGrad, as the optimizer for this.
We also used the Xavier initialization scheme
to initialize the weights of the neural network,
and used a constant internalization
for the biases of the neural network.
So these box plots show the errors in pixels
of the different models that we trained.
So the gray box plots represent neural networks
that use only the touch samples
which has also recalled the approach
by Henze and colleagues.
And on other box plots, represented different combinations
of the IMUs that we trained.
So in general, what we can say here
is that we found that using any IMU combinations
reduces the prediction error significantly
in comparison to using just the touch samples,
and the best model for finger input
is using the IMU, which is placed on the wrist,
and for the stylus input, the best model
is to using the model that has the IMU on the stylus itself.
The average prediction errors in pixels
are about nine, 21, and 35, on average,
with three prediction levels, zero,
33 milliseconds, 66 milliseconds, and 99 milliseconds.
So here we'll show you some video demos
of our prototype and how it works.
So this video shows the touch latency
without any prediction, so this is what you get
when you use the Nexus 7 out-of-the-box,
and as you can see, there is a clearly visible gap
between the stylus and the writing of the word Hello here.
And this shows how it looks like
when we use the approach by Henze and colleagues
using just the touch samples.
So as you can see, the gap between the stylus
in the writing is smaller,
but there is also clearly visible jitter involved
when you write the word Hello.
And this is how it looks like when we use our approach
with a 33 milliseconds prediction,
using the IMU placed on the stylus.
So here you can see that the jitter has noticeably decreased
in comparison to the previous video.
And this is how it looks like
when we use a prediction of 66 milliseconds into future,
and what you can see here is, while the gap is smaller,
there is also much more prediction error,
it means jitter, involved.
So after we developed a model
and found the best IMU combination,
the next step is to evaluate the effects
of the latency reduction on the user.
So we conducted a user study in which participants
use our system, once with a stylus,
and once with a finger input.
We let them use three prediction levels,
means zero to 66 milliseconds,
whereas the prediction were ones with
and ones without the IMU.
And this results in 10 conditions in total
since the prediction level of zero milliseconds
cannot make use an IMU.
So each condition consists of two tasks,
a Fitts' law task and a painting task.
In a Fitts' law task, participants had to drag a tile
into a target shape, which you can see
on the picture on the right.
In the painting task, we instructed participants
to draw a picture of an island.
So we used the Fitts' law task
to determine the throughput, which is a combined measure
of accuracy and speed, and we used the painting task
to collect qualitative feedback on the latency reduction.
We randomized the order of the conditions
and recruited 16 participants for the study,
also with a 50% rate of female participants.
So we found that using the IMU
significantly increases the average throughput.
So the throughput with IMU are shown in red,
and without are shown in blue.
And we also found that the prediction levels
have a significant effect on the average throughput,
so this means the higher the prediction level,
the higher the throughput.
When participants rated the lag
and unpleasantness on a seven-point Likert scale,
we also found that significant less lag
was perceived when an IMU was involved.
Further participants received more unpleasantness
for higher production levels
because there is much more jitter involved.
In general, participants mentioned that lower latencies
are less frustrating and have a better accuracy.
So based on their own experiences using touchscreens
and also the experiences in the study,
the envisioned that latency reduction
can be a special useful for use cases such as drawing,
gaming, and also writing applications.
So our contribution is a system called PredicTouch,
a system that uses the neural networks
and inertial measurement units
to reduce the touchscreen latency.
In comparison to the previous approaches
by Henze and colleagues, we used only the touch samples,
we could show that in including IMUs
decreased the prediction error significantly.
And as a result of this latency reduction,
we showed that the user throughput
can be increased by up to 17%,
where they will perceive noticeably less lag.
So in future work, we plan to investigate
more specialized model like recurrent neural networks
or long short-term memory models
to further reduce the prediction error in general,
and also we want to investigate techniques
known for machine learning computations,
such as, examples, the bagging or boosting
to reduce the prediction error.
So thank you for your attention
and I'm open to any questions now.
(audience applauding)
- [Man] Hello, Ben Jeremy,
also from the University of Stuttgart.
Do you also use the IMUs from the smart phone or tablet
to train your data?
- No, we only use the IMUs that we placed on the hand
to catch the micro-movements of the hand,
how it moves on a touchscreen.
Did not consider the IMUs in the tablet itself.
And also the reason why we did not do that
is that the accuracy in the sampling rate
of the IMU in the tablet was not as accurate
as the one we could get from the external sensors.
- [Man] Okay, so I was just thinking
because, if you look at Daydream,
where they have special requirements for the IMUs itself,
maybe they are appropriate to get rid of external sensors
and maybe apply the same mechanism
to the internal measuring units
with your prediction capabilities.
So I think it would be interesting,
well, it's nice work.
- Yup.
- [Man] More questions?
So I wanted to ask about RNNs and LSTMs.
I was wondering why you didn't start with those networks
which are designed for time-based data,
which handle the feedback loop.
So you used a feedforward network, is that correct?
- Yup.
- [Man] So why did you choose that network
and not RNN right from the start?
- So reason is that previous work from Henze and colleagues
who used a neural network just on the touch samples,
and what we wanted to show is the effect
of adding the inertial measurement units
to improve the prediction error in general.
So the RNN (mumbles)
the IMUs will be basically the next step
to reduce the prediction error in general,
which could also be applied on the touch sample approach
by Henze and colleagues.
- [Man] Did you do that?
Are you gonna do that next, or have you actually tried it?
- So for only the touch samples,
there is actually a poster at this year's Mobile CI,
where we showed that just applying the LSTMs
already reduces prediction error,
and this can also be applied on our approach with the IMUs.
- [Man] Questions, so maybe I could ask a quick question
if that's so.
For the stylus, you used that, the smart watch,
but you could also integrate the IMU
inside the stylus, as has been shown in some projects.
Do you think that will increase the accuracy
if you have the IMU closer to the tip
rather than just on the wrist?
- Actually for the IMU use case,
we used the IMU on the stylus itself.
- [Man] Oh yeah, okay, so with the--
- So for writing, it was on the wrist,
and for using the stylus, it was on the tip of the pen.
- [Man] Okay so let's think our speaker again and--
(audience applauding)