- Okay, so, good morning everyone.
My name is Gery Casiez, my talk is about
the merger on characterization of end-to-end latency.
So the end-to-end latency or lag
of a graphical user interface
is defined as the duration between user action,
typically is a displacement of a finger,
and the corresponding on-screen visual feedback.
Here is the displacement of the square.
So it has been shown that
latency can be perceived on touch surfaces
even when it is a few milliseconds,
and that preference degrades from 25 milliseconds,
so it's important to measure it,
and also to determine where it comes from.
So the most common procedure to measure this
end-to-end latency is to use an external camera
and to record both the input device
and the associated response on-screen.
And then to analyze the video to match two related events
in order to count the number of elapsed frames between them,
so this is a tedious and error-prone process.
To make this task easier, there are other methods
that have been proposed, some requiring no extra hardware,
and asking users to perform particular task.
users using specific hardware to do repeating missions.
So these techniques have three main limitations.
First it is not possible to measure latency
while the user is interacting with an existing application.
Second, you can measure end-to-end latency
but it is hard to know where it comes from.
And third, the proposed methods
are limited to some technologies.
They mostly work with Dutch.
Some contrast to these methods,
we proposed a low-cost technique to measure our latency
while the user is interacting with all your applications.
Second, our technique has the ability to measure
hit on a wide range of interactive systems
that include any touch sensor with any technology,
and any interactive system with a physical button.
And last but not least, we can insert probes at hardware on
software levels to determine where the latency comes from.
So let's see how it works.
First, we need to determine when the interaction occurs.
So as we want to let users
interact with existing applications,
we want to leave their finger pulp-free.
At the curl of this detection,
there is a vibration sensor made of pezory twitch polymer
loaded with mass.
It is good to, it has to crane
that one can comfortably wear when interacting,
as it leaves the pulp of the finger free.
So when a shock occurs, the mass bends,
the sensors film due to exoration,
and this results in a strain
that creates a change of voltage
across the electrodes of the sensor.
So the process or so two pentimeters
to adjust the sensitivity
of the input and output detections.
So the vibration sensor or signal
first goes through an RC high pass filter
before going to an interrupt pin
of an original micro-controller
configured to detect rising hedges.
An optional operational amplifier
allow adjusting the sensitivity,
and this threshold
is adjusting using our first potency meter.
So first we measured the response time of our sensor
as it can affect the measure of latency.
We did that by tapping a copper plate with a finger wrapped
with aluminum foil to know exactly when the contact occurs.
So response time depends on this threshold too then,
so if this threshold voltage is too low,
false positive can occur, but from a voltage of 1.5 volts
we had no false positive for two and 20 measures we did,
and by bypassing the amplification, the response time is
no more than six milliseconds with low standard deviation.
So we noticed some signal bounces, 'cause the maximum time
between the first and last interaction are six milliseconds,
so we debunked the vibration sensor signal
by ignoring the interrupts for two and 20 milliseconds
after the first interaction,
which is very conservative.
So the screen response is detected through a change
in its brightness, ideally from black to white.
We use a photodiode that can be tapped on a display,
and has a fast and consistent response time
with good visible light sensitivity.
We again use a handheld amp has a voltage operator
to trigger an interrupt and adjust the sensitivity
through a second potency meter.
The original response time of the photodiode
and obtain an average of four microseconds
with very low standard deviation and without signal bounces,
so no debounce is necessary for the photodiode.
So connecting the pizzle and photodiode
to the same original probe provides an efficient way
to measure the end-to-end latency of a system, but
we wanted to complement this approach with software probes.
So that we know,
we use supports Raw HID communication at 1000 hertz.
So to evaluate the temporal cost of communication,
we performed repeated measures of front-trip exchanges
between the host computer and charge we know
using HID API.
So the front-trip was an average
1.2 millisecond with a maximum of two milliseconds,
so the normal time for sending a report
from the hardware to the host computer
can thus be reasonably estimated to one millisecond.
So when a user action is detected
through the vibration sensor that we know,
not the time of this peso event,
and forwards it to the host computer in a Raw HID report.
This report is received through HID API by the host computer
which logs the event at the time of its arrival
according to its on clock.
At some point the input event is delivered
to the application by toolkit, and upon reception
of the input event the application updates its
internal state and repaints one or more of its windows.
This is the time where we can repaint a specific window
of ours over which the photodiode is placed.
So at each of these steps, we can insert a software probe.
This creates an update, the photodiode triggers,
the original sensor will sense a Raw HID event,
that is received by the host.
So the end-to-end latency of the system can be computed
in two ways, by subsorting the pizzle time
from the photodiode time using the original clock
they used or those of the host computer.
So when using the host computer prog values latency
can be further characterized by looking at the recorded
times of the values software probes.
So if you have the
source code of an application,
you can turn one part of the screen from black to white,
and you can also insert software probes at different places.
If you don't have the source code,
you can only measure the end-to-end latency
by finding a widget
that changes its brightness upon detecting a touch event.
Here are the measures we did.
First we did measures on the same hardware as the one
I presented two years ago to measure latency using a mouse,
and we obtain measures that are very close.
So the main difference between the results,
so the difference is below two milliseconds.
Also the end-to-end latency computed
with the original clock values is very close,
below 0.1 millisecond to the one computing
with those of the host computer, which also illustrates
the efficiency and robustness
of our communication protocol.
Then we investigated how input and output
frequency effects latency,
we measure instance of HID pulling rates
using a C++ qt application and a Logitech G9 mouse,
that can be configured to run
between 125 hertz and 1000 hertz.
So the time to receive an HID event
is only three milliseconds compared to 11 milliseconds
for more classical mouse.
And the gap here between repaint screen and end-to-end,
further an important code contribution of the drawing
and refresh time to the end-to-end latency.
So here increasing the input frequency up to 1000 hertz,
only results in marginal improvement
of less than four milliseconds.
We then measure the inference of the display frequency
using a Dell monitor capable of refreshing up to 144 hertz,
and we performed measures with gret and qt
at 60 and 120 hertz.
So here doublings, this pair frequency are thus reduced,
around 16 milliseconds
with gret and qt.
And here, 21 milliseconds of latency
was the lowest measure of latency we obtained
through all our measures.
We then compared different toolkits and web applications
on different touch devices for each system,
we developed a native test application using Java Android
or Cocoa and alternative ones
using qt, Java effects and Unity's holding.
We also developed three HTML5 JavaScript
using standup canvas, CSS and web chair.
Overall, the observed latency ranges from 48 to 276
milliseconds depending on the device system and toolkit.
For the same toolkit,
results can be quite different between devices and system.
For instance our qt application
has high latency on the Nexus 10
with 160 milliseconds.
But the same application on the Moto X2
exhibits only 70 milliseconds.
The same is true for HTML5
canvas where differences can be huge.
The general rule we observe is that
using adaptive language toolkit is usually the best choice
for minimizing the end-to-end latency.
While using HTML5 web chair
seems a good alternative
for cross-prom web development.
We finally measured the latency on the MacBook Pro
when using its internal keyboard and track pad
as well as external good response.
With both the internal and external track pads
we were surprised to measure high end-to-end latency
above 230 milliseconds.
After observing this however, we hypothesized the system
waits for more than 150 milliseconds upon touch detection,
possibly to determine if a gesture or other contacts
would follow before generating the event.
And to wait we decide put to this we repeated the measures
by placing a first finger on the track pad
on using a second one, two times the surface,
and here is the time until repaint
dropped drastically around 30 milliseconds.
So we performed similar measures with the keyboard
and did not notice significant differences
between the wired and wireless version.
So as conclusion,
first I presented the first method
to measure and characterize latency
while interacting with an application.
This works both on touch surfaces
and button-equipped devices, for touch,
input latency is in the range 15 to 30 milliseconds.
We observed that most of the latency is
due to the output latency
which might come from the gooey toolkits,
the operating system the graphic card
and the physical screen.
Overall, output latency is in the range of 90 to 250
milliseconds across all the measures we performed.
The best ways to reduce latency
is to prefer using native toolkits,
and incasing output frequency when it is possible.
So all the materials to replicate the electronic
broad the source code available at this address,
and also you are welcome to come tonight to try the demo.
Thank you.
(applause)
- Alright, thank you Gery,
just gonna ask a question really awkwardly here,
unless someone else has a question, speak up now.
Alright, I got one.
So, what I liked about this really.