Hi everyone, I’m Axel and I’m going to present the work I did
during last summer internship at Google in the Chrome team in Canada.
I’ve worked with Ella, Jingjie and Navid from Google
and with Géry and Mathieu from the University of Lille and Inria in France.
To understand how asynchronicity of input and output frequencies can lead to spatial jitter,
let’s observe a timeline of events during a viewport scroll at constant speed.
Like standard mobile devices, let’s consider we have a 120Hz input touchscreen.
In our first scenario, our smartphone has a 60Hz display, which is the most common case.
As we can see, input events are well distributed with exactly 2 input events between each frame.
This means that the visual response follows the input trajectory
and is delayed in time by a ~mostly~ constant shift (here in red).
Everything is then working as expected.
Now, let’s imagine a 90Hz smartphone display, as they start to be commercially available.
As you can see, input events are no longer well distributed between the frames anymore.
This means that, when scrolling the viewport at constant speed,
the delay can vary between the last input event received and the display,
and so does the visual speed of the viewport.
Every element of the pipeline is working as expected, and yet,
the asynchronicity between input and output rates introduced spatial jitter.
We ran a small experiment whose main objective was to validate our mathematical model,
we developed in the paper.
We used Pixel 3 and Pixel 4 with a 120Hz input touchscreen as input devices
and we asked 12 participants to perform 3 tasks:
A search task, a navigation task and a read task.
We then built a simulator to replay the recorded data
and simulate frame times with different output frequencies.
We find that the empirical data (in blue) follows a very similar pattern
as predicted by the theoretical model (in red).
We also see that for the 90Hz frequency, using a 120Hz input rate is close to
the worst amount of spatial jitter that can be introduced by asynchronicity.
Whereas 60Hz is at a minimum.
The raw collected data, without simulation this time,
confirm that the Pixel 4 has significantly more spatial jitter than Pixel 3 due to the 90hz display,
which also corresponds to user feedback collected by Google.
Finally, we propose a new technique to reduce the spatial jitter using linear resampling
when the frame times can be known in advance.
Let’s take an input trajectory with input events in blue, and frame times in green.
The idea of linear resampling is simple,
it consists in computing an input position for a fixed horizon delta from the next frame time.
Here, the red doted lines are the horizons. To achieve this, we have 2 cases.
If the last input event received from the system is after the horizon.
We use linear interpolation between it and the previous ones
to compute the resampled position.
If the last input event received occurred before the horizon.
Then, we use linear extrapolation to compute the output position.
And so on…
Using the simulator, we applied our resampling technique
and observed spatial jitter across different output frequencies
and using different values of delta for the horizon.
Results suggest that our technique is able to decrease spatial jitter
regardless of the output frequency compared to the baseline in blue.
We also confirm that increasing the horizon further reduces jitter,
which makes sense because it means more interpolation than extrapolation and therefore,
fewer prediction errors.
However, this comes at the cost of increased latency,
since the horizon is further away in time from the real frame time.
The following chart shows the tradeoff between latency introduced and the spatial jitter.
Our results show that for an input frequency of 120Hz,
4 ms, here in orange, is the best value for the resampling horizon
as it doesn’t introduce latency over the baseline (here in black),
while still significantly decreasing the jitter.
To conclude, in this paper,
we describe a cause of spatial jitter
We propose a theoretical model,
and a new metric to measure it
And we propose a new linear resampling technique to reduce this phenomenon
This technique is already used in chrome for android
to address this issue of asynchronicity-induced jitter.
Thank your for your attention.