Hello everyone! I'm Habiba Farrukh from Purdue University and today I'll be presenting our
work titled 'S3: Side-Channel Attack on Stylus Pencils through Sensors'.
Over the last decade, stylus pens have appeared as one of the most popular accessories for handheld
mobile devices. Stylus pens let users write, draw and make selections in a variety of generic and
custom-designed applications with ease and have several useful applications like note-taking,
content selection, art and design, and text input. In this work, we focus on the recently launched
Apple Pencil's second generation which is the stylus pen supported with the Apple iPad Pro.
Unlike its first-generation, this new version of the Pencil attaches to the iPad body magnetically
for charging. While this new feature of the Pencil has improved users' experience,
the changes in the Pencil's body have opened the door to potential privacy leakage.
We observed that when a user is interacting with the screen using the Pencil, the embedded
magnets in the Pencil's body have an impact on the iPad's magnetometer. As you can see here
there is a visible change in the magnetic readings when a user writes on the screen.
Based on this observation, we present S3, a novel side-channel attack on stylus pencils through
sensor data. This attack can be launched by an adversary having remote access to a user's device
through a benign application that can collect magnetometer data in the background to infer
users' handwriting as they interact with the screen. Here, we note that iOS does not allow
third-party applications running in the background to access touch information
such as the position of the Pencil tip when a user is writing in the foreground.
However, any iOS background application like a simple benign activity tracking app or a music
app can log the magnetometer, accelerometer, and gyroscope data without needing any explicit user
permissions. Therefore, the adversary can collect the magnetometer data from
the user's device in the background to track pencil's movement from the magnetic readings
to infer users' sensitive information while they interact with the screen using Apple Pencil
In our preliminary experiments, we observed that the magnetic readings
experience subtle fluctuations when different letters and words are written on the screen.
To infer users' handwriting, an intuitive approach may be to train a learning model
to identify useful patterns such as letters, numbers, and shapes from similarities between
the magnetic data collected over time for a given set of characters. However, this is not easy since
the Pencil's magnetic impact changes when it is held at different locations or with different
orientations on the screen for a given character. Therefore, to launch a realistic attack, our
system must track the Pencil's movement when users are writing with their natural handwriting style
by holding the pencil in different orientations and on different locations of the screen.
Keeping these observations in mind, we design S3 to infer users' handwriting
from motion sensors data only. While a user is writing on the screen we collect the motion sensor
data in the background and pass it onto a server. There we perform stroke detection using the
sensor data to estimate the start and end of each stroke. To track the Pencil's movement, we build
a multi-dimensional particle filter which solely operates on the magnetic readings corresponding
to the stroke. The particle state includes the Pencil tip's location and the 3D orientation of
the pencil represented as quaternion vectors. We randomly initialize the particles representing the
Pencil location and orientation all over the iPad screen and define the movement model for Pencil
such that at each timestamp the particle state is updated based on a human writing behavior model
which predicts the location and orientation changes based on the last few time stamps.
At each timestamp, the particles are resampled based on a weight function
that uses the magnetic readings from the sensor and that from a 3D magnetic map of the pencil
which we generate offline through wardriving and magnetic field reconstruction technique.
Finally, the particle filter outputs the tracking traces of the stroke
which are then used by the attacker to guess what the user has written
To evaluate S3, we implemented our system on an Apple iPad Pro running iOS 12.0.
We asked 10 users to write letters, numbers, shapes, and English common words and sentences
on different areas on the screen in different environmental settings and iPad positions.
Here, we show some of the tracking results generated by S3.
S3's tracking results are human-readable and allow an adversary to access a user's sensitive data.
Overall, an attacker can correctly detect 96% of the letters, numbers, and shapes on average
using S3. S3 can also accurately track 93% and 80% of the common English words and sentences
written by the subjects. It is also robust against different environmental noise
and achieves an accuracy of more than 90% in all of the different environmental settings.
Please refer to our paper for more insights and details of our work and thank you for watching.