Hello everyone, my name is Jun Gong. I am a research scientist at Apple. Today, I am
gonna to talk about Acustico: Surface Tap Detection and Localization using Wrist-based
Acoustic TDOA Sensing. This work was done during my internship at Facebook reality labs,
where I was advised by Aakar and Benko.
Before I talk about motivation and related work, first of all, I just want to quickly
show you what is Acustico and how it works.
Acustico is a passive acoustic sensing approach that enables tap detection and 2D tap localization
on uninstrumented surfaces using a wrist-worn device with two accelerometers and two microphones
underneath.
For example, in this AR calculator app, user can input the numbers on any surfaces nearby
and get the calculation results.
I will talk more details about implementations and applications later in this talk, but before
that, let's start with motivation and related work.
As computing devices become increasingly ubiquitous, there is a pressing need for input technologies
that can be always available.
While wearable devices like smartwatches, enable always-available touch input, it comes
at the cost of small physical size, which limits user's input due to the fat finger
problem.
To mitigate this problem, in UIST 16, we proposed WristWhirl that leverages the wrist as a joystick
to perform 2 dimensional continuous gestural input without using the touchscreen.
One year later, we presented Pyro by taking advantage of the area around the device to
provide an expanded volume for subtle gestural input.
This time, we looked into another potential solution: exploiting the surfaces in the environment
around us, which provide a large area for accurate and comfortable input. We developed
a sensing technique that can detect and localize taps on uninstrumented surfaces using a wrist-worn
device.
So before I dive in, first of all, let me briefly explain how Acustico works.
When a finger taps a table, vibration can propagate outwards from the point of contact
through the air, which we call sound wave. This is in contrast to the vibration propagates
through the surface itself, which we call surface wave.
We used four sensing elements on the bottom of the wrist to detect when the leading edges
of the surface wave and sound wave reach them separately.
Two microphones on the front sense the sound wave while two accelerometers on the back
sense the surface wave.
Based on the time differences of arrival between each pair of the sensors, we localize the
taps in two dimensions precisely.
We are not the first one trying to use time differences of arrival analysis to interpolate
the tap location. For example, Toffee augments the mobile devices and laptops with four piezo
sensors, similar to the accelerometers we used, and demonstrates accurate angle interpolation
for the touch events around the devices. Just like this video shows.
However, we are the first to embed acoustic sensors in a wristband form factor to detect
and localize finger taps. The wristband location and form-factor introduce two primary challenges.
First is the small distances between each pair of sensors.
Due to the small form factor of the wristband, the distances between each pair of sensors
are much smaller compared with existing work. For a regular wristband, it is usually like
4 cm * 1.5 cm. So this requires a more precise estimate of
time difference of arrival.
To mitigate this issue, we first included both accelerometers and microphones in our
wristband and calculate the time differences of arrival between these two different types
of acoustic sensors. Since the propagation speeds in surface and air are different, this
would actually add extra time differences. Let me show you a simple example to explain
why.
Let s assume there are two taps C and D that are aligned with two sensors A and B on a
wooden surface.
We assume the signals that are captured by Sensor A and Sensor B have propagation speeds
of V_A and V_B respectively.
With that, time difference of arrival between Sensor A and Sensor B when tapping at C should
be BC/V_B - AC/V_A.
Similarly, time difference of arrival between Sensor A and Sensor B when tapping at D should
be BD/V_B - AD/V_A.
Our goal is to maximize the difference between these two time differences of arrival so that
these two taps can be easily distinguished.
If sensor A and B are the same, this means that V_A equals to V_B equals to a constant V.
Adding this constraint, we can easily know that there won't be any differences between these two time differences
of arrival. However, if sensor A and sensor B are not the same. Let's say Sensor A is an accelerometer
and Sensor B is a microphone. Then we have propagation through a wooden surface is about
two times faster than the propagation though the air. Populating this relation into the
formula, we can get the difference between two TDOAs is not zero anymore, is CD/V_A.
We actually added extra time difference by using two types of acoustic sensors.
And then, we also selected highly sensitive accelerometers and microphones in our wristband
prototype and leverage a DAQ that can support as high as 1MHz sampling rate.
We use this custom hardware setup to capture the small time differences of arrival.
Here is our wristband prototype. After testing many different types of acoustic sensors that
are available on the market. We chose two accelerometers and two MEMS microphone breakouts
that are most sensitive to the tap signals.
Four sensors are situated on the bottom of the wrist with a mechanical structure.
The accelerometers were connected to an ICP signal conditioner, which is used to amplify
and filter the raw signal. Because we are targeting at a time difference of arrival
in microsecond level, we used a PicoScope DAQ for sampling the signals from both accelerometers
and microphones in 1MHz.
And further, the second challenge of a wristband form factor is the inconsistent coupling between
sensors and surface.
Since the accelerometers are embedded on the bottom of the wristband, the coupling between
the accelerometers and surface highly depends on how users put their hands on the surface.
And the accelerometer signals may also be affected by any hand movements right before
tapping. Both introduce noises and instability in the received signals at the accelerometers.
This inconsistency makes it difficult to reliably locate the taps by simply using mathematical
triangulation.
To allow for more tolerance, we chose to use a machine learning regression model to estimate
the tap's 2D coordinates.
So to be specific, first of all, the raw sensing data from two accelerometers and two microphones
are collected and sliced using a 0.1 second sliding window.
And then, within the sliding window, we use four different methods to estimate the time
differences of arrival between each pair of the sensors. For example, we calculated the
time differences between the first peaks or minimum peaks or maximum peaks. In total,
we have 24 features.
After the feature vector is packed, we feed it into two independent Random Forest regression
models, one for X position and the other for Y position.
We used the output from the models as our 2D tap coordinates.
Before I talk about the evaluation, let me first show you a short demo video.
As you can see here, the system can detect taps using a simple thresholding algorithm
based on the sensor readings. And two independent regression models for both axes are used to
interpolate the tap location in 2D.
To validate the accuracy and robustness of our sensing system, we ran a 20-participant
user study.
Since surfaces made of different materials have different properties and wave propagation
velocities, to fully evaluate the system performance, we asked participants to tap on four stools
made of wood, plastic, glass and steel. We also included a sofa arm to see how the system
performs under non-rigid surface.
For each surface, participants were asked to wear the device on their non-dominant hand
and tap at a 3 by 6 grid using their fingerpad. Each target square is 1cm by 1cm. So we have
18 taps for each session. And we repeat this for four times.
For the wood surface, we tested more conditions. We asked participants to tap using their fingernail
instead of fingerpad. We are trying to see if using fingernail to tap will improve the
system s performance because the sound is clearer.
And we also included a two-hand configuration where participants were asked to use their
dominant hand to tap, just like this figure shows. The reason why we want to include this
condition is that we try to see how system performs without the influence of hand movement
on the accelerometers.
We calculated the leave one-session-out accuracy for each participant under different conditions
by training the model using the data from three sessions and testing it using the remaining
session. Due to the time limit, I won't go into the details of the results. Here are
several primary takeaways:
First of all, Average error distance across all conditions were 7.5mm for x-axis and 4.6mm
for y-axis respectively.
Second, Steel surface had the best performance while sofa arm had the worst performance.
It might because the steel surface is homogeneous, which reduces the dispersion and reflection
of the surface wave while the sofa arm is just on the opposite side.
Using fingernail to tap did not improve the performance.
And lastly, two-hand condition is better than one-hand condition since the effect of hand
coarse movements on the accelerometers was completely removed in Two-Hand condition.
We also looked into the feature importance of TDOA features from each pair of the sensors.
As you can see in this figure, for Y axis localization, the TDOAs from different types
of sensors proved more important, which validates our idea of using the
propagation speeds difference in surface and air to add extra time difference and improve
the sensing resolution.
That's for the study results. Next, I am goanna to show you the demo applications.
We built four demo applications to showcase the possibilities of Austico.
The first application we built is a dial pad for AR devices. User can simply tap at different
locations to enter a phone number and call the person he or she wants to contact.
Similarly, we also developed a calculator for AR devices, as I showed at the beginning.
Users can input the numbers on any surfaces nearby and get the calculation results.
Our third application is a whack-a-mole AR game. Users can hit moles by taping on the
surface, which offers an immersive gaming experience with corresponding haptic tactile
feedback.
Finally, we combine the Acustico with an optical flow sensor underneath to additionally track
the user's forearm motion on the surface.
Using the optical flow sensor, we created a mouse experience where the user simply moves
their wrist to control the pointer position. Left and right mouse click are invoked by
the user tapping either on the left or right spatial location with respect to the wristband.
We believe this can be a good alternative input approach for a remote display.
With that, I would like to conclude our work with three take-home messages.
We present a passive acoustic sensing approach for wrist-worn devices to detect and localize taps
Our system can reliably detect taps and yield high localization accuracies across different surfaces
We believe Acustico holds the potential to further enrich the input expressiveness of
computing devices
That's all I want to share about for Acustico. Thanks for listening.