Hi everyone, I am Yuntao Wang from Tsinghua University.
I introduce you FaceOri, a technique that can track head position and orientation using
ultrasonic ranging on earphones.
This work is a collaboration between Tsinghua University and the University of Washington.
Earphones are one of the most ubiquitous accessories we use every day.
We wear them around the house, at work, or on the go.
With the earphones’ cord getting cut, the input microphone has migrated from placement
on the cable to a position at each of the user’s ears.
While most earphones leverage the microphone to take calls and, more recently, to enable
the active noise cancellation functionality, we find that this unique placement of these
sensors can be used to unlock a broader range of context-aware interactions when the earphone
is transformed into a spatial input device.
This motivated us to develop FaceOri.
FaceOri uses the speaker on any commodity device to generate an inaudible, ultrasonic
chirp, which is picked up by the set of microphones on the user's earphone and then processed
P10: to calculate the user’s face orientation and distance with respect to the device.
We adopted FMCW ultrasonic ranging as FaceOri’s core method.
Traditional FMCW radar uses linear frequency changing signals for distance measurement.
This is accomplished by comparing the frequency of the received signal to the transmitted
signal.
The change of frequency is the time delay, also the distance the signal travels.
However, there is an assumption that the transmitter and the receiver share the same clock for
time synchronization.
While, in our case, the speaker and the microphone are located at different devices.
Their clock chips are not time-synchronized.
Therefore, we used a calibration procedure to solve this issue.
Referring to MilliSonic, we calibrated the clock by placing one microphone against the
speaker for a couple of seconds.
Therefore, FaceOri can establish a reference position and handle the continuous clock time
drift between the transmitter and the receiver by applying a linear curve fitting solution.
Since the microphones are located on the side of the head, minimal head shifts can cause
the microphones to be occluded from the speaker.
This loss of line-of-sight (LOS) significantly degrades the signal-to-noise ratio (SNR).
These issues can significantly harm the tracking performance.
Therefore, FaceOri adopts an inaudible triangular modulated chirp signal to reduce the Doppler
effect.
Further, FaceOri uses a non-coherent integration method with CFAR adaptive thresholding algorithm
for a better SNR.
Once we obtained the three distances from the device to the three microphones.
We can calculate the yaw, and pitch angle of the face to the device with the formulas
on this slide.
FaceOri also supports the binary attention classification method that detects whether
the user is orienting at the device.
This can still be useful in many scenarios.
Different from the continuous tracking functionality, this binary attention classification is calibration-free.
To validate our method, we used a galaxy S21 smartphone as the transmitter.
The receiver includes a Mpow H19 headphone, a Zoom H6 audio interface, an MPU-9250 IMU
module, and a Thinkpad X1 Carbon laptop as the signal processor.
The MPU 9250 module is used to compare the performance of FaceOri to an IMU-based solution
and explore a sensor-fusion method.
FaceOri software was implemented in Python with functions including streaming and logging
raw audio signals, performing calibration, estimating orientation, and displaying the
results.
Lastly, we conducted a user study with 12 participants to evaluate FaceOri’s performance
in a 3-dimensional space with 3 different height levels and 9 grids in the 2D space.
We used OptiTrack to provide the ground-truth data.
The result demonstrated that our system could achieve a dynamic tracking accuracy of 10.9
mm in distance, 3.7 degrees in yaw, and 5.8 degrees in pitch.
It can also determine whether one is looking at a particular device with a 93.5% accuracy.
When compared with the existing FMCW ultrasonic ranging solution - CAT, FaceOri achieved a
better tracking performance since it is more robust against the Doppler effect and line
of sight issues.
When compared with an IMU-based solution, FaceOri can achieve a significantly better
tracking performance in the yaw but not the pitch angle.
When compared with RGBD based solution, FaceOri is less accurate, but FaceOri has advantages
in a wider field of view, preserving visual privacy, and has the potential to support
the interaction with devices without cameras.
Finally, we explored the possibility of a sensor fusion method by integrating FaceOri
and the IMU.
Therefore, we can track the yaw and pitch angles with higher accuracy than the IMU-based
solution but consume less power than the sole ultrasonic-ranging-based solution.
Our work can support a wide range of applications.
FaceOri can capture activity or gestural input, for example, FaceOri can track and count user's
exercise by continuously measuring the distance between the earbuds and the smartwatch, enabling
a more interactive exercise experience.
FaceOri can also make interactions more context-aware and intuitive.
In the first example, the laptop locks and dims the screen when the user moves beyond
a certain distance away from the device.
In another example, the user is preoccupied and unable to provide touch input.
When the user looks at the device, it can turn on the screen to allow the user to see
their incoming messages.
In this multi-device scenario, FaceOri can automatically pair the mouse and keyboard
to the device that the user faces.
To conclude, FaceOri has the following 3 main contributions.
Firstly, we proposed a spatial input technique that applies ultrasonic ranging to enable
continuous head orientation and distance tracking using a built-in set of microphones in the
commodity ANC earphone.
Secondly, we implemented an end-to-end system and user evaluation demonstrates FaceOri’s
high dynamic performance in continuous tracking and binarized attention detection.
Thirdly, we explored the application space afforded by FaceOri with prototypes of selected
demonstrative experiences.
We hope techniques like FaceOri can help to make our experiences of the future more human-centered.
Please read our paper for more details.