Greetings everyone, I am Karan Ahuja, a fourth-year PhD student at the
Human-Computer Interaction Institute of Carnegie Mellon University. Today
I am going to talk to you about my project Vid2Dop, which utilizes mmWave doppler radar
for privacy-preserving human activity recognition.
This work was done with my collaborators Yue Jiang, Mayank Goel and Chris Harrison.
Millimetre-wave doppler radar is a new and promising sensing approach for human
activity recognition, offering signal richness approaching that of microphones and cameras,
but without many of the privacy-invading downsides.
This is because unlike images and audio which can tell you what the person looks like or the
contents of their speech, the mmWave radar sensor returns a radial velocity across a distance map.
This has made it very popular for ranging
and pedestrian detection based tasks for autonomous driving.
For example, the signal on the right here shows that “something”
4 meters away from the camera is moving at 0.7 meters per second in the direction of the sensor.
For the purposes of activity recognition, we want to characterize and identify these motion
profiles, and one of the best ways to do it is to train a supervised deep learning model.
This is an example of different activities and the number of samples for each class.
As you can see here, unlike audio and computer vision approaches that can draw from huge
libraries of videos to train deep learning models, Doppler radar has no existing large datasets,
holding back this otherwise promising sensing modality.
We considered possible sources of training data, including depth camera and motion
capture datasets, but these were much smaller than equivalent video data sources such as the
YouTube 8M which alone has about 250 thousand labelled video instances spanning 1000 classes.
Thus, we set out to create a software pipeline that converts videos of human
activities into realistic, but synthetic doppler radar data. Our synthetic dataset
consists of 10.4 hours of data curated from the different video datasets highlighted in the Table.
Here is an overview of our software pipeline which we will cover.
The first step of our processing pipeline is to take an input video and fit a 3D mesh
to humans in the scene.
We can then position a virtual camera to synthesize any view of the user.
This simple manipulation – essentially a form of data augmentation – means we
get a multiplicative effect on every second of the input video.
Next, we estimate the magnitude of reflected RF and any induced Doppler shift by computing
the radar cross-section and radial velocity of each vertex in the mesh,
with respect to a virtual doppler sensor.
We then perform back-face culling and also account for any occlusion from the user.
Now – left only with vertices that are visible to our virtual sensor.
We can compute a holistic synthetic Doppler signal by summing all the individual vertex reflections.
The envelope of this initial, simulated doppler signal roughly follows that of actual doppler
data. However, the correspondence can be greatly improved with an encoder-decoder model
trained with real-world data. This is the final output of our pipeline,
and what we use to train a convolutional neural network for activity recognition.
Once trained, our model can predict activity using live doppler radar data, as seen here. This model,
which recognizes 12 exemplary activities, was trained solely with data drawn from public
video datasets. Note that the video of the user shown here is for reference only, and
our approach is only using the doppler data for recognition, which is much less privacy-invasive.
Here you can see a few more plots of the synthetic and corresponding real-world doppler signal.
To evaluate our system, we collect data from 10 participants over 12 activities.
When we train our model on the synthetic Doppler data generated from the videos and test it
on the real-world data we collected in our user study, our system has an accuracy of around 81%.
When we add real-world doppler data into the training mix and run a leave,
one participant, out cross-validation the accuracy increases to 93%.
While our approach is more privacy-preserving in comparison to cameras and microphones,
the logging of activity data in itself can cause privacy concerns and implications. Such concerns
would have to be mitigated by privacy-aware design choices by consumers and manufacturers alike.