Hi everyone.
My name is chitra.
Today, I'm going to present our work on teaching RF to sense
without RF training measurements.
This is in collaboration with Herbert, Belal and Professor
Mostofi at UCSB.
Wireless signals are everywhere these days and researchers have
utilized them for various applications, like imaging,
person identification and tracking, and crowd analytics.
In recent years,
we have also seen great advancements in machine
learning and deep learning.
So one natural question is: Can we use machine learning for RF
sensing applications?
Training a machine learning
algorithm requires a lot of data.
However, such a massive RF dataset is not readily available.
As such, one needs to collect massive RF data for many
different activities and configurations, which is not
feasible.
Also, due to the lack of such data,
the trained model cannot generalize beyond the training
setup.
So is there a way to use machine learning for RF sensing
without collecting RF training data?
In this paper, we show that this is indeed possible.
More specifically, we show how to train a human-motion-related
RF sensing system without collecting any real RF data.
We use online videos of human activities and translate them
to instant RF data.
This allows us to train any human-motion-related RF sensing
system without real RF training measurements,
while ensuring generalizability and robustness to environmental
changes.
Now, let's see how we enable this.
Given a video of a person performing an activity,
we first extract and align the 3D mesh of the person.
Then, based on the given RF setup, we simulate the received
RF signal that would have been measured if the person in the
video was performing the activity in the RF area.
This allows us to generate a large amount of RF training
data without collecting any real RF measurements.
Such data can then be used to train a machine learning model.
We next discuss our proposed approach in more detail, in the
context of gym activity classification.
In this task, we want to use WiFi signals to classify 10
kinds of workouts using machine learning.
First, we create a large video dataset of these activities
from YouTube videos.
Then, for each video frame, we utilize a state-of-the-art
computer vision algorithm to extract 3D mesh of the person
in the video,
Applying this to all video frames, we get a moving 3D
model describing the person's motion.
However, these 3D points lie in arbitrary coordinate systems,
depending on the camera view.
For example, these two 3D model face different directions.
In order to generate massive RF training data from these videos,
we need these 3D models to be in the same coordinate system,
independent of camera views.
We do so by performing eigen-analysis on the 3D torso
points.
The vector pointing towards the front of the person will
correspond to the smallest eigenvalue.
And the vector pointing towards the top is the one with the
largest eigenvalue.
This provides a view-independent coordinate
system.
This allows us to then align the 3D models, regardless of
their original camera views.
Finally, we translate the video to RF.
We simulate the wireless signal that would have been measured
if the person in the video was near WiFi Links
For this purpose,
we use Born approximation, which efficiently calculates
the reflected paths off of each 3D body point, and then
combines them to obtain the received signal.
We then use short-time Fourier transform to calculate the
spectrogram of the generated RF signal, which contains key
information on motion of different body parts, and can
be used for feature extraction and classification.
Here is an example spectrogram of a person performing deadlift.
We can see that the signal contains higher frequencies
during the second half of the exercise,
since the person stands up faster than they bend down.
Next, we validate our video-to-RF framework.
Here are the spectrograms for the real and video generated RF
data for 4 activities.
The real and simulated ones match closely.
Once we generate our RF training set, we extract a
total of 33 features from their spectrograms to train a neural
network classifier.
We also position the person in many different locations to
make training data more general .
We extensively tested our trained model with 10 test
subjects in 3 real WiFi environments and a total of
1543 repetitions of activities.
We achieve a classification accuracy of 81% when testing on
individual workout repetitions.
By fusing the classifications on multiple consecutive
repetitions of the same exercise, our accuracy
increases to 86%.
This is comparable to the state-of-the-art in RF-based
gym activity classification,
but without using any real WiFi data for training .
In conclusion,
we have proposed a framework for training RF sensing systems
without any RF training data
and by translating the vast available online videos to the
RF domain.
By utilizing this framework, we have, for the very first time,
successfully trained a WiFi-based gym activity
classifier, which only uses videos for training.
Moreover, our proposed framework can be readily
applied to other motion-based RF sensing applications.
That's all for my talk today.
Thank you very much.
You can check out our journal paper for more details and
results.