Hello, everyone. My name is Hongfei Xue, a PhD student in State University of New York
at Buffalo. Today I am going to present our paper “DeepMV: Multi-View Deep Learning
for Device-Free Human Activity Recognition”. Recently, significant efforts are made to
explore device-free human activity recognition techniques that utilize the information collected
by existing indoor wireless infrastructures without the need for the monitored subject
to carry a dedicated device. Most of the existing work focuses their attention on the analysis
of the signal received by a single device. In practice, there are usually multiple information
sources available for the task of activity recognition as showed in this figure. Each
of these devices can be regarded as an information source and provides us a unique “view" of
the observed subject. Intuitively, if we can combine the complementary information carried
by the multiple views, we will be able to improve the system performance.
To achieve this goal, we need to overcome the listed three challenges.
First, the data heterogeneity. The data heterogeneity is caused by different types of signals and
their different sampling rate. Second, the multi-view data fusion. Different
sensors may carry different amount of information. An ideal activity recognition approach should
be able to capture the variance of information quality among the sources and rely on more
informative ones. Third, environment-independent recognition.
the wireless signals arriving at the receiving devices usually carry substantial information
that is specific to the environment and the human subject.
The solution is our proposed DeepMV Human Activity Recognition System. As illustrated
in the image, the system is composed of 3 components, the data collection component,
the data preprocessing component and the classification component.
We collect various raw sensory data from heterogeneous wireless sensors in different environments
and forward them to the data preprocessing component. After feeding the deep learning
model with the preprocessed data, our system outputs the activity labels.
Let’s look closer into our deep model. It has a Feature Extractor to extract high-level
features from heterogeneous data and fuse them into a combined representation. The model
also has an Activity Recognizer to infer the human activity labels from the representation
and Domain Discriminator to infer the domain label.
In order to learn transferable features for different environments. During the model training
we make the feature extractor play a minimax game against the domain discriminator. Eventually,
the domain discriminator will be cheated by the feature extractor. As a result, the domain-specific
features are removed and the common environment-independent features are remained.
Note that these is a Hierarchically-Weighted-Combination module in the feature extract to selectively
integrate different views’ representations according to their contributions to the model
performance. If you want to know more details about our
model, please refer to our paper. This is our experiment settings. We collect
both Wi-Fi signals and Ultrasound signals from three different rooms. And there are
eight volunteers perform 8 daily activities and 1 undefined activity in the marked activity
area. Then I’d like to report the results. Note
that the data for the training is collected from room1 and room2; the data for the testing
is collected from room3. We report the result on 12 heterogeneous views and 9 Wi-Fi views.
We can see that our model outperforms the baselines with a large marge. We also conduct
many other experiments to study the limits of our model, please refer to our paper.
Lastly, we conclude our paper as follow. We identify the opportunities as well as challenges
in device-free human activity recognition with heterogeneous multi-view data.
We propose DeepMV, a unified multi-view deep learning framework, to learn informative representations
of heterogeneous device-free data. DeepMV can combine different views’ information
weighted by the quality of their data and extract environment/subject independent information
to improve the recognition performance. We set up a testbed using COTS WiFi and acoustic
devices, and collect real-world activity data. We empirically show that the proposed DeepMV
model can effectively recognize activities and outperform the state-of-the-art human
activity recognition methods on the collected dataset.