Hello everyone, my name is Lei Bai, a phd candidate at the school of computer science and engineering, the university of new south wales.
I am glad to present our work entitled “adversarial multi-view networks for activity recognition” at Ubicomp 2020.
The technical background of our work is human activity recognition, which is a significant step towards human computer interaction
and could enable a series of promising applications such as assistant living, skills training, health monitoring, and robotics.
Generally, there are vision based and sensor-based solutions for human activity recognition.
Our work belongs to the later one due to its low cost, less invasive and better privacy preserving ability.
Formally speaking, sensor based HAR system involves multiple devices attached to different parts of the human body.
Each device contains multiples sensors that keep recording human movements.
an inertial measurement unit typically contains nine sensors, that are 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer.
Thus, our task aims to learn a function F(theta)
that classifies the multivariate time series from d sensors with a sliding window T to N potential classes
In this work, we solve two major problems in sensor-based human activity recognition
The first one is accurate multivariate time series representation
which is difficult due to the complex spatial correlations among sensors and temporal correlations between time periods
The second one is the subject variance
which inherently exist in HAR because people tend to perform the same activities differently due to personal characteristics
such as gender, height, weight, and strength
Target on the first problem, we propose a multi-view data representation module
which extract different spatial or temporal correlations with specific CNN kernels of different shapes
including Single-Time-Spatial correlations, Single-Sensor-Temporal correlations
Multiple-Sensors-Temporal correlations, Sensor-as-Channel-Temporal correlations, and Time-as-Channel-Spatial correlations
For each view, we use a cascade CNN networks including three blocks to process the input data
The representation from each view are combined by a trainable Hadamard fusion module
to get the final representation of the input sensory segment
Our multi-view data representation module is robust
can capture more comprehensive spatial and temporal correlation in the sensory streams
To solve the subject divergence issue, we propose a Siamese adversarial framework
to train our data representation and classification module
The framework takes the data representation module as the generator
and introduce a discriminator to judge whether two representations come from the same person
In the training process, the MVRM try to generate representations
without personal characters so that the discriminator can not recognize
and the discriminator try to distinguish the representations from different person
The whole framework works together and help the data representation module be subject invariant
To evaluate the model, we conduct experiments on three datasets
that are MHEALTH, PAMAP2, and UCIDSADS, which have different scales and activities set
Moreover, in order to overcome the biases coming from different subjects
we update the leave-one-subject-out evaluation protocol to iterative leave-one-subject-out
which assesses the model iteratively with Left-One-Subject-Out protocol on every subject separately
In each experiment, we train the model from scratch and test the model with one subject’s data
Finally, we will get subject_number results for each model and take their average as the final performance
Our experiments have got several findings.
First, our model can achieve the best performance among a set of comparison methods in the three datasets consistently
Second, among the five proposed views, using the SST view, MST view, and TCS view together is the best
Third, the performance of HAR models on different subjects varies significantly
Previous conclusions made with the leave-one-subject-out protocol may be highly-biased
Besides, our Siamese adversarial training framework can boost the performance across the three datasets consistently
Last but not the least, there are some hard-to-recognize activities
which lead to the performance decrease of HAR models in some subjects
We call this phenomenon as class collapse
which should be a very important problem for further research
That’s all. Thank you so much.