Welcome everyone, we will now introduce our paper:
Contrastive Predictive Coding for Human Activity Recognition.
I am Harish Haresamudram, and this work has been done at Georgia Tech in Atlanta.
The proliferation of wearable sensors, such as inertial measurement units and accelerometers,
present on connected devices including smartphones, smartwatches and earbuds has rendered the large-scale movement data
collection a simpler process than the annotation. In the typical data collection protocols,
participants are often invited for collecting data in laboratory settings where video is
is recorded synchronously for post-hoc labelling. This process is challenging as it is expensive,
time-consuming and poses privacy concerns.
Due to factors like these, the publicly available labeled wearable sensor datasets
tend to be limited, covering a handful of activities in their study.
This disparity between the ease of data collection, and the laborious nature of
data annotation presents us with an opportunity. Can we leverage large scale un-labeled data to
train models that can improve activity recognition performance on the smaller scale labeled datasets?
To achieve this goal, we apply a self-supervised learning objective called Contrastive Predictive
Coding pre-train on unlabeled activity datasets
The intuition behind CPC is that predicting only
the next timestep requires the model to learn local variations. However, predicting farther into the
future, is a much more difficult task and requires the model to learn the slowly varying features
or the global structure of the time series data. This process results in more discriminative features.
We can begin with a brief overview of the self-supervised learning workflow.
It is a two-stage process where the network is first pre-trained on the unlabeled
datasets and then fine-tuned to the specific activities of the labeled dataset.
In the first stage, encoder weights are pretrained using self-supervision such that they are
useful for the downstream activity recognition task. For the second stage, the learned encoder weights
are frozen and only the classification layers are updated via backpropagation
to classify the activities present in the smaller scale labeled dataset.
CPC based pre-training begins with a raw input window of sensor data.
It is passed through a 1D convolutional encoder, to process the raw input window
and map each timestep into a series of higher dimensional z-vectors.
Then, we pick a random timestep ‘t’ within the input window and pass all z-vectors
prior to ‘t’, to the autoregressive network, which in our case is a GRU.
The GRU summarizes all the z-vectors into a single context vector Ct.
The prediction networks, W, are a series of linear layers that input the context vector
and predict ‘k’ future timesteps from t+1 to t+k.
Here, the starting point is chosen at random for each window, whereas the
number of future timesteps `k`, is kept constant during training and used as a hyperparameter.
The network is trained to minimize the InfoNCE loss between the ground truth future z-vectors and
the future predictions. This boils down to identifying the true future
z-vector for each timestep from a collection of negative examples drawn from the same batch.
After pre-training is complete,
we take both the 1D Conv Encoder and the GRU and freeze their weights.
A simple MLP is used as the classifier to recognize the activities of interest.
CPC is benchmarked on 4 datasets, including Mobiact , Motionsense,
UCI-Har and USC-HAD, covering mostly locomotion style activities such as walking,
sitting, running, etc.
First, we compare the activity recognition performance of CPC against the supervised learning
baseline, which is DeepConvLSTM. Here, we see that CPC,
which unsupervised, outperforms DeepConvLSTM on
two out of benchmark datasets, showing increases of over 3% in each case .
Here, we note that the performance of CPC improves over the DeepConvLSTM even though it has no access to label information
whereas DeepConvLSTM does. Further, as only the classifier is updated with labels for CPC
it used a fraction of the number of trainable parameters compared to DeepConvLSTM.
Next, we compare the performance of CPC against unsupervised learning baselines in activity recognition,
here we only show the best performing baselines for clarity.
CPC shows an improvement of 2.3% improvement on Mobiact over the Conv. autoencoder. For Motionsense
and UCI-HAR , we see more modest improvements over the baselines. For USC-HAD, the Conv AE performs considerably better than CPC. This showcases the effectiveness of CPC as an
unsupervised learning algorithm, as it outperforms the baselines on three out of four benchmark datasets.
Next, we take two examples to show the performance of CPC on a scenario of vital importance
to activity recognition – which is when very limited labeled data are available for supervised learning. Here, we pass small
amounts of labeled data to the models and plot the mean of the F1-score obtained after 5 randomized runs.
Here as well, we show the best unsupervised baselines for clarity.
On Motionsense, CPC significantly outperforms
DeepConvLSTM, obtaining improvements of over 10% when
just 10 labeled windows per class are available. Compared to multi-task
self-supervised learning, showing an increase of around 5% when a
100 labeled windows/class
are available for fine-tuning.
The trend follows
over to USC-HAD as well, with CPC performing much better than
DeepConvLSTM at over 5%.
Compared to the convolutional autoencoder,
CPC performs better when there are fewer labeled windows per class available.
This excellent performance of CPC, in a scenario of vital importance to activity recognition, really
showcases the utility of CPC as an effective representation learning algorithm for human activity recognition datasets
This is true in particular when there are very limited labeled data available for supervised learning.
In conclusion, we have adopted and applied the CPC framework to
activity recognition datasets.
CPC shows strong performance on both supervised and unsupervised learning baselines and
excellent performance in particular when there are very few labeled data available for finetuning.
Finally, the CPC can be easily integrated into Bulling pipelines.