Today, we introduce IMUTube. Automatic Extraction of Virtual on-body Accelerometry
from Video for Human Activity Recognition. This work is done by Georgia Tech Ubicomp,
CBA lab and Oxford OxMLSys lab. And, my name is Hyeok Kwon.
Wearable sensor dataset collection is difficult. User recruitment is expensive.
Annotation is time-consuming. And, privacy or other practical reasons sometimes
make recording Impossible. Those challenges make a number of limitations
in wearable dataset. Our benchmark datasets, such as opportunity
or pamap2, are small. They only have upto 20+ participants, few
tens of hours of recordings. Also, they are heavily label-imbalanced.
Opportunity dataset have over 70 percent of samples as Null classes.
Also activity classes has not expanded much from the past.
We are living in 2020, but we are still using benchmark dataset PAMAP2 from 2012, and the
activity classes in PAMAP2 has not changed much from 2004.
Those limitations make huge discrepancy between collected samples and the population dataset
in the wild. And, thus making limited room for improvement
for Deep Learning. Human activity recognition didn't gain much
from deep learning method. F1-score only had few improvements for Opportunity
dataset for past 7 years, in deep learning era.
Now that we looked into the limitations in wearable datasets, let's see activity datasets
in other communities. In computer vision community, they have massive
labeled online human activity videos. Activity classes go up to hundreds and thousands
from everyday activity to sports. And, each activity class have hundreds and
thousands of video samples, as well. Also, recent advance in computer vision technology
enables motion tracking from the video dataset. So, here is our vision, IMUTube.
Online video repository has unlimited amount of human activity videos.
Computer vision now enables full 3D human body tracking.
If we can extract virtual IMU data from the videos, then we can build HAR datasets with
far greater breadth and width. Conventional wearable data collection happens
in the lab. People do activity in the lab.
And, annotators observe and annotate. And, such limited small datasets are used
for HAR model training. IMUTube is different.
We retrieve online videos, and process 2D pose, 3D pose, and extract IMU from on-body
location. The large virtual IMU datasets from large
video repository are now used for HAR model training.
Here is an example of IMUTube dataset, two cyclists.
IMUTube pipeline reconstructs full 3D motion of two cyclists, and extract virtual sensor.
The example shows right knee accelerometer from right-hand cyclist.
Hi, I am Catherine Tong from Oxford, and now that we have introduced the IMUTube pipeline,
we want to ask, can we train HAR models from virtual data?
Now, to answer this question, we performed a set of extensive experiments, evaluated
on common HAR benchmarks. Specifically, we extracted virtual data from
videos coming from the Realworld dataset and we also created a large-scale in the wild
video dataset including videos from Youtube. Our results are very promising.
Here is a chart comparing performance of random forest trained on real data versus virtual
data in four different scenarios. And we see clearly that the F1-scores are
very comparable. Our results and experiments overwhelmingly
support our thesis that IMUTube can generate virtual data that is useful for training HAR
models. Firstly, we show that we can learn capable
HAR models just learning from virtual data along. And secondly, we also see evidence
that mixing virtual and real data for training can bring performance gains. Thirdly, we use
transfer learning to extend our use cases to even more scenarios that are realistic,
for example, when labels are missing. On the whole, we have shown that IMUTube can
be a viable method for automated large-scale data collection for activity recognition,
and, thereby having the potential to extend this field to everyday complex activities
and so many more opportunities. Our results are very promising, and we believe that a
collective approach is the way forward. we therefore call out to different communities
from computer vision, signal processing and activity recognition to extend this work in
the ways outlined in detail in our paper. We believe that this should lead to on-body,
sensor-based HAR becoming yet another success story.
These are our details and we welcome you to contact us for discussion, thank you.