Hello, my name is Peter Washington and today I'm here to talk about
activity recognition with moving cameras and few training examples,
with a focus on applications for detection of autism related headbanging.
Activity recognition computer vision algorithms
can be used to detect the presence of autism related behaviors, including what are termed
restricted and repetitive behaviors, or stimming, by diagnostic instruments.
Examples of stimming include hand flapping, spinning, and headbanging.
Here we focus on headbanging.
One of the most significant bottlenecks for implementing such classifiers is the lack of
sufficiently large training sets of human behavior specific to pediatric developmental delays.
The data which do exist are easily recorded with a handheld camera, which is itself
shaky or even moving, posing a challenge for traditional featured representation approaches
for activity detection, which capture the cameras motion as a feature.
Here we see a representation of six sequential frames
using different popular feature representation techniques for activity recognition.
In subfigure A we see the unmodified sequential frames.
In subfigure B, we see the frames represented with dense optical flow.
In subfigure C we see the frames represented with Lucas-Kanade
optical flow displayed over the original image.
In subfigure D, we see Lucas-Kanade optical flow in isolation.
In subfigure E we see pose estimation using body key points displayed on the original image.
In subfigure F, we see pose estimation without the original image in the background.
All methods displayed here contain noisy features unrelated to headbanging,
many of which occurred due to camera movement.
We therefore chose to use the following feature representation
which is only consisting of the head key points and subfigures A through C.
We see the representation for clips containing head banging and in subfigures D through
F we see the representation for clips not containing head banging every five frames
in the original video our sample to generate the input sequence to the neural network.
We noticed that while headbanging clips show different key points for each frame,
non headbanging clips show comparatively stable key points between frames.
Here, we see the model architecture that we used.
We use the time distributed convolutional neural network trained to detect hand flapping from short
video sequences. Each input frame is fed into a CNN which extracts visual features from the frame.
These visual features are used as input to an LSTM network,
which makes the final binary prediction of "head banging" versus "not head banging".
On the self stimulatory behavior data set the mean F1 score,
which is a popular machine learning metric that is the harmonic mean of precision and recall,
is 90.77% across three cross validation folds.
The individual F1 scores per fold are 83.3%, 89.0%, and 100%.
We note that no child who was in the training set per fold also appeared in the test set,
giving us confidence about the generalization of our classifier.
Our next steps are to expand to a larger training set, expand to real world clinical videos
when applicable, and finally to apply a similar method as described here
towards other stimming behaviors such as hand flapping and spinning.
Thank you so much for your time.