Hi my name is Chris Clarke and today I'm going to be talking to you about our paper on Reactive
Video. Videos are a really popular medium for learning
physical movements and exercise due to their convenience, cost effectiveness, and the wide
variety of videos on offer as a result of the low barrier for content creation.
However, videos cannot support users when exercising and interaction on-task is limited.
Reaching for the remote, or the device itself, breaks flow and immersion, and touchless technologies
such as voice can only be used for issuance of explicit commands.
To address this we introduce Reactive Video – a system which uses adaptive video playback
for a more immersive experience and to better assist in learning new movements.
Here is an example showing the system in action.
Adaptive video playback infers the pace at which a user is performing a physical activity
and dynamically adjusts the play time so that the video reflects the user's movements. The
system is designed to work with existing video content, providing a level of interactivity
without additional burden for content creators. We also incorporate real-time guidance and
feedback mechanisms to better support users and encourage safe and correct form.
When developing adaptive video playback we took inspiration from direct manipulation
video players… …which allow users to navigate through a
video by directly manipulating the content… In order to be able to use existing videos…
We extract the skeleton of the instructor, for example using OpenPose….
… and the user's skeleton is captured in real-time, in our case using a Microsoft Kinect…
The principle is to then match the user's joints with the instructors to adapt the playback.
However, there are unique challenges that arise when we consider adaptive video playback
in contrast to traditional direct manipulation video navigation.
Adaptive video playback consists of three stages…
The first stage spatially aligns the user's skeleton to the instructor's in order to compensate
for differences in physiological variability and/or different camera perspectives between
the input and recording devices. The second stage is to estimate the play time
in the video which corresponds to the user's movement,
We use probabilistic approaches, which represent information as probability distributions,
and which are typically more robust in the face of uncertainties than single best guesses.
We investigated two approaches – a discrete Bayes filter and a particle filter - both
of which are capable of tracking multiple hypotheses of where we believe the user’s
movement corresponds to. And finally, we have the rendering stage which
is where we determine what to show to the user.
To demonstrate the feasibility of adaptive video playback and evaluate the probabilistic
approaches we conducted an evaluation. We use six different exercises specifically
chosen to be simple enough for inexperienced users to complete, whilst also containing
challenging elements for the approaches. We recruited 8 participants
For each video, users had to mimic the onscreen exercise using four different modifications,
which include pauses, changes in tempo, and parts in which the user performs the exercise
in reverse. When we plot the play times estimations of
the algorithm against the original video, we would expect it to track according to the
tempo modification used. In general, when you take into account user
error, the approaches were able to successfully adapt their playback across the videos, and
overall the discrete Bayes filter outperformed the particle filter.
Throughout the development of Reactive Video we had several visitors to the research lab
to test out the system, and we gained some very interesting insights which require further
investigation. One of which was an interesting causality
dilemma. In Reactive Video, as the name suggests, the video is reacting to the user's movement,
and therefore the user is the causal source - user leads and the video follows. However,
when users are learning a new skill or movement they will attempt to mimic the instructor.
With adaptive video playback, this can lead to a causality dilemma - a subtle circular
feedback case where it is not clear who is leading, is it the user or the video…
I think this is a really exciting space to be working in and we have only scratched the
surface, and if you'd like to know more, to get involved, or if you have any other questions,
then please email me. Thanks for listening!