Hello everyone, my name is Hui-Yin Wu and it's a great pleasure to do this
presentation on our paper Joint Attention for Automated Video Editing.
In this work we designed a metric based on cognitive attention that we then used to
implement and evaluate three automated editing methods for recordings of
corporate meetings.
This paper is accepted to ACM interactive media
experiences 2020 and is co-authored by myself,
Trevor Santarra, Michael Leece, Rolando Vargas and Arnav Jhala. We're
affiliated with North Carolina State University and UC Santa Cruz in the USA,
and Université Côte d'Azur in France.
So here's the idea: when the
number of people are occupying the same space their attention is drawn to the
salient focal points within the space these can be objects or other people.
This attention is expressed for example through head and body motions: people
will look at what others are looking at. Audio is another index: when a person or
object emits a sound, people will be drawn to and look at the source of the sound.
With the development of smart meeting technologies our capability to
record meetings and gatherings is greatly augmented, with opportunities
towards virtual conferencing, for example. This brings in challenges of digital
curation. Taking the example of the AMI database,
it features over 100 hours of recordings in smart meeting rooms, with seven
cameras: three overhead cameras in the center, left, and right, and four close-up
cameras on each participant, plus individual headsets. From all this
audio-visual data, how can we create an edit for someone who did not participate
in this meeting, showing at each moment, the most interesting event. So using the
idea of joint attention, we want to create an edit that shows the most
salient point in the room at any given time. To this end we propose and
implement three editing techniques based on joint attention: the audio-based edit,
rule-based, and long short term memory neural network based edits.
There's a wide expanse of literature concerning automated editing so here I'll just
touch on those that incorporate models and
indices of human attention. So in terms of interactive systems human attention or
joint attention has already been used for smart camera switching and for
introducing cuts into 360 videos. For example, on social cameras cameras and
mobile devices use the camera orientation to determine the focus of
the person operating the camera. In 360 videos, the user's head motion is used to
introduce cuts towards the next point of interest. In all of these works though, a
big challenge remains, however, in identifying the joint attention from the
content and the people within the videos and not necessarily the person operating
the camera or the viewer. Video summarization is another relevant field
that uses attentional cues with the goal to extract a storyboard or a sequence of
still frames that represent the entire video. Notably long short term memory
networks have been used because of their ability to model long term dependencies
across a period of time. However, the goal of video summarization is not really to
produce an edit from multiple camera streams, but just to select still frames
that are representative of the video content. Finally there's a lot of work on
automated editing systems for virtual environments so notably
placing and selecting cameras in the 3D environment that fulfill discourse
goals, and also ensure the spatial and temporal continuity of motions and
actions, and apply common patterns of editing from filmmaking to achieve story
or emotional goals. So 3D camera planning varies greatly from our current
challenge in two main ways first of all the cameras can be positioned anywhere
inside a virtual scene but in smart meeting spaces in real life they are
fixed in a certain position and second of all the absolute positions
of the cameras, characters, and objects in a virtual environment are known directly,
and completely accurate, whereas in real life, they can only be estimated.
The attentional features we use in our three automated editing methods are audio and
pose. Audio is pretty straightforward: it's the individual amplitude of the
headsets for each participant. For pose, we take inspiration from automatic pose
analysis software OpenPose in order to extract the facial landmarks of the
participants inside the videos, and using the visibility of the landmarks on the
face -- the two eyes, nose, and two ears -- we can approximately estimate the
orientation of the head and use that, given the configuration of the room, use
that to determine where the participant is looking within this environment.
To obtain the ground truth we asked a human expert to edit three videos based on the
definition of joint attention that is what is the focal point of the room at
any given time. So here's an example of the human expert edit: my opinion I think
it's better under the flip because whenever you want to talk okay so
then you can speak then you can close it but if you put it on the-- on the flip okay
then technically I don't think it's feasible
because most of the time if you speak then it will be recognized. But if you've already
got the remote control on your hand and need to open the flip to use the voice
why use the voice why not just use your hand? I mean the whole point of the voice
is if the remote control is sitting there and I'm too lazy to reach over and pick it up
The first approach that we implemented is the audio based edit it
is a naive approach that chooses the cameras stream showing the person who has
the highest microphone amplitude on their headset, and so there is no
smoothing, that is minimum shot length is one second, and contains only close-up
shots of the participants. Here's an example of the audio based edit:
My opinion I think it's better under the flip because whenever you want to
talk okay so then you can speak then you can close it but if you put it on on the
flip okay then technically I don't think it's
feasible because most of the time you speak then it will be recognized. But if you
already got the remote control on your hand and need to open the flip to use
the voice, why use the voice why not just use your hand? I mean the whole point of
the voice is if the remote control is sitting there and I'm too lazy to reach
over and pick it up.
The second method that we implemented is
the rule-based edit. Using the extracted pose we determine the focal point of the
room and give a score to each object and participant in the room then the
camera stream showing the object or participants with the highest score is
chosen. There is a smoothing of an average shot of six to seven seconds and
in the rule-based edit there are parameters to adjust such as the weight
on the pacing, and the importance of the pose direction. So here's an example of
the rule-based edit. My opinion I think it's better under the flip because whenever
you want to the talk okay so then you can speak then you can close it but if
you put it on the -- on the flip okay then technically I don't think it's feasible
because most of the time you speak then it will be recognized. But if you've
already got the remote control on your hand and need to open the flip to use
the voice why use the voice why not just use your hand? I mean the whole point of
the voice is if the remote control is sitting there and I'm too lazy to
reach over and pick it up, I can just use my voice.
The final approach we implemented is a long short term memory neural network.
We take as input both the preceding and current audio and pose data and train
the neural network to output the score of the joint attention of each item
or participant in the room. And so the intuition for the LSTM is that it
can model long-term decisions across multiple frames. There is no smoothing
involved here, so the minimum shot length is one second. And the neural
network was trained on two expert edits at any time and
tested on one ,and cross validated three times. So here's an example of the LSTM
based edit. My opnion I think it's better under the flip because whenever you
want to the talk okay so then you can speak then you can close it, but if you
put it on the -- on the flip okay then technically I don't think it's feasible
because most of the time you speak then it will be recognized. But if you've
already got the remote control in your hand and need to open the flip to use
the voice why use the voice why not just use your hand? I mean the whole point of
the voice is if the remote control is sitting there and I'm too lazy to reach over and
pick it up, I can just use my voice. Okay and now I'm supposed to -- I have one question
will there be anything written on the buttons? Like that people know or they
have to learn that from a piece of paper which button does what? -- Yes it will have
these buttons will have the numbers and all the rest of the buttons will have
symbols -- will have symbols so that -- so that the user -- like the user really knows
you know it doesn't have -- because we need the symbols because
we're going into an international market we can't have anything that's language
dependent but.
We evaluated the three approaches against the human edit and
through user evaluations. Compared to the human edit, the audio edit was more
jittery and contained only close-up cameras while the rule based edit was
less reactive to quick exchanges between the participants. The LSTM edit had more
adapted pacing and higher camera variety and with a pre-trained model could
conduct editing in real time. We then conducted a user evaluation comprised of
a pre-survey to see if there was any preference between different meetings
and a second part of nine pairwise comparisons of two clips selected from a
pool of 36 clips where there were three clips per meeting and four editing
treatments per clip. And so the users had to choose between any two pairs of clips,
which one was easier to follow and which one was preferred. The participants were
not informed that there were different editing treatments between the video
pairs. First looking at the pre-survey on the meeting
preferences, we found that it had no effect later in the pairwise
comparisons on whether a video clip was found easier to follow or more preferred.
In pairwise comparisons we found that the human expert edit was still
preferred above all other methods and the rule-based
edit was the least preferred of all, and LSTM performs better than audio in pairwise
comparisons. Though no two clips in the pairwise comparison were from the
same meeting, we regrouped the preferences for each meeting and found
that for consistently, throughout all meetings, the human expert edit was
always the most preferable and most easiest to follow, whereas the rule-based
edit was least preferred and the LSTM-based edit and the audio-based edit
closely followed the human -- preference for human expert edit.
There were a number of limitations for this work notably the training data of expert edits
was very limited and so we were not able to train on multiple kinds of
editing styles. And also pose detection for real-time systems is a bottleneck to
deploying our approaches in real-time that involved pose. So in the future
work we would like to evaluate on a larger scale to observe statistical
significance between different editing treatments on the perception of users
and we would also like to evaluate whether different editing techniques
would be more suitable or less suitable to different contexts.
Thank you very much, and please contact us if you have any questions!