Hi everyone! I'm Gaurav Verma – a computer
science PhD student at Georgia Tech.
I'll be presenting our work on
Non-Linear Consumption of Videos Using
Sequence of Personalized Multimodal
Fragments.
My co-authors on this work are Trikay Nalamada,
Keerti Harpavat, Pranav Goel, Aman Mishra, and Balaji Vasan Srinivasan.
This work was done when all the authors
were affiliated with Adobe Research
in India. We start by noting that formats
like video and audio or linear in nature.
What we mean by linear is that the
viewing time for a video is in
proportion to the duration of the video
and they can only be consumed in their
original order.
The problems that arise due to this
linear consumption of videos
exacerbate as videos become longer in
duration.
Given all this, in this work, we ask the
question: can we overcome the
shortcomings of this linear viewing
experience?
And as a possible solution we present a
method to generate sequence of
personalized multimodal fragments.
We term this method as NoVoExp – which is
short for Non-linear Video Consumption
Experience.
Let's try to understand why such a
method that enables nonlinear
consumption
is desirable. First of all, we want to
consume videos in a targeted manner
and for that it would be helpful to
provide a quick overview of the content
that is covered in the video without
watching it in its entirety.
For instance, let's say we have this
video that talks about the daily life in
Palermo
that is around 10 minutes long. Our
method outputs sequence of multimodal
fragments that the viewer can skim
through to get a sense of the overall
narrative of the video.
Second, the viewer can also use the
sequence of multimodal fragments
to quickly navigate to the specific
parts of the video that they are
interested in.
Let's say i'm interested in the bit
about coffee and cake. So, I can simply
click on the fourth fragment in the
sequence
and the video will start playing from
there. For the earlier parts, I can just
look at the earlier fragments in the
sequence to get a quick
overview of what was covered in the parts
that I'm skipping.
And lastly, not all viewers are
interested in watching the same content.
For this video some viewers may prefer
to have multimodal fragments that
highlight the jolly and fun loving part
of the video,
while other viewers may want the
multimodal fragments to highlight
sentimental bits in the video.
To that end, our method also takes the
segment-level viewer preference into
account
while generating the sequence of
multimodal fragments. Here on the screen
you see the output for a jolly and fun
loving viewer.
And here you see the output for a viewer
who prefers to watch thoughtful and
compassionate content.
Now, before moving on to our proposed
method I'll quickly mention some of the
related work that has been done in
industry and academia.
There are platforms like YouTube that
allow the publishers to provide manual
annotations
so that viewers can navigate through the
video quickly.
But since this is manual, it's not really
scalable.
VideoKen provides automated annotations,
but they are in a textual format and
only work well for a certain type of
videos.
There has been prior research in
summarization, and multimodal
representation learning,
as well as visual storytelling that can
be used to generate a personalized
sequence of multimodal fragments.
But each of them individually miss out
on some of the other aspects
that are crucial for our purposes. For a
detailed discussion
of the prior work please refer to the
paper.
I will now discuss our proposed approach
NoVoExp.
The approach comprises of four major
steps. We will discuss all of them in the
next few minutes.
To help understand these steps I'll use
a running example –
an approximately 10 minute long video
describing the daily life in Palermo.
The first step is to extract out key
visual and auditory information from the
video.
For visual part, we identify the
keyframes in the video
and store them in a temporally ordered
manner. For auditory part, we use a text
to speech api to obtain the timestamped
transcript of the audio. Now, the
granularity at which the transcripts are
timestamped
and the granularity of the timestamps
for keyframes can be different,
so we create contiguous multimodal
clusters by combining
transcripts and related keyframes
together.
These clusters are contiguous because
our method ensures that each of them
discusses
parts that are coherent together. You can
think of these multimodal clusters as
parts that talk about
one separate unit in the video. But what
we have right now is a long transcript
and multiple images in a cluster. We need
to find a concise piece of text
and a single image that is
representative of this cluster.
Accordingly, the next part is focused on
selecting a representative multimodal
fragment for each cluster.
This is where we take into account the preference
of the viewer.
Using state-of-the-art representation
learning approaches, we find the
sentences and images that represent that
cluster,
while highlighting aspects that viewers
are most interested in.
They could be looking for content that
is either compassionate,
or fun, or even adventurous.
Once we have selected an image and
sentence,
we put them together to give us one
multimodal fragment for each cluster.
In the last step, we again take the
viewer preference into account to
generate a consumption sequence
that prioritizes quicker consumption of
segments that the viewer would be
interested in
and at the same time ensures that we
respect the original narrative in the
video
to ensure coherence in the final
sequence.
You must be wondering how we define user
preferences and how do we infer
attributes that align with these
preferences from the extracted images
and text.
Based on an existing annotated data set
called the Pitts Ads dataset,
we train a machine learning model to
identify the sentiments
that are evoked by the frames in a video.
Then we combine certain sentiments to
lie in three broad preference categories.
Jolly and fun-loving viewers prefer
content that is amusing and cheerful.
Adventurous viewers prefer content that
shows a lot of activity
and has thrilling moments. Thoughtful and
compassionate viewers prefer content
that covers emotions like empathy
and love. This figure provides a complete
picture of our methodology.
Now we will discuss the evaluation of
our proposed method and how it compares
to trivial adaptations of existing
methods.
For evaluation we constructed a dataset
by collecting videos from YouTube that
belong to different categories;
specifically entertainment, lectures, and
tutorials.
These three categories present a lot of
variation in content
and allow for robust evaluation – for
example,
entertainment videos involve a lot of
scene transitions
whereas lectures and tutorials do not. We
chose the videos to have sufficient
variation in their duration as well – some
videos were short,
others were medium and long in duration.
Our evaluation comprises of two parts. In
the first part we conduct automated
evaluation using key metrics,
and in the second part we conduct a
human survey using Amazon Mechanical
Turk (AMT).
Let's discuss the automated evaluation
first. Our proposed method
gives a sequence of multimodal fragments
as output for each input video.
We want the images and text in these
fragments to have relevance
with the original video. Also, we want
them to cater to the target viewers'
preferences
as well. We also want to assess if the
multimodal fragments can cover all
key parts of the video. We want the
multimodal fragments to be diverse and
not redundant.
As last two points, we want the image and
text within a single multimodal fragment
to be relevant to each other
and the text to be grammatically correct
and coherent.
Here are the results for the
entertainment category of videos.
As we can observe our proposed method
performs the best for many of the
metrics
that we just discussed. Even though the
proposed model is not the best in terms
of certain metrics like relevance,
NoVoExp is the only model that is able to
strike the reasonable balance across
all the metrics. Another interesting
observation here
is that even if we change the duration
of the video, the trends largely remain
the same.
Similar trends are observed for the
videos that belong to the lectures
and tutorials categories. Last part of
the evaluation was to first show the
original videos and the generated
multimodal fragments
to human annotators separately and ask
them to write a description for the
content that is covered in them.
Once we had the description written from
the annotators, we could quantify how
similar they are.
After this, we ask the annotators to
watch the video and the fragments
together,
and answer subjective questions related
to their coherence,
diversity, and alignment with the target
preference.
For all video types across all duration
categories,
we found that the descriptions that
annotators provided
by just looking at the sequence of
multimodal fragments
were quite similar to the descriptions
that they gave for the video.
Also annotators provided high scores for
relevance, preference alignment, coverage,
diversity, and coherence. To sum up the
presentation,
we proposed a method called NoVoExp that
uses state-of-the-art computer vision
and NLP techniques to enable efficient
consumption of linear media –
specifically videos. Based on empirical
and subjective evaluation we show that
NoVoExp is effective in generating
sequence of multimodal fragments
that can enable a personalized and
quicker consumption of videos.
Thank you for your time and listening! If
you have any questions or comments about
the work
please feel free to reach out to me or
Balaji.