Hi, my name is Ailie Fraser.
This is work that was done at a summer internship at Adobe Research,
in collaboration with Joy Kim, Valentina Shin, Joel Brandt, and Mira Dontcheva.
This paper proposes a human-in-the-loop approach for segmenting creative live stream videos,
to make them easier to browse and navigate after the stream is over.
People live stream all kinds of different stuff, from video games, to music, to fitness, to education.
and now more than ever, live streams are becoming a daily activity for many people,
whether that's watching them or making them.
And many are trying out streaming for the first time, as a way to share their knowledge and connect with other people.
One type of live stream that has been steadily growing in popularity is creative live streams,
where artists will broadcast while they work on creative projects such as illustration, graphic design, sculpting, and so on.
Creative live streams can be a valuable learning resource for viewers,
as they show the entire unedited process behind creative work,
not just the step by step instructions like a tutorial might.
And while we typically think of these videos being watched live in the moment while they are streamed,
many platforms also save the videos as archives for rewatching later.
People might do this to catch up on an artist they missed, learn how to do a task, or pick up some helpful tips.
And while these videos contain a wealth of knowledge, they are often multiple hours long,
and contain a lot of content that you might want to skip,
like the streamer saying hello to everyone, or other random conversation they have with the chat,
or repetitive or boring parts of the creative process.
So it can be hard to find a particular moment or section that you might be interested in watching.
Given this challenge, our research explores the question,
How might we make creative livestream videos easier to watch and navigate after the fact?
One way to do this is with a table of contents that divides the video into sections.
Some people manually make these for their videos, for example posting timestamp links in Youtube video descriptions,
or even including them as part of the interface on sites like Skillshare.
But for streamers who produce hours of content every day
and often don’t plan out the structure of their videos beforehand like you might when making a tutorial,
coming up with a table of contents can take too much time and effort.
But what if we could automatically generate a table of contents?
Creative live streams contain a lot of different data sources we could leverage to do this.
There's the visual content of the video itself,
there's the audio narration of what the streamer was saying, which can be transcribed to text,
if the streamer's using creative software we can sometimes get a usage log of their commands,
and of course there's the chat that shows when viewers engaged with the stream when it was live.
Our approach that I'll be presenting today focuses on using audio narration and command usage
to segment creative live streams,
and is motivated by prior work that segments videos
using either of these data sources, which I'll mention a bit later.
Prior work has shown how summary interfaces can help viewers navigate
educational livestreams, like in the StreamWiki interface,
or gaming livestreams, like with the HelpStone interface.
Motivated by these systems, our work explores how we might similarly summarize creative live streams
by leveraging usage data that we have from creative software,
to help viewers get that additional information about what they're watching.
To inform our approach, we conducted interviews with both streamers and viewers,
and we had some viewers segment a sample data set of livestreams
to see how people might approach the task of segmenting live streams,
and we had multiple people segment each video
to see if they do it differently or if there were trends in how people segmented.
A quick overview of our findings from this work:
We found that streamers saw value in having a table of contents for their viewers,
but they didn't feel they had the time to make it themselves.
Viewers, at the same time, do want some kind of overview of the creative process,
so that they can navigate to different steps and gain a quick understanding of what happened in a stream.
And finally we found that different viewers segment the same video differently.
Overall there was not a lot of consistency between different viewers in how they segmented a video.
This suggests that there is no one “right” way to segment creative live streams.
They often don't have as obvious of a structure as tutorial videos.
While this makes our task harder because we don't have any obvious ground truth to aim for or learn from,
we suspect it also means that having any reasonable table of contents will be better than nothing.
Our goal now is to automatically come up with a table of contents for live stream videos,
dividing them into sections based on the steps the artist takes in the creative process.
To do this we have usage logs from the streamer’s software that they recorded with a plugin,
as well as a transcript of what they were saying.
But we still want the streamer to maintain agency and control over their content,
so we allow them to optionally edit the generated sections and label them with meaningful titles,
which should be fairly quick and easy to do when the sections themselves are already made.
We designed a segmentation algorithm to come up with this table of contents, and it has 3 main steps:
First we identify candidate section boundaries in the video,
we identify two special sections that we call an Intro and an Outro,
and we combine the rest of the intervals in the video into a set of optimal sections.
First, to identify potential boundaries for sections,
we use the start and end of every sentence in the transcript,
as well as every occurrence of a command as long as it does not fall mid-sentence.
Putting these together we get a set of candidate boundaries
that indicate moments where a section could start in the video.
Next, we observed a few key types of sections that almost always occur in creative live streams.
First, when a stream first starts, the streamer will often put up a static image,
or something that says the stream is starting soon, while they prepare behind the scenes.
Typically, nothing happens while this screen is up,
so for the purposes of segmentation we can leave it out.
But even after the streamer starts up their video, they often spend the first few minutes
saying hi to viewers, introducing themselves, and like we see here
showing preparations that they've done beforehand, before they actually start doing the creative work.
Similarly at the end of a stream, the streamer often spends a few minutes
summarizing what they've done and saying goodbye.
So we call these sections an Intro and an Outro.
We can identify these sections by simply looking at the first transcript sentence,
as well as the first and last editing commands in the usage log.
Then, for the rest of the video, we need to figure out how to combine the remaining intervals
into a set of sections for the table of contents.
To do this we designed a dynamic programming algorithm
that starts from the first interval after the introduction,
and gradually builds up the optimal set of sections by adding in one interval at a time,
and figuring out how to combine it with the sections before.
Finally when we get to the end, we've segmented the entire video.
To determine what the "optimal" set of sections means, we defined a scoring function
that combines 4 heuristics based on our own experience as well as prior work.
First is the duration of a section. We don't want sections to be very short,
or very long, because we think these will be less helpful for navigating the video.
Second, prior work has shown that certain commands often indicate the start or end of a subtask,
for example creating or selecting a new layer often indicates people are starting a new subtask,
and saving the document often indicates they're finishing a section.
so we prioritize sections that start with layer selections and end with saves.
Similarly, people often tend to use different groups of tools or commands for different subtasks,
so we calculate how often different commands happen together
and then prioritize section boundaries that happen between less-related commands.
Finally, prior work has also found that people will often say certain phrases
when they're starting up a new section or finishing a section.
For example, sections might start with phrases like the word "start" or "next",
and end with phrases like "we're done" or "that's all".
So we prioritize sections accordingly.
To evaluate our algorithm’s output with streamers, we built a prototype interface
that shows each section to the right of the video as well as overlaid on the timeline,
and within each section displays all the commands and transcript sentences that it contains.
Once the streamer has labeled their sections, it looks something like this,
where we can see the title of each section in the video.
We gathered feedback from 2 streamers on our algorithm's segmentation as shown in this prototype interface
applied to their own videos, and we gave them the opportunity to label the sections
and edit the segmentation boundaries if they wished.
The streamers had different styles and types of tasks that they streamed:
Streamer 1 took a more instructional approach, explaining how to do what he was doing in the stream,
while Streamer 2 was more casual, conversing with viewers while also working on his comic illustration.
We found that both streamers thought the segmentation made their videos easier to navigate.
When editing, both streamers ended up combining some of the generated sections together,
resulting in fewer sections overall.
While Streamer 1 stopped at that, Streamer 2 also made some edits to the boundaries of the sections.
This is because the segmentations for his streams seemed a bit less effective overall,
perhaps because his streaming style was more casual compared with Streamer 1
who followed a more step by step process and talked more about what he was doing.
In addition, the task of poster design in Photoshop
also tends to involve different sets of commands for different parts of the process,
whereas comic illustration tends to use the same few tools throughout,
so the command heuristics for the scoring function may have been less effective for Streamer 2.
We also compared our algorithm’s segmentation to the segmentations we collected from viewers in our formative work,
and since these segmentations differed considerably from each other,
we didn't have a single ground truth to compare to,
so we instead compared how consistent our algorithm was with the other viewers.
We found that for 3 of the 4 streamers in our dataset,
the algorithm was reasonably consistent with the viewer segmentations.
Overall it seems to do best for streams that follow a linear step by step process,
for tasks that use different sets of commands for different subtasks (like with the poster design),
and for streams that include more instructional narration from the streamer.
Moving forward, some ways that we could improve and build on the segmentation
would be to incorporate other sources of data to help improve the segmentation algorithm,
which might make the streamer's job of editing even easier.
Especially for tasks like comic illustration that mostly use the same commands throughout,
perhaps having some visual understanding of what's going on on the canvas
or where the streamer is working on the canvas could help with this.
We think our approach could also be extended to other types of live streams
where we have some kind of timestamped metadata that we can work from.
For example in other creative software, many include save commands and layers
so similar heuristics could be applied.
Or even in less-related software, like productivity software such as Excel,
as long as you can gather usage data of some kind and come up with heuristics,
we think our approach could apply.
Finally, what are other ways we might help people navigate live stream archives, beyond a table of contents?
In our formative study, some viewers also showed interest in seeing a set of highlight moments from a stream,
or you could also imagine creating a shorter video that summarizes what happened in a long stream.
It would be interesting to explore how we might automatically generate such summaries for creative live streams.
Together with Adobe’s Behance team, we’ve also deployed a simplified version of the algorithm
more broadly on Behance’s live streaming platform that you can check out today.
and it shows how a stream can be divided up into chapters.
It will be interesting to see how this feature gets used and evolves over time.
So what I'm hoping you'll take away from this today
is that creative live stream archives are really a rich source of expert knowledge that are currently hard to tap into.
There is no one "right" way to segment a creative live stream,
but in this work we proposed one possible approach that leverages both command logs and transcripts,
and we find that it works reasonably well for streams where the streamer talks about their process
and uses a variety of different commands.
And finally, we can keep the streamers in the loop by giving them control over the segmentation.
This way the system does the difficult part,
and then the streamer can tweak and label results to their liking.
And with that, I'd like to thank all my collaborators who worked on this project,
and thanks for the opportunity to present.