This is Peggy Chi from Google Research.
I’ll be presenting an automatic approach for instructional video creation
given a Markdown-formatted tutorial document.
Last year at UIST,
we introduced a method that converts a web page into a short video
that presents key messages of the page.
Our output videos maintain the proper assets and visual design
extracted from the source webpage.
Moving forward,
we’ve been actively exploring to include more information from a document for video creation,
especially to consider a voiceover
and enable viewers to follow and interact with a video
through more control such as voice interaction.
One specific domain we look into is online tutorials.
During the pandemics, you likely have experiences to look for tutorials,
such as to repair something or to cook a nice meal.
Often, you may see a web tutorial like this example that describes a step-by-step procedure.
Each step may include a step image or a video clip,
with text instructions that describes the task from high-level to detailed tips.
Many platforms support community-based efforts
for contributors around the world to compose a tutorial.
Such platforms like wikiHow, iFixit, or Instructables
commonly follow the Markdown language format for collaborative editing,
similar to GitHub and Wikipedia.
However, to reach out to a wider audience,
whether for viewers who prefer to follow a video,
or have limited literacy skills to consume the text-based instructions,
it requires a significant amount of work for contributors to create a video.
In this work, we introduce an automatic pipeline
that extracts information from a tutorial and converts it into a video,
with a goal in mind for the task following.
We select text instructions from a document
and use Text-to-Speech to synthesize the voiceover,
aligned with the step images or videos with text titles.
Here we will show an example output video from our system.
Please observe how the Text-to-Speech voiceover matches the visuals.
Follow this tutorial to learn how to tie a bandana like a headband.
First,
Use a square bandana to form an even headband.
Then,
Lay the bandana flat and smooth out any wrinkles.
Next,
Bring the bottom corner to meet the top corner to form a triangle.
Make sure that all of the edges of the bandana line up when you fold it.
Fold the long end of the triangle to make a long strip.
Take the bottom of the triangle and fold it to form a strip about 2–3 inches wide.
Then fold the strip over and over
towards the point of the triangle until the fabric forms one strip.
Then,
Spray the folded bandana with hairspray.
In the output video,
you might notice that we added some connection words
such as “First” and “next” with pauses in between.
You might also notice how the voiceover helps focus on the visuals in the video
such as "lay the bandana flat"
as well as describing the details
such as “2-3 inches wide”
This experience can be quite different from reading a document
where our eyes navigate between elements,
This can also be different from using a conventional screen reader
to follow the content linearly without verbal cues of the document structure and pauses.
To help viewers follow the instructional content
we provided a user interface that presents the video, with a step-based timeline.
Users can navigate using voice commands
such as “go back to Step 2”,
or asking a question
such as “How do I turn it into a strip?”
where the system will jump to the specific point in the video.
Users can also reveal the text instructions in this mixed-media interface.
Now, I’ll briefly go through our automation pipeline
which includes three phases
from analyzing a document, enhancing the content, to creating a video.
Given a Markdown-formatted document
our pipeline extracts the components
including text, multimedia files, and styles
based on the Markdown annotations into steps.
We enhance the text content by adding connection words.
We then synthesize the voiceover by converting each sentence to Text-to-Speech audio files,
each has a duration in seconds.
Next, we select content into a video.
A text-based document typically contains a lot of text instructions and can be very detailed.
To make a video engaging,
our study shows that it is not suitable to present all instructions at once.
Therefore, we select the materials based on the time constraints.
We always present the step titles,
and based on the length of step images or videos,
we include supporting text to avoid excessive looping or static visuals.
Then, we allocate the materials into a video timeline
by aligning the voiceover, text overlays,
and images or videos with visual effects and pauses in between.
Our output video covers full or partial content of each step.
In the UI, viewers can choose to review the full step,
where the voiceover will go over the full text instructions.
Since the entire voiceover can be much longer than the source step image or video,
constant video looping might not be the best choice.
To make the visuals more engaging, we use Computer Vision techniques to identify regions of interest.
We adjust the camera motion to a looped video.
The automatic editing decisions can be visualized to a human editor for reviewing the details
including how each sentence is included or excluded in the video,
the loop times of a step video
and the duration of each element.
We ran our pipeline to generate videos for 40 tutorials.
Here shows a selection of the results.
We’d like to make a note that
the content from wikiHow is shared with permission from wikiHow for this project.
We conducted three user studies
to understand how viewers follow the instructional videos we created.
The takeaways include:
our videos provide good pacing,
support task following through the presentation and the voiceover.
Please find more details in our paper.
With that, I’d be happy to answer questions.