Instructional makeup videos are extremely popular on YouTube catering to a diverse
audience and garnering millions of views.
They're effective for conveying actions that are difficult to describe
with words or static images alone.
However, it's often difficult for people to follow along with an instructional
makeup video using a timeline based UI.
In particular, the linear nature of the timeline makes
it difficult for the user to..
Follow along at their own pace.
Jump ahead or replay steps since there are no step boundaries.
Or to get a sense of how many steps there are, what kind of
makeup is applied and more without watching the video through first.
In contrast, many of these limitations of instructional videos
or affordances of traditional texted tutorials, which allow users to
easily follow along at their own pace.
However, traditional text tutorials are much less expressive
compared to video tutorials.
Our work aims to combined elements of texts and video tutorials
to get the benefits of both.
In order to do this, we need to first segment the video into steps.
Prior research in cognitive psychology suggests that people
mentally segment procedural tasks into object action hierarchies.
In a formative study of makeup tutorials, we found that the coarse level object
categories correspond to facial parts.
Specifically the eyes, broader face and lips.
While fine grand action steps entail manipulations on these facial parts,
such as apply the liner on the lash line.
We also found that fine-grain action steps are defined by a product such as bronzer
in step one, and an action such as apply.
A step change occurs when the product being used changes.
For example, going from bronzer in step one to blush in step two.
Using this structure, we developed a pipeline that: given an instructional
makeup video as input, automatically segments the video into a two level
hierarchal tutorial and visualizes the resulting hierarchy in a mixed media UI.
Now we'll go through our UI in more detail.
At the top left is the video metadata.
And then below that is a video player.
At the very bottom, the overview timeline gives a breakdown of the
tutorial by facial part sections.
Finally, on the right is the steps panel, which displays all the steps.
Users can browse it's tutorial by scrolling through the steps and the
steps panel and reading the summary text.
They can also hover over the step thumbnail to preview the video
of the step as an animation.
They can click on a step to play it.
As always I've applied an eyeshadow primer all over the lid.
We've also implemented the following voice commands with which users
can navigate the interface.
Notably users are able to navigate with high level intentions by searching
steps, facial parts, products, and more.
Now, I'll talk a bit more about how we segment the video into fine-grain action
steps, as well as coarse-grain events.
To first identify the beginning of a new action step, we look for product
changes in both the audio and the video.
For example, to find a new step boundary in the video, we leveraged an observation
that before creators use a product, they tend to introduce it visually by
featuring it prominently in the scene as seen in this first row of frames.
In contrast when they use the product as seen in the second
row, the face is shown prominently with an applicator moving over it.
We first run an object detector to detect all products in the scene.
Since frame F does not contain a product, we judge that is not
a product introduction frame.
Next, we run face detection to see if there are any faces on the scene.
Since frame A contains only a product and does not have any faces
in the scene, we judge that it is a product introduction frame and
thus marks the start of a new step.
Otherwise, if there is a face, but it's not overlapping with the product as
in the case of frame B, we also mark that as a product introduction frame.
If the face and product are overlapping, we calculate the motion
of the product against the face.
If it's fairly, still, as in the case of frame C, we mark that
as a product introduction frame.
Otherwise we assume that the video shows a makeup application action and move on.
After we have all the steps, we cluster them into facial part groups.
For a given action step, we identify the facial part by looking for
sub components in the step text.
And then we grouped consecutive steps with a common facial part together.
To validate our system, we had eight users use the system to follow two tutorials:
one in our UI and one with the YouTube UI.
Overall, users found that our UI was easier to navigate and felt more
confident in their results when using it.
Additionally participants felt that the automatic step
segmentations were fairly accurate.
We also asked users to compare the hierarchically segmented
tutorials with a one-level version showing only the action steps.
All participants expressed preference for the hierarchal version.
Many participants appreciated that the hierarchal version helped them to
quickly get an overview of the tutorial.
For participants who only wanted to follow part of a tutorial, the
hierarchal version helped them to find sections of interest much more quickly.
With that thank you for listening and please find more
technical details in our paper.