Hi, my name is Minsuk in this video I will introduce RubySlippers an interactive
system designed to support content-based voice navigation for how-to videos this research was
done at KAIST with awesome collaborators Mina Hhuh and my PhD advisor Juho Kim
learning from how-to videos involve complex navigational scenarios for example people often
revisit previously watched segments to clarify misunderstandings skip seemingly familiar contents
or jump to the later parts to see the result and prepare for future steps a common strategy
people use in traditional video interfaces with mouse and keyboard is content-based referencing,
for example, peaking the video's content either by scrubbing or hovering on the timeline for
thumbnails or performing sequential jumps to move the playback position 10 00:00:57,280 --> 00:00:62,640 but how to videos for popular tasks such as cooking makeup and home improvements require 11 00:00:62,640 --> 00:01:08,000 manipulations with physical objects and involve physical activities and as a result
viewers need to use both hands to control the video and carry out the task
ideally voice user interfaces like alexa or google assistant can provide an opportunity to separate
the two activities controlling the video with voice and applying the instructions with two
hands the most straightforward and standard method of supporting voice interactions for
video interfaces is to directly time translate the timeline manipulation into voice commands
such as navigating the video with commands like skip 20 seconds or go to 3 minutes and 15 seconds
however this temporal referencing strategies require a different mental model for navigation
than directly manipulating the timeline because it limits the user's ability to peek into the content
through a formative study we have identified four challenges in supporting content-based referencing
in voice user interfaces for how-to videos first counter-intuitively users want to
use succinct keyword-based queries instead of conversational commands to alleviate the burden
of constructing sentences second users cannot recall the exact vocabulary used in the video
third users have difficulties remembering the available commands and finally unlike timeline
interactions voice inputs suffer from speech recognition errors and speech parsing delays
to address these challenges we built ruby slippers a prototype system that supports voice control
temporal referencing and content-based referencing through keyword-based queries
using ruby slippers viewers can describe parts of the video they would like to navigate to using
keywords ruby slippers returns a list of scenes resulting from the keyword-based search below the
search bar the corresponding locations on the timeline are marked with vertical orange lines
users can update the current query by adding replacing or removing keywords
the search result is shown in chronological order each item has a visual thumbnail
timestamp transcript and the keyword suggestions for further query specifications users can also
browse search pages with voice commands the recommendation panel right below the video player
provides suggestions which adaptively change at each interaction interval the system displays a
word cloud for initial seed it serves as a keyword summary that can help users recognize and remember
the main events happened in each video when user starts the navigation it shows available commands
so that the user can smoothly connect to the next interaction keywords that can be added
to the current query are suggested in support of users narrowing down or fixing the search
among the results we'd like to highlight the cognitive load measures from nasa tlx survey
our participants were able to perform all three navigation tasks with fewer commands
and less frustration using ruby slippers than the conventional voice-enabled video interface
also our participants enjoyed the keyword-based queries and other features of ruby slippers our
participants found it as convenient as a shortcut some participants found recovering from lost
navigation less frustrating and even useful one participant found this keyword-based referencing
had some learning effect and that they were able to remember keywords and the key concepts using
ruby slippers we'd like to thank national research foundation for their support in this research
we'd also like to thank fellow members of the KIXLAB for their valuable feedback and comments