Thank you for your interest in this talk.
My name is Ilmi Yoon.
I am presenting “Human in the loop Machine Learning to increase Video Accessibility for
Visually impaired and Blind users.
This is a collaborative project of several enthusiastic researchers from SFSU, USF, UW
and SKERI.
This work presents our design of an interactive system where both human input and the latest
developments in machine learning maximally complement each other to increase video accessibility.
We all know that videos have become an important mode of communication these days.
; Youtube has 5 billion views per day.
however, the visual information conveyed through these videos is often inaccessible to millions
of blind and visually impaired individuals.
who might significantly benefit from this content for education, employment, social,
and entertainment communications.
Videos are made accessible with audio descriptions that narrate visual scene elements such as
scene changes, actions, facial expressions, and important texts on screen.
A significant bottleneck in producing the audio descriptions is the time and cost to
produce them professionally.
Professional service is estimated at a rate of hundreds or thousand dollars per video
whilst the turnaround time is typically two weeks to a month.
YouDescribe was founded in 2013 to produce audio descriptions through crowd-sourcing.
It is a free platform where about 2500 sighted volunteers have described over 4000 Youtube
videos.
A challenge that has surfaced is that novice users typically state that describing the
video is difficult and time-cosuming.
It definitely bears a heavy cognitive load and is pretty tedious to record each narration.
Even though there are many kind hearted people who like to help, it is not easy to jump in.
In addition, there are wide ranges in quality of audio descriptions dependent on volunteers.
The goal of the project is to develop a Human-in-the-Loop Machine Learning (HILML) approach to produce
high quality video descriptions for blind and visually impaired users while maintaining
a low bar of entry for voluntary describers.
This is Our workflow.
We have interviewed several experienced visually impaired users and describers - to pinpoint
the fundamental features in lieu of supporting the describers while preserving the needs
of the audience.
Visually impaired users prefer concise and succinct descriptions.
Capturing texts-on-screen is important; These are all factored into this Video Understanding
module.
And then the interface presents novice volunteers with processed contents that can be revised.
Once the describers finalize the contents in text, the same texts are converted to speech
for the end users, and the describers don’t need to record
the narration.
The video is played on the left-hand side and the edit/input interface is on the right.
In the Human-Only condition, describers need to create their own scene segmentations and
generate all of the descriptions from scratch.
In the HILML condition, the scenes are automatically segmented for users.
Automated description from images in the video and text on screen is generated for users
in script editors, allowing users to edit the automated output.
This automated text can then be edited by the describer (the text shown here is the
automated output with no human edits).
We have 2 hypotheses.
Hypothesis 1: The Human-in-the-Loop Machine Learning (HILML) system, where scenes are
automatically segmented, text on screens are captured, and base scene descriptions are
pre-populated, is faster and easier to use for novice video describers than a human-only
control condition.
Hypothesis 2: The quality of the video descriptions and topic understanding is significantly higher
for the HILML system for blind and visually impaired raters compared to a human-only control
condition.
We have designed experiments to study our hypotheses.
▪ Experiment 1: We investigated the use of the HILML system for video description
generation for novice describers.
▪ Experiment 2: We asked blind and visually impaired participants to rate the videos created
by the describers in Experiment 1.
The first experiment studies the first hypothesis - supporting novice describers with the
features/tools discussed earlier.
To guarantee fair comparison, participants were asked to
describe the video in 2 conditions.
One with HILML support and one without.
To reproduce similar experiences, we filtered through a
pool to find two very similar videos.
Not identical but with similar contents and similar length.
We asked the YouDescribe content manager.
And she recommended the cooking tutorial videos, seeing as they were consistently found within
the wish list on the website.
We then picked two minute videos.
Both with music playing throughout and with text on screen appearing with instructions
and ingredient measurements.
The results from experiment 1 are shown here.
22 novice describers participated and they generated AD on 2 similar videos – one with
the HILML condition and the other with Human only condition.
We altered the order equally.
Qualitative data was collected through user surveys such as “Did you prefer using the
system with provided text or the system without?” and “Which system was quicker?”
The describers very clearly expressed a preference for the HILML conditions.
Also it is notable that user surveys illustrate that the describers were pretty neutral about
if the provided text was accurate.
This reflects the limitations of the Deep Learning technology that doesn’t capture
actions or detailed ingredients as accurately as hoped.
Even though the describers rated these Deep Learning generated text as neutral, it was
still beneficial to have guidance.
Participants commented -- “it was helpful even though I ended up using my own text.
I feel like it gave me some type of guideline as to what I should put, and not too many
details.”
Through the gray area though, the fact remains that while not entirely helpful, the design
wasn’t harmful, which is encouraging as Deep Learning captioning technology advances
very rapidly these days.
We performed a t-test in the meantime (in seconds) that describers spent on a video
across the HILML and Human-Only conditions.
Results showed that users described videos with the HILML condition significantly faster
than the Human-Only condition.
Results of the NASA-TLX responses also demonstrated that describers found the HILML system to
be significantly less cognitively demanding, less frustrating, and require less effort.
This is consistent with the subjective questionnaire data in which describers found the HILML system
to be easier to use, less frustrating, and more enjoyable than the Human-Only
condition.
Participants mentioned: “The provided text reduced the anxiety of doing it right or wrong,
and to me it helped me stay focused on the task that is asked for me to do.”
Experiment 2 was about the quality of the audio description rated by visually impaired
users, or in this case, raters.
All raters worked remotely and were each given half of the 44 videos described (as each of
the 22 describers had produced 2 videos).
Each Six raters were given videos produced by a group of 11 describers.
They did not know which videos had been produced by the HILML system or by the Human-Only condition.
Raters found the videos described by the HILML system to have a significantly higher quality
of video description and understanding of the topic than the Human-Only control condition
These promising results suggest that such technology can soon be used by many sighted
volunteers around the world in the creation of video descriptions for blind and visually
impaired users.
Our hope is that this will start decreasing the great digital divide between blind and
sighted users.
For future work, we are working on Improving the accuracy and quality of the machine-generated
descriptions
We plan to Compile and release a standardized, well-structured video description data set
under a free, open-source license
We are currently Integrating the HILML system into YouDescribe, called YouDescribeX
I'd like to thank my coauthors, who are shown on the screen here, for working with me on
the paper.
I also thank Ability Cetnral for funding our work.
And thanks to you all for your time and attention.