We present our work, “Automated Video Description 2 0:00:05.240,0:00:10.930 3 existing video accessibility platform --> YouDescribe, 4 We are an interdisciplinary team across San Francisco State’s Computer Science, Psychology,
and Special Education departments, with additional 6 0:00:21.830,0:00:28.000 7 that roughly 285 million people worldwide 8 are visually impaired --> including 39 million blind individuals. In addition to thousands
of videos that are uploaded daily to social 10 0:00:38.410,0:00:44.000 11 streaming apps such as Netflix, Without audio 12 description to narrate visual information --> blind and low vision consumers can miss information
that is critical for education, employment, 14 0:00:52.860,0:00:58.780 15 description --> can be defined as “narration 16 added to the soundtrack to describe important visual details that cannot be understood from
the main soundtrack alone. This includes narration 18 0:01:09.670,0:01:14.859 19 that can be described in an audio track that 20 is synchronized with the video, 21 To support the incredible amount of videos uploaded to YouTube, YouDescribe is a free
online platform that hosts over 3000 sighted 23 0:01:28.530,0:01:33.329 24 joint effort between the Smith Kettlewell 25 Eye Research Institute and San Francisco State University. Anyone can create a free account
to describe any YouTube video, and anyone 27 0:01:43.950,0:01:46.740 28 0:01:46.740,0:01:52.719 29 unique visits every month from 126 countries --> 30 we recognize that novice describers can struggle with knowing how to produce high-quality descriptions,
and there is a significant bottleneck due 32 0:02:01.700,0:02:09.409 33 the YouDescribe wishlist remains undescribed! 34 We were inspired to develop a hybrid system: A NarrationBot that automatically generates
baseline descriptions whenever there is on-screen 36 0:02:19.689,0:02:25.689 37 description at any time during a video 38 This flowchart shows how with NarrationBot --> various modules
extract on-screen text detects scenes and 40 0:02:36.189,0:02:39.049 41 our system converts the descriptions from 42 text to speech and finds appropriate times in the video to play the descriptions. These
descriptions play inline, or in sync within 44 0:02:49.250,0:02:53.450 45 While the NarrationBot automatically detects 46 when descriptions are provided --> the human viewer can use the InfoBot to determine when
more description is desired at any point in 48 0:03:01.939,0:03:08.020 49 scenes of a dog --> woman, or car in a video, 50 I could activate the InfoBot which pauses the video and ask: What kind of dog is it?
What is the woman wearing? How big is the 52 0:03:36.109,0:03:49.480 53 0:03:49.480,0:03:53.530 54 current datasets for machine learning are 55 limited to distinct object types, Yet --> this type of hybrid system holds promise for achieving
a balance between automated and customized 57 0:04:05.180,0:04:10.370 58 recognize contextual clues to be more tailored 59 to needs for video description, We honor the input from blind and low vision users in helping
us develop these tools. Some of our participants 61 0:04:20.260,0:04:23.450 62 We conducted a user study with 8 participants 63 to explore how solo or tandem use of the NarrationBot and InfoBot impacted blind and low vision
users’ comprehension and enjoyment of various 65 0:04:31.530,0:04:40.950 66 differences when participants used the tools 67 in tandem and no significant difference between using one tool alone versus having no accessibility
tool at all.
This project was made possible by funding 70 0:04:54.220,0:04:58.540 71 Eye Research Institute,