After this talk you will know how to help people with visual impairments to understand gif contents
on their phone using Ga11y an automated gif annotation system for visually impaired users
hi everyone i'm Mingrui Zhang from the University of Washington and i'm presenting this work
in collaboration with min and Mingyuan Zhong and my advisor Jacob Wobbrock
so this talk is about GIFs we love gifs and we use it everywhere on the internet famous memes are
made in GIF we send stickers in gif to chat with others and sometimes we use it for fun moments but
for people with visual impairments gifs are almost unavailable for them because most of the gifs
are enabled when you are using a screen reader to focus on a gif usually it will say something like
"unlabeled" and here is an example in twitter "your message June 20th 21 attached GIF from you"
so it only say attach GIF from you without any annotation provided and some of you might say
"wait a minute why can't they generate captions automatically?" with advancement in computer vision
algorithms screen readers can now generate auto descriptions for static image well for gif they
are far more complex than a static picture because they are composed of multiple pictures together
this is when i use google cloud vision system trying to generate a description of this chief
a cat nodding its head but the description was completely wrong there are many challenges to
describe a gif automatically the first one is the dynamic scene actually a gif can include multiple
scenes from different sources it can be hard for the algorithm to connect them logically
another challenge is that many GIFs include characters or cartoons from clips like movie clips
and to make an understandable description the algorithm need to know about the background information
of those clips
lastly the most difficult part is a nuanced emotion expressed in many gifts
for example here the man is nodding and saying right but he looks reluctant
for now only humans who are aware of those cultural background or the meme backgrounds
can decipher those subtle expressions
so given those challenges how can we design a system that generates
annotations for GIFs? the answer is to use humans well a more realistic solution is to combine the
power of machine learning and crowdsourcing we could use computer vision to provide a fast
but rough descriptions and then let the crowd power to improve the description quality
and this is what Ga11y does. there's a 11 in the name Ga11y because accessibility is 11 characters
plus a and y and here's an overview of the system it has three components a mobile client
that is able to recognize gif on the screen using heuristics from app specific ui structures
and the user could double tap the gif to request annotations
the requested gif is then sent to the server
and the server will perform virtual similarity comparison using the perceptual hash function
if there is no similar GIF found it will return an automatically generated
description using the computer vision algorithm on a random frame the GIF will also be posted
on the annotation website anyone can go to the website to provide their annotations
once the GIF is annotated the human annotation will be returned if it's requested a second time
so here is a demo of requesting annotation with that automatic description returned "your messages, get GIF description"
Automatically generated caption is diagram. Best guess is thompson. The label is cartoons and gestures.
and here is a manually labeled gif uh annotated from our website "get GIF description button"
Last GIF:A brown dog wearing a small hat sits on a chair inside a burning room filled with fire and smoke.
It stares blankly ahead and then picks up a mug from the table in front of them and says "This is fine."
They smile and look at the mug while the room continues to burn..
Usage Scenarios: It can be used by people going through a difficult situation but act...
okay um it's too long so i didn't play it fully but you can get the sense that this
description is very detailed and explain the gif. one challenging issue for crowd sourcing
annotation is to gain high quality annotation To ensure high quality annotation we need to design
a good instruction task
we explored three instruction styles freeform structured and semi-structured
for freeform it's the same as a base instruction
that the crowd workers will freely compose their annotations with the instruction
for structured we derive several important points for a gif annotation and let the crowd workers
fill each point with a structured answer and we then concatenate those answers and some of the
questions include what are the characters what are the activities in the gif
if there is any background information there any text
and the semi-structured task design actually combines
the structured and the free form and but still let the annotators freely compose their annotation
we tested these three task designs with some of the example gifs on mechanical turk platform and then
we evaluate the annotations we cited users on the same platform as well as blind or low vision users
via remote meetings here is the feedback from sighted users the semi-structured annotations
gained best scores in each category and here is the evaluation from BLV users they also preferred
annotations from the semi-structured tasks they mentioned that the structured annotation
sounds like robot so we just deployed those semi-structured task design in our annotation
website after having the system we invited 12 BLV users from both china and the us to install Ga11y
on their phone declined and let them use for three days during the time we act as crowd annotator
to provide annotations for their requests and the user system quite often over the three days
they requested for over 500 GIF annotations although the machine generated annotations was
not accurate participants appreciated how fast the result can be proceeded here is an example
of the human annotation pieces participants mentioned they were very accurate and very
detailed but some of the participants complained about the length of the annotation they were too
long and hoping that Ga11y could provide a shorter version of those annotations sometimes machine
alone cannot solve the problem and especially when it includes cultural or subjective content
like gifs so we need to make use of both machine and human power to create adaptive and inclusive
interfaces as one of the our participants said "it was important to speak the language others speak."
for more technical and evaluation details please refer to our paper we also open
source the Ga11y system and welcome everyone to extend the idea to other applications thank you