Thank you for that introduction.
I'll try to keep it a little shorter
so that we can wrap up on time.
So as he said, my name's Kyle Reinholt.
I'm from the University of Colorado Boulder.
And I'm here to present my work.
I describe on behalf of my co-authors and colleagues,
Darren Guinness and Sean Kane.
And this work involves creating accessible
touch screen artwork, utilizing behavior of sighted people.
So someone who's blind is browsing
the web needs to have access to the content that's
within the page.
This is most commonly represented
using a text description.
When a blind person encounters this image,
the computer would say, storm below Mount Fuji
from the series 36 Views of Mount Fuji.
This description doesn't tell us much about what
is actually in the image.
However, somebody who is going to describe this image
might start by talking about the giant wave that's
looming over the tiny mountain in the background,
And already, we have more information
about what's actually in the image
and where the objects are.
And this is the type of information
that we want to capture.
When a content author puts an image into their website,
it's up to them to provide a description for that image.
If they do not provide a description,
then it's as if the image is not even there.
So the content author is responsible for manually
submitting a description along with that image.
However, people don't want to put
in the effort to provide a thorough description.
And so to lighten the load, we use automated approaches
such as artificial intelligence like Microsoft CaptionBot
or Google's Vision API to provide descriptions of images
automatically.
Here CaptionBot thinks that this is a closeup of a logo.
And Google Vision API thinks that this is a shoe and a hat.
And Google Vision is 90% confident
that there is a shoe in this image.
And though this approach could lighten the workload,
the accuracy of these systems with artwork is still not--
well, the accuracy is just not there yet.
And so to bridge the gap between these automated and manual
approaches, we came up with the idea
of use-driven accessibility.
This is a concept where we can create accessible artifacts
from interactions with an artifact
without requiring any conscious effort or work.
So imagine that you're at an art exhibit with your friends.
And you're looking at an image and discussing it.
There's a lot of information that's going on
during this transaction.
And so if we could embed sensors in this image, such as eye
trackers and microphones, we could
record some of that data that's happening in this transaction,
and then use this data for later consumption.
This approach would give us additional details
about the image.
And we can also embed this hardware inexpensively.
And so the idea is to collect cited--
is to collect data from a sighted person looking
at an image while talking out loud.
And we know from previous work that when
somebody describes something, that they look at it
and they fixate on it.
And so this is an example from our work
that displays the same thing happening.
The timestamps from the description
are highly associated with the timestamps from the gaze.
And so our system, EyeDescribe, takes three things
as input, which is eye gaze, an image, and a description.
And this can be queried and aggregated in ways
that we can output a touch-accessible image where
someone who is blind or visually impaired
can physically scrub over the image
and hear descriptions of the objects
within it using this speech and gaze data.
And so we created this interface to answer two big questions.
We wanted to know if using eye gaze and speech data
could create representations or useful data.
And we also wanted to know if we made this interactive if people
would understand where things were in the image.
And so initially for the system to work
we had to collect gays and speech
data associated with images.
So we did a study where we had six people come in
and describe 12 different images.
We recorded the data using an eye tracker and a microphone.
So the eye tracker would yield an x, y coordinate
and a timestamp.
And the descriptions that we recorded with a microphone
were transcribed using IBM Speech to Text API, which would
yield a string and a timestamp.
And so this acted as the input for the EyeDescribe features.
And then users could access that data
using four different features.
So a person who's blind or visually impaired
might want a detailed description of an image.
So the first feature allows a user
to be read a description, a detailed
description of the image.
Where the second feature, the user
may not want to hear a bunch of detailed information.
And so this will allow them to get
a quick overview of the image.
And this feature lists the most frequently used words
associated with that image.
And then there's a third feature where the user can explore
the image by scanning around.
And then there's also the search feature
where the user can query the system
for the location of an object.
And now show you examples of each of these features.
Animals are posing--
So the user can perform a single tap
to here a detailed description.
--inside of the barn.
There is dog in the center, a cow on the right.
On the other side, there is a chicken
towards the front with a sheep directly behind it.
The user can double tap the image
to get a list of the common objects.
The objects in image--
dog, sheep, ram, chicken.
And the user can explore by touching and dragging
their finger around.
Goat looking at horse, looking at horse, looking at sheep.
And the user can say the name of the object,
and the system will verbally guide them to its location.
So in this video, the user performs a two-finger tap,
and then the system using directions
such as left, up, right, or down will guide the user
until it reaches the location of the object, where you can query
such as saying, where's the dog, or the cow,
or something along those lines.
So these are the four features in the interface.
And using it, we conducted a study
where we wanted to know if blind people using it
would be able to tell where things were in the image.
So we had seven blind participants.
There were three phases.
There were 11 images.
And the study lasted approximately 90 minutes.
Here's just an example of some of the images that we used.
They were from three categories such as landscapes,
still lives and portraits.
And in the top row, it's just some examples of objects
that we could tag using gaze and speech.
And so this study consisted of three different tasks.
The training phase was used so that people
could understand the features and gestures in the interface.
And then we performed a second task
where there were three images.
And participants could explore them for as much time
as they'd like.
And they would perform to think aloud
so that we could assess their reactions in real time.
And then it would also allow them to further familiarize
themselves with the interface.
And then for the third task, we would give the participant
the interface.
And they'd have five minutes to interact
with an image within both conditions
that I'll talk about in a second.
Then we would take the interface away
and ask them to place the location of an object.
Then we measured the distance between the placed object
and the actual object in inches.
And after all three tasks, we conducted
a semi-structured interview to further assess.
And so in comparing location accuracy,
the data shows that people were able to better recall
the locations of objects using these spatial features where
they could physically touch the screen
versus the nonspatial features where the system would just
read them a description.
So we performed a one way repeated measures ANOVA.
And this found that there was no significance
between the spatial and nonspatial conditions.
But what this says is that the images created by our system
were able to be understood at a similar level
as having a thorough description of the image.
And also the information that we received during our study
show that there was an overall positive feedback
about the system.
For example, P1 said it allowed me to get a mental image drawn
in my brain.
It's like the spatial features allowed the user
to draw a picture of the image in their head
because they could feel where things were located
in relation to one another.
P1 also said, I like the ability to explore and look for things.
And viewing images for the participant
became more of an active experience
rather than a passive one.
And P3 said the worst was when there was no explorer mode
and just going off of people's descriptions.
And this participant like the ability to explore and discover
things for themselves.
So this assessment or this feedback
showed that people enjoyed exploring artwork
on a touchscreen.
And so to further improve this interface,
I describe could be extended to adept descriptions
based on a gazers behavior.
Meaning, if these images were displayed on a digital screen,
as somebody is looking at the image,
they could zoom in to certain areas or cropped areas.
And maybe this the image could actually
query the gazer to explain more about certain features.
And so then on the user side, the user
could query the system for the gazers behavior,
such as, what did people look at the most?
Or what did people look at first?
Or what did people not really look at at all?
And it's also providing the ability
to dial into different layers of the description
or data such as colors and textures and even
subjective feedback.
And so in this work, we explored creating accessible images
for people that were blind or visually impaired
using sighted people's data.
And we believe that understanding how people engage
with a piece of art can provide unique insights
into how to make that art more accessible for everyone.
And I just wanted to mention that for this talk,
I extracted a very high-level view.
We had to perform a feasibility study and another user study
that we discussed in great detail in the paper
that I did not talk about today.
So please read the paper.
And that's all I have for you today.
And I'm open to any questions.
[APPLAUSE]
So thanks a lot, Kyle, for this interesting and short talk.
Great!
Are there any questions to Kyle?
Yes, please.
Just repeat your name so that we get even more familiar with it.
Hello.
So I'm [INAUDIBLE] from [INAUDIBLE]..
Actually, as I work with people who is visually impairment,
I have a lot of questions.
So I just-- maybe we'll talk later for the next one.
But the first one is in our previous work,
it's a really strong requirement for people
with visual impairment was to be able to use two hands.
So they can have one reference point
and makes a difference between two points
to have absolute and relative positioning.
And so do they explain some of the results?
And did you observe feedback?
So could you repeat the question in a different way?
Are you saying that people might use their hand
to reference a location and that might influence the results
in one way or another?
Or--
Yes, that's actually two question in one.
The first one is, do you [INAUDIBLE] such feedback from
the user--
the users of [INAUDIBLE] a blind person?
And did they-- do you think it may have changed
the result regarding positioning or relative positioning
of elements?
So I'm sorry.
I don't understand the question.
I don't know if it's because I can't hear you very well.
So I'd definitely like to discuss this offline
if you stick around after this.
I apologize for not having a good response for you up here.
[INAUDIBLE] later.
[INAUDIBLE] thinking, just a short question,
technical question.
So you recorded, I guess, I understood,
and I noticed that you always have
rectangular bounding boxes.
So of course you are only looking
at the fixations I assume.
So did you also consider to have a better estimation
of the extent of an object?
For example, adding some computer vision and methods
just to have an initial seed point like I'm
looking on the sheep.
And, of course, the computer visually algorithm
might be able to determine the boundaries of the sheep better.
Because, in cases that also in your example image where
you have a lot of overlap or maybe
cascaded objects in the scene, this
might be very beneficial to see the difference.
Mm-hmm.
Yeah, I really like that suggestion.
And for future directions, I think
that would be a great place to take this work.
So this was just early development
in creating a spatial map for images.
And just so that people could get
the relative spatial information from the image
to build a mental model of where things were.
But to that, also we analyzed the gaze
using four different metrics where
we looked at raw gaze accuracy.
So if somebody said dog, we would
look at how many gaze points fell within the bounding box,
and then divide it by the total number.
And then we also looked at fixations like you said.
And then we did rock kernel density estimation
where we did a kernel density on raw gaze points
and then also did the density on fixation points.
But to find the edges of the actual objects,
we did not use any computer vision.
Yeah. .
Good OK.
Any-- I have a last--
or you have a question?
No.
OK.
Last short question-- do you think
it's always helpful to let blind people look into our brains?
In other words, do our descriptions of sighted people
really help them?
Or maybe since they have a different mental model
of exploring the world at different representation,
it could be more beneficial.
So, yeah, there's a lot of representations
that are more beneficial.
But when it comes to exploring images on the web,
a lot of people don't put the effort in
to even describe images.
So this is just a way to extract more data,
as much data as we can, from people just interacting
with things to create a more accessible internet, especially
in the context of artwork.
Because most of the time, and if you go and look on you
know digital collections of artwork,
such as like the Metropolitan Museum of Art, a lot of times
you'll see that the descriptions are
intended for sighted people.
And they assume that the user is sighted.
And it'll talk about the historical details of an image
rather than where objects are located and things like that.
Right.
So let's thank Kyle again if there's no more question.
[APPLAUSE]
And also sorry I forgot that, of course, let's
also thank [INAUDIBLE].