- All right, thank you.
So yes this is the work of my students,
Leo Neat, Ren Peng and Siyang Qin,
supported by the NAH.
So this is work about access technology
for people who are blind.
And if you are blind,
there are a bunch of things that are difficult to do,
that I released in no particular order
in this acuity chart here.
And what I'm really interested in in this work,
is this one here, which is,
the ability of reading signs posted on walls.
What in the computer region is sometimes called scene text.
Text that can appear everywhere.
There are lots of these types of signs
in public buildings, like this one here,
and made for people who are sighted.
It would be great if blind people
were able to access those.
Who don't need to change infrastructure,
they are right there.
So how can a blind person read a piece of text?
So there are apps on smartphones,
and they work really well.
Optical character recognition, OCR apps.
But there is a problem.
And the problem is a problem that I like to call
computer vision without sight.
So, if you're a sighted person,
you're taking a picture,
you normally look through the viewfinder,
or the screen where you're pointing.
But if you're blind, you don't know
where the text might be,
and so in this example here,
a blind woman might be looking for an exit sign,
the OCR might be able to read it
if it was aimed at that,
but she doesn't know where that is.
So that's kind of like a chicken or egg problem,
that all camera-based systems for blind people
have to deal with.
So let me show you an example here
of computer vision without sight.
And this is based on the, probably the most used app
for text for blind people,
that's made by Microsoft, it's free,
it's called SeeingAI.
It has a modality that's called Short Text,
and it does just that.
It just reads any text that it finds.
So here an example of a blind person using Short Text.
So you see that worked fine.
But if you listen to the text,
it's kind of difficult to make some sense of it.
Why so?
Because this is the part of the image
that was seen by the camera.
So the camera was not framing the text correctly,
but there was no way that this guy
could have known, because he can't see
through the viewfinder.
Okay, so you need to scan the scene with your camera,
but here comes another problem.
That SeeingAI like other OCR system,
do not work while the camera, you move it.
So this is video I took,
so long as I move the camera,
the system is not reading anything.
As soon as I stop the camera.
Why is that?
Well, because as I move the camera,
the image undergoes motion blur.
And so here you see there is some sort of text,
but it's blurry.
I can't really read, probably you can't,
OCR certainly can't.
So, that's kinda like a double whammy.
You need to scan the scene to find the text,
but text is not gonna be found
if you are moving the camera.
So to deal with this, we decided
to play with something that's called a text spotter,
text spotters are algorithms that are able to detect
the presence of text, not to read it,
just to figure out that there are some patterns,
like Nicky's before,
we could tell that a blurry thing was text,
but we could not read it.
They can be very fast, they can be very robust.
We developed one of our own,
and we decided to apply it for this system.
So our text spotter is based
on a fully convolutional network.
It doesn't run on the smartphone
because of memory problem.
It could, just we didn't have the resources
to make it work on the smartphone.
So what we did instead,
we streamed images from the smartphone
to a server that then ran that,
and then the server would send us back information
about any text it found.
And the whole thing worked
at about two frames per second.
Now speed is important, because if you are scanning
something in front of a wall, for example,
if the frame rate is too low, you might miss something.
This is some examples of images
whose text has been spotted by the algorithm.
Its really good.
For example, it basically recognizes
different types of stuff, different types of background.
Like the fire thing, the title.
It works also when the text is only partially visible,
and critically, it can also detect text when its blurry,
which is exactly what we want to do.
So how do we use this is a full system?
Well, look, this the microsystem, the SeeingAI system.
There's an image, an input, it runs OCR,
if it finds something, it read it aloud.
Our system has both text spotter and OCR,
so its a little bit more complex.
So it requires action from the user.
So it works in this way.
The camera looks at the images,
as soon as it finds an image with some text,
the phone vibrates for a little bit.
That prompts the user to stop the camera,
so that an image of the text,
the highest resolution that is stable can be taken.
So the text spotter just works as a prompter
that there is something there, so stop the camera,
we take a good picture, now we should be able to read it.
And here is an example of our system,
we call it Spot+OCR in action.
So you see this young man who is scanning that wall,
and at some point you notice he stops
in front of the name tag,
that's because he felt the vibration of the phone,
and after that an image will be taken,
it will be read.
Okay, you also heard that there is some accidental
information in the interface.
On top of this, we also built another system
for open spaces.
Now you could just walk around a wall in an open space
to try to find text.
Blind people tend to get disoriented in open spaces,
so we thought about a different mechanism,
one in which you just sit in the middle of the space,
and you just take very high resolution pictures
sampling the whole space,
so that you'll be able to capture all text.
Now, if you're taking high resolution pictures,
the OCR we are running, the Google OCR API,
will take up to several seconds,
but that's okay, because you are careful
to sample the space correctly
so any text that will be there will be read.
The system has a mechanism that allows a person
to take pictures at exactly the right time,
so as the user starts taking a picture,
system prompts the user to rotate the camera,
and it prompts you when to stop,
a picture is taken, prompts the user
to move again, and so forth and so on,
until the whole space is being covered.
And agan here, there is no text spotter,
we're not gonna have moving images,
we simply space the high resolution.
Okay, so some experiments.
Actually these are the type of images
that we didn't capture using this guided OCR system
by a blind person, you'll see a little yellow arrow
that meaning the prompt to turn the phone around,
in practice there will be an audio prompt.
And I'm also showing which parts of the image
have been detected, the text was read by the OCR.
Notice to the left, that's a lot of circles,
that it being read like "0000,"
that's all zeroes too, so yeah, to have
an option so that you stop this very long list
of things being produced.
The system reads everything that is found
at each picture.
Okay, so we ran experiments with seven blind participants.
And they're shown in the next picture here.
Now we done the full 360 degree circle.
So these are our participants,
and what they did,
and I'm gonna be very brief here,
details in the paper,
they first tested the system in two corridors,
actually we had two corridors on three floors,
a narrow corridor and a wide corridor.
They were tasked with going into the corridor
back and forth and find all of the text
they could find on the left and the right wall.
So they experimented both the SeeingAI system
and our Spot+OCR system in different types
of corridors.
We also had three different open spaces,
elevator lobbies, like six by six meters,
in which they tested in three different floors
all three different systems,
SeeingAI, Spot+OCR with text spotter, and Guided OCR,
where you just sample the space.
One thing that we found out,
some people, probably frustrated
because SeeingAI was not able to find text,
they decided to explore the environment with their hands,
here is an example.
So this person found that next to each door
there is a name tag, so as soon as they found the door,
with the white cane he would start exploring the wall
and then he knew where to aim the camera at.
My students thought it was cheating,
I thought it was not cheating,
you do your task the best way you can.
But of course that complicates the analysis of the results.
So here are the results.
What I'm showing here first is a test in the corridor,
the first plot shows the recall rate.
How many pieces of text there were
out in the environment,
name tags and the like, were correctly read,
read in their entirety.
So this I'd normally show pieces of text,
but if only half, the right half of those
were read, because just a camera
was aiming a little bit much to the right,
that would not count as a success.
So we have the seven participants
and highlighted those who actually
did some tactile exploration to find text.
They only did it with the Microsoft app,
with the SeeingAI.
Green bars are for our system, the blue bars
are for Microsoft SeeingAI,
so you see that for those who did the tactile exploration,
they were very proficient, they had good recall rate.
Those who didn't do it, the SeeingAI system
basically did not work.
Some were not able to find a single sign.
And again, because if you're moving the phone
at a certain speed, it will not find anything.
I'm also showing the speed, the hourly speed
in the corridor in meters per minute,
so this is a mistake, a typo.
The use of SeeingAI made for a faster exploration,
and that's significantly faster.
And this is because our system requires
action, you feel a vibration,
it will stop the camera, and aim.
And there are also some false alarms,
so the precision of the Spot+OCR is only 64%,
it means that many times you feel the vibration,
you stay there, wait for the picture,
and the system says, "No text found."
Why so?
Maybe because of a false positive in text spotter,
maybe because you find some text,
the OCR simply was not able to read
because, for example, it was too far.
Our results with open space system,
where we tried all three mechanisms,
it turns out that again, Guided OCR,
the one in which you take pictures,
correct the sample and angle,
gave significantly better results in terms of recall rate.
No major difference in terms of time.
There was an exit survey, I will summarize here.
Our participants found the Guided OCR significantly
easier to use than the others.
They really liked the vibrational feedback,
the fact that at some point the phone vibrates,
they know there is something there.
Now maybe that's not really right away,
but you can try other times to explore the space
until at some point it might be able to read it.
Some desiderata feedback, reduction of false alarm,
confidence indication.
Would they use it in a public setting in people around?
Probably not.
They told us that, and it's very common,
by the way, with all camera systems,
if there are people around you,
you don't want to be seen testing,
things taking a long time,
that's pretty typical, I'll say.
However, if there is nobody around
and you're looking for the exit sign,
you might want to use a system like this.
That's the same for SeeingAI and our system.
And I conclude with these observations.
So the Spot+OCR system with text spotter
is definitely better at finding text
than SeeingAI, unless you start exploring
the scene with your hand.
How many people will do that?
Not clear.
Not everybody wants to touch all of the walls
not knowing exactly what is there.
It does require, however, significantly more
user interface action by the user,
and it takes more time.
In general, to find text, distance is very important.
If I'm too far from the text,
I'm walking in front of the wall,
and I'm too far away,
there is not enough resolution for the OCR
or the text spotter to work, however,
if I'm too close to that, because of the limited frame rate,
if I'm moving the camera like that,
I might miss some text, because my camera
is in between, in the gaps between two frames.
The most proficient users tested
at multiple distances, in fact.
Another thing is that the general concept
of computer vision without sight,
framing text is very difficult.
In many cases the poor results we get,
both with SeeingAI and with our system,
were because only half of the text was seen,
like in the first video that I showed you.
It is very difficult for a blind person
from the operator system to understand
how you should move the camera.
So this is an opportunity for better feedback.
We have experience with a similar system
in which we gave feedback to blind people
on how to move the camera to better frame a document,
and we're planning to explore that type of interface
for our system for seeing text.
And I'm done here, I'd be happy to answer your questions.
(audience clapping)
- [Kang] Hello, my name is Kang Shen,
I work for Argo AI, and my question is,
have you thinking about generating
a panoramic picture first, and then doing the OCR?
- So the question was why taking multiple pictures
through an OCR and trying to use those,
and why not doing it in panoramic?
That's where we started from.
I wanted to do panorama, panorama is condensed information,
right, so instead of having multiple overlap pictures,
they become one.
So in a sense they make for a faster processing,
you just process one general image
rather than multiple images.
What we found is that even though panorama
generation is possible, it still needs guidance
when to take pictures, so that they overlap enough.
When you do your stitching in a panorama,
sometimes there are some tiny defects that occur,
that's normal, that's where I started.
If text happens to be there, OCR is not able to read it.
So we had a number of false alarms in that situation,
so that's why we decided to go for this lower system.
I like the concept of redundancy a little bit,
because I don't know if it proves visible
in one of those examples, but for example,
text plus seeming one was detected
in one picture, but not in the other picture,
they still saw the text, that's
because the difference in the different conditions.
But this is definitely a proper question,
and I think eventually, ideally, that would be the best,
but it didn't work well for us.
- [Audience Member] Thanks for your speech.
I have a question, in your experience,
do you think that having, for instance,
an RFID instead of having to scan text
is better or worse, or do you have experience about that?
Thanks.
- It's doing the exit signs today.
So having an RFID at the sign,
or even a QR code perhaps, I would say a couple of things.
First of all, an RFID might not be the best example
because you need to be pretty close
to be able to read it, probably
less than a meter, so we are back to square one.
I don't know where the RFID is until I get there, right.
That's the problem with RFID with blind people.
The second issue is that somebody needs to put it.
So in the old times, back about 10 to 15 years ago,
we thought that every single item in the market
would have an RFID because we thought
because the SR1 sent, that didn't happen.
So the whole work here starts from the assumption
that there is not gonna be any infrastructure change,
the signs are already there.
So certainly adding things, like RFID,
and maybe a IB card or something else,
will simplify things, but that's an infrastructure change,
it cost money, somebody has to do it,
and it's much less sustainable in my opinion.
Thanks, though.
- [Moderator] Maybe one last question?
- [Audience Member] Thank you for your talk,
it was interesting to listen to.
I was just wondering if there was any collaboration,
wait, do I have to use the mic?
Oh, okay.
I was wondering if there's any collaboration
between the system when it tells the user,
when it vibrates, that there is text in this picture,
does it at any point consult a language model
to see how likely is this text to be the complete image,
you know, like, that example that we saw,
where it was just seeing half the picture,
does at any point it consult a language model,
and say maybe that's not the whole thing?
- Yeah, that's a very good question.
So would there be an interface to help you frame
the system better, I would say,
probably not, maybe even at the text spotter level.
The quick answer is that it does not have it here,
but that was exactly what I was talking about here.
Let me just mention a quick thing,
in our system developed for document reading,
we had a system in which you would move
the camera over the document, and the system
would tell you move it up, move to the right, move left,
really verbally, because it was able to do
some processing of the data to understand
that if you move to the right,
good probability you're gonna get the whole frame.
We didn't have it here.
It is a little bit more complex
because the degrees of freedom are more.
So you have three, well two-dimensional space
more or less, and three orientations.
So, you had to convey a little bit more information,
but I think that's exactly where we are moving.
Thanks.
- [Moderator] Great, Lester Ingber, again, thank you.
(audience clapping)