Hello, I'm Antonio Torralba, and I
want to talk to you about how do we teach computers to see.
What we want is we want to build systems
that are able to process the world just like humans do.
We want systems that can understand the objects
that they have in front.
For instance, one example of a task that we want to solve
will be something like you have a picture like this one here.
And you want the system to be able to tell you
all the different and objects that compose the scene.
You want to label every single pixel and tell you,
this is the road, this is the sky,
this is a person, a car, and so on.
And there has been an enormous amount of progress
of solving this task.
There are many other visual tasks that you want to solve.
But this is one important example
that is one of the core areas of research in computer vision.
And the way in which you just solve this task,
you use machine learning, like deep learning.
But one of the things that you need
is you need to train your system.
You need to teach the system to solve this task.
And in order to do that, what you do
is you have pairs of examples in where
you have an input and the desired output.
And you have to provide to the system
with lots and lots of examples that look just like that.
And you need hundreds of thousands
of those examples in order for the system
to learn to make that association.
So how do we do that nowadays?
Well, with crowd sourcing.
You hire a bunch of people.
You put them in a room.
And you don't let them out until they
finish labeling all the images.
And that's how did you teach the systems to see.
And they are starting to work pretty well.
There are some really amazing results out there.
But this is kind of unsatisfying, no?
Because this has nothing to do with how we do learn to see.
In fact, the way that we learn to see
is in a much crazier environment.
It doesn't have a lot of structure.
So let's see an example of a kid learning to see.
[VIDEO PLAYBACK]
- In the summer, I like to lie in the sun and watch the birds.
And I like to watch the frogs in the pond.
Then I curl up in my hollow tree and dream about spring.
The End.
- [CRYING]
- Do you want to read another book?
[END PLAYBACK]
So just like machines, kids also need a lot of data
and they don't like it when you stop giving them data.
[LAUGHTER]
But look at this environment.
This is really amazing.
Because the kid needs to learn what
an object is by just looking at these pictures.
And the supervision, how the parent
is teaching the kid to see, it also
comes in a form that is also unknown for the kid.
Because it's in the form of a language
that the kid also doesn't know how to interpret.
So both the thing that he's trying to learn
and the supervision, the supervision
that he's getting both come in an uncoded form.
And he has to decode both simultaneously.
So it's a very challenging task.
And this is one of the things that we are trying to do.
So one of the things that are really interesting when
we tackle problems like general AI and so on
is that it's not enough with the research that
is done in a single lab.
We need to really bring expertise
across different labs.
So what I will show you is I have two glimpses of [? Air ?]
[? Force ?] in which we try to solve this problem.
So the first one is in collaboration
with a speech group.
And it's in the part of the thesis of David,
and also his supervisor, Jim Glass.
And what we are trying to do is to have
a system that can learn both to identify objects
and to understand the words that come with them in a very
similar setting to the one that I just played before.
So what we have will be images like that.
And we just ask people to give verbal descriptions
of what they see.
They don't need to come in any structured form.
Just talk about the picture.
So let's watch one example.
This is a photo of a girl standing
in front of a lighthouse.
The little girl is wearing a blue print dress.
She has blond hair and blue eyes.
The lighthouse in the background is white with a red roof.
So David collected hundreds of thousands
of pairs of images and descriptions
of the content of the picture.
And there were no specific instructions.
You know, people just had to talk about the content
of the picture.
So now the challenge is, can we build
a system that will be able to identify both objects
and images and words on the [? row, ?]
a speech signal without any additional supervision.
So the input will be images with pixels,
and the system really doesn't know
which pixels correspond to an object,
and then the raw audio signal that you hear from the speech.
So again, this is something that doesn't really
tell you where the water starts and where
it ends, where the water is, and what
are the pieces that actually refer
to the content in the picture.
So we want to build a system that
is able to put these things together and find
the correlations.
So I won't go into the details of how it works,
but basically the system doesn't learn
with any additional supervision.
It just takes this input, both these, the set of images
and the associated raw audio signal that goes with it.
And the output of the system is the discovery
of groups of cells that seem to mean something,
they seem to go together, it's an object and pieces
of the speech signal that seem to refer to those.
So I'm just going to show you what
the final output of the system one
is once it has been trained to solve this task so that you can
see how it actually has learned to recognize objects and words.
So what I will do is I'm going to play the audio
and the system will highlight the regions
in the picture that seem to correspond to different pieces
of the speech signal.
This is a photo of a girl standing
in front of a lighthouse.
The little girl is wearing a blue print dress.
She has blond hair and blue eyes.
The lighthouse in the background is white with a red roof.
It isn't perfect, but it's quite amazing
that it's been able to learn this
from that unstructured type of input data.
And it's just wonderful to look at many examples like that.
It's just amazing how it's able to locate
words and objects in pictures.
So let's look at a few more examples.
Here we see a stretch of grass and a small road leading up
to a very old ancient ruin.
Here in this picture, there are some skiers
going up a mountain.
So the system has been able to put together
pixels in the picture, recognize that this
must be some object that appears over and over again,
and it seems to be related to that particular piece
of the speech signal.
So this is a really interesting setting
which now you are able to learn in a way that doesn't require
the same type of manual supervision
with detailed annotations as we had before.
But still, kids get to learn in even crazier environments.
So many times, the kids actually don't even
need the parent with them to learn.
Most of the time, they are just playing by their own,
touching the world, seeing how it feels and so on.
And they also learn without the need of a parent.
And so let's watch a kid learning by its own.
Just playing, touching things, hearing the sounds
that it makes.
And then you put it in the mouth just
to get the final bit of information
that you need just to make it all worth it.
So what is really interesting is that this kid is learning,
is playing with the world, because he's learning about it.
And it's the combination of all the different sensory
modalities that give the supervision that you need
in order to also discover that there are things
in the world that they look in a particular way,
that they feel in a particular way,
and they make a particular set of sounds.
So it is through these interactions
that you also get to learn a lot about the visual world.
So in particular what is interesting
is that this system is actually self-supervised.
It's teaching by itself to discover
what the world is all about.
So, in fact, when you are in the world,
you are always listening to it, you are hearing noises.
And what you hear and what you see
is also correlated, just like when you were,
when we were watching these images with descriptions
of the content of the picture, the world
is also describing itself without no one having to talk
about what the content is.
You are in the street and you hear the noise of cars.
And it's only when you see a car that you
hear the noise of a car.
People talk and they make special noises.
And there are all kinds of objects in the world.
And all of them are making their own type of noises.
So if you think of those just as speech signals,
you can also try to discover what are the sounds that you
hear in the world and how do they correlate
with the visual world.
And by doing that particular correlation between these two
sensory modalities, you also get to discover objects
in the world.
So that's another collaboration also with the [? Brannon ?]
Quantitative Science department with Josh McDermott
and other PIs the computer science laboratory--
and artificial intelligence laboratory.
[VIDEO PLAYBACK]
[STATIC]
So this is the type of videos that we will use to learn.
So you just see some particular setting,
and you hear the ambient noise that goes with it.
And the nice thing is that you can now
have the system just watch all kinds of movies
and learn from them without no one having to actually provide
any supervision.
And there are lots and lots of copyright free movies
out there.
And when you watch them, you understand
why they are copyright free.
So we just have the system to watch all of these movies.
And what is the result?
The result is that the system also
learn to correlate different types of sounds
with what you see.
So here's an example of images.
[STATIC]
And it locates the sound, which pixels in the image
might correspond to that sound.
[CHATTER ON VIDEO]
It doesn't know what people is.
It only knows that people make this sound.
So, which is basically, it's just missing the word "people."
It already knows how to detect them.
[END PLAYBACK]
So building systems that can learn by themselves will open
up the possibility of having machines to learn to solve
tasks that maybe we cannot even provide supervision because we
don't know how to solve those tasks.
So artificial intelligence is going
to require better understanding of how humans
learn to recognize the world.
So thank you.
[APPLAUSE]