Vision is so natural for humans.
You just open your eyes and you understand.
Vision is a very, very complicated process, but we don't register
most of it consciously.
And so it's very hard for us to realize why is it such a hard
problem for the computer.
I try to understand how we can make computers
see and understand our visual world.
For example, if I am looking at this redwood tree,
I am not just looking at this tree.
I'm looking at it through the prism
of what I have seen before.
So how do we train computers to see?
A lot of the current.
Computer vision systems are
very much about translating the visual world into a
few keywords.
Which of course is quite limiting.
Because a picture is worth a thousand words.
The field of computer vision
today is undergoing a big shift.
We were working really hard for many decades.
And we had basically nothing to show for it.
The problem turned out to be much
deeper and much more exciting than anyone imagined.
When people were starting to look at
artificial intelligence
in the early sixties at MIT, they realized that vision
should be an integral part of the story of a robot.
The robot has some cameras and the robot is going to sense its environment
and do actions.
The vision part of this grand goal of an AI robot was.
Considered to be a summer project.
The idea was that if
you write a smart
enough algorithm,
you will teach the computer how to see.
Only much later did we
realize that.
what the computer needs is to connect
what it's sensing with what it has seen in the past.
One of my superpowers that helped me early
on was that my vision is not great.
A lot of people I interact with every day don't know that I have poor vision.
I am able
to compensate with good visual memory.
I just remember a lot of things instead of seeing them.
In a sense, my brain fills in a lot of the missing details
and that was very important for me early
on to realize the huge importance of large-scale visual data
playing in this enterprise of seeing and understanding.
That we're not seeing
just with our eyes, we are seeing with our eyes and our memory.
The pattern
of bark on the redwood tree, for example.
The way it looks is defined by so many different things:
The genome of the tree,
the weather. And it's impossible to
basically write down the precise formula which would represent it.
But it is possible to capture
a lot of examples of different trees and different patterns of
bark and that we can use to model
what this bark might look like.
Data is
absolutely fundamental to machine learning in general,
and computer vision in particular.
Everybody is always excited about algorithms
and actually, it's the data that's doing a lot of the heavy lifting.
So one way to think about what my lab is doing is to give data
the appreciation it deserves.
In our lab,
we do many different things
connected with visual data:
scene understanding, image generation
and editing... Basically, modeling visual world
by creating or modifying it.
Our work is actually starting to be in the real world.
Self-driving cars, of course, software
like Adobe Photoshop,
what we call computational photography.
There is a lot of computer vision that happens with the latest smartphones from Google and Apple.
So there are two basic
paradigms in contemporary computer vision.
The older and still most prominent.
paradigm is supervised learning.
It's basically you get a huge amount of images,
for example, or a video,
and then for every image, some human is going to go and annotate
what is on that image.
That supervision is often something that can introduce more bias.
It comes from the labels.
And now
you have millions of these labeled images.
The neural network
is going to try to learn the association
between a particular image and the corresponding
label, and some of those categories are not very meaningful.
What we do in our lab is
what we call self-supervised learning.
We try to build models that do not
require this linguistic supervision -- they can understand
the world from the raw data itself,
much like animals are able to understand the world.
So, for example, we can have an image
and then make some holes in that image and then have the computer fill in
those holes.
Or we can try to predict in a video what's going to happen next.
You get rid of the labels and so you can get rid of a lot of the bias that is
baked into those labels.
One issue that I have been
thinking for a while:
We, biological agents, are experiencing the world in a continuous fashion.
We're going through life continuously
adapting in this kind of never-ending learning loop.
For example, you fly
into an airport that you have never been before,
and yet
you're able to navigate in that airport, even if it maybe looks very different
from the other airports you're used to.
For computers, this problem is more extreme because computers are less able
to generalize. In most machine learning algorithms,
we have a fixed data set.
It might be a very big dataset, you know, billions of images.
We let the computer train on all the data in this dataset.
And then that model is frozen. It's fixed.
We go out into the real world and we just hope
somehow we are able to perform well on this data.
One of the projects that
we've been doing in our lab is what we call test-time training,
which is the idea that every time we are faced with a new
piece of data -- a new image, for example,
we use that image to adopt our model.
It's always updating.
When the environment changes the model
hopefully will change with it.
So, for example, for self-driving cars,
a lot of the self-driving cars are based in
California, where it rarely snows, almost never.
And then they go to Minnesota.
There needs to be a mechanism
for the car to somehow adapt.
Right now in our field, we are finding things out all the time.
The latest text-generative models like Chat GPT have been very exciting for me.
I really should not have
been surprised,
but I still was very surprised
how good those algorithms were
once you give them enough data.
Huge, huge amounts of data
give you these almost magical abilities to generalize, to do analogies.
A lot of my students have started to get interested in
robotics and the connection between robotics and computer vision.
They are talking about this interaction between the data and the algorithm
and trying to figure out what is happening on the inside.
It's kind of like doing neuroscience on the computer.
Hopefully that might even give us some insights
into how biological agents see the world.
I think my proudest
achievement has
veen producing the next generation.
I have had absolutely wonderful graduate students and post-docs.
I feel like this is really my biggest contribution to science.