Translator: TEDx Milano Translators Reviewer: Michele Gianella
When my children's friends ask me, what's your job?
I reply: "I make little eyes for robots".
To be precise, my job is inventing the algorithms
which are the part of a robot's brain
in control of the eyes.
Sight is very important for us:
we explore the world with vision, with our ability to perceive it.
Consider that 30 percent of our neurons
are dedicated to vision.
Only 4 percent to touch.
Also consider that there is a part of our brain
which is expressly dedicated to face recognition.
And that the first face a newborn baby learns,
one of the very first things that a newborn learns,
it to recognise its mother's face.
Because it depends on the mother for protection, for food.
And therefore, our ability to perceive is essential to understand the world,
and that's why we have to try
to give robots these same perceptual abilities,
if we want them to really become a part of our life,
and help us.
But seeing is more than perceiving.
Let's do a demo.
I need your help.
Raise your right hand if you think this is a cup.
Whoever thinks it's a cup, keep your right hand up.
And now, instead,
whoever thinks this is a pen holder raise your left hand.
You can all keep your hands up.
You have all surrendered.
(Laughter)
What did I do?
What happened there?
When I showed you this object,
you recognised it as a cup
because over, and over, and over again you drank coffee, tea,
or whatever your favourite hot liquid is
from an object like this.
Hot liquid, because I'm holding this object by the handle,
because cups have handles.
Because we want to use them, they are containers we want to use
to put some hot liquid inside.
So the hand must stay away, I don't want to burn myself.
But it's also a container.
And sometimes, I'm sure,
you've seen it used, or used it, like a pen holder.
Because it's a container, I can put things in it.
I can even put water in it.
I can even put a seedling in it, why not.
So, what am I saying here?
That the perception of this object doesn't tell me everything
about what the essence of this object is.
We understand what we see
not just based on what we perceive,
but we also understand it based on other modalities.
On what we do with this object.
And then, what happened in our brains,
when I showed you this cup?
What happened is that we have models,
that can be defined as "meta-modal", formed of many modalities,
where the perception is what makes us click,
it's what we hook on to.
But then there's all this other information
which depends a lot on all the things we can do with these objects.
On all the actions we've done with these objects,
and that's how we store it in our memory.
That's how you understand what you see.
And this is the type of meta-modal representation
that we need to make robots understand the world,
to show them how we see.
But it's still not enough.
We have to take one more step, we have to go further.
I've got a story for you.
This is a photo from my family album
from an Easter many, many years ago.
I'm the girl worshipping the Easter egg,
because I -
I liked chocolate, I still like it - that's how it is.
But I was also a disciplined child,
so I waited until the end of lunch
to scoff the egg.
And it didn't end there.
Because besides being very greedy, I've also always been very vain.
I liked, and I still like, clothes.
And this paper was beautiful!
All shiny and colourful.
And so, after eating the egg, I took this paper,
I wrapped it around myself and showed up this way in the living room.
I had never seen anyone use the Easter egg paper like a dress.
Nobody had ever told me
that the paper from an Easter egg could be made into a dress.
But I had seen so many fabrics.
I had understood, I had seen that clothes are made of fabric.
I had seen my mum at the sea in a sarong, which looks a lot like this object,
taking it and making it into a dress.
And I recognised that this beautiful coloured paper
had these same properties of flexibility.
So what did I do?
I imagined
that using the paper from an Easter egg
you could also do what you do with a sarong.
And I did, and I wore it.
This is the point, this is the main thing.
Besides being able to understand the world
with representations that go beyond how we perceive it,
with meta-modal representations,
we are able to imagine, we are able to go further.
When we see something,
we are able to not only think about what it is,
to understand and grasp what it is,
but also to imagine what it could be.
This is one of the great triumphs of human intelligence,
and is still today one of A.I.'s biggest limits.
I'm sure you've heard about robots doing concerts.
These robots perform perfectly, exactly,
the actions and gestures
they were taught by their teacher.
If this teacher is a great musician,
the concert will be wonderful.
But it will be exactly nothing but that single and unique interpretation.
Compare this with what human beings are capable of.
Think of a jazz orchestra made up of musicians who meet,
play together for the first time
and bring their talents together
and create in the moment, through improvisation,
a wonderful music, that no one ever heard before.
This ability to imagine, this ability to go further.
This is one of the great challenges which artificial intelligence,
one the research in this field is trying to tackle now.
It's not easy: we have to move
from an analogue world to a digital world.
When I work on images,
the images are made of pixels, so everything is discrete.
And what we have to do
is write programmes, and write algorithms
that manage to capture, at least in part, what I've told you about.
To do this we have a tool, for a couple of years now -
five years, to be precise: Deep Learning.
A bundle of algorithms
that try to simulate the behaviour of the human brain.
We implement functions that mimic neurons,
and also mimic synapses, the connections between neurons.
And these algorithms are called large capacity algorithms,
because they're algorithms that can handle millions and millions of bits of data.
In my case, images.
These images are the "food",
the "fuel" if you like, that we put in these algorithms,
which produce - they are data-squeezers, so to say.
We feed them with data,
and they can distinguish the noise from the signal.
Which is one of the essential details.
And since there are many, many images on the web,
there are lots of data,
trying to build meta-modal representations,
especially in cases with no concrete examples,
in my research group my vandals and I told each other:
"But why don't we try teaching the robots?
Teaching them to learn from the web?"
After all, when we don't know something,
what do we do now?
We get out our phones, and search the web.
Why not the same for the robots?
This is an example of what we do.
Here you can see the robotic arm.
We asked this robot to clear a table
after we had breakfast.
In the robot's memory there were various objects:
there was a cup, a banana.
But there was no jar of honey.
And so the robot perceived that there was something on the table,
but it couldn't say what it was.
And so it went to search on the web.
What did it do?
It took a photo,
and with this photo it found many other similar images,
many other similar perceptions.
And after a while, it started to notice
that these similar perceptions it found on the web
were very often associated with one phrase: jar of honey.
And that's how it learned the name of that object.
And with the name of that object
it could search for other information too.
It understood that it's an object
that's composed of parts, because it has a lid,
and therefore, when a robot has to go and get it,
it's not a good idea to pick it up by the lid,
as if it's not screwed on,
I'll take the lid off and the jar will stay there.
It learned there's a certain weight that can be expected,
so the strength that the robotic arm must use, to pull up the jar,
will be a certain type.
It learned that it's a rigid object, it's not like a milk box.
So when it presses, when the hand closes,
the tactile control must be properly modulated.
Etc, etc.
And I'll show you that this information,
which we find on the web in the form of text, so as a symbol,
we learn it through actions, from our experience;
and we are able to - click! take it from our experience,
and make it symbolic and accessible to everyone.
And so, after learning all these things,
the robot discovered
that this mysterious object is a jar of honey
and it was able to pick it up and continue clearing the table.
We have built
a meta-modal representation of a jar of honey,
using the web as a starting point.
And what do we do with our imagination?
With the ability to imagine?
What we imagined is -
why don't we give the robot hallucinations?
In a legal way, sure. Only legal.
This thing you see here is a hallucinated apple.
We took from the web some pictures of apples,
and we taught the robot
to imagine what this apple would be like in three dimensions,
because the information I find on the web is all two-dimensional images;
but I need to understand the depth, the shape of the object.
And therefore, starting from this image we managed to hallucinate the form.
This is a small result,
but for us it's actually a very important step.
Let me give you another example.
At the moment I see - what I can, with these lights,
I can see all your faces.
I can't see the backs of your heads,
but I've seen people many, many times,
and so I'm assuming
that the back of everyone's head is spherical.
Maybe there, where I can't see,
there's an alien from Star Wars with a beautiful curly head,
and so there I'm wrong.
However, if we're all human beings here, our heads are round.
We do this all the time.
When we perceive things in an incomplete manner,
so as to pursue, in any case, the goal we must have,
to have a model of the world,
we complete it with whatever is most likely.
But this ability to complete is the ability to imagine,
because usually I go with what's most probable.
But designers,
the guys behind the characters for Star Wars,
look for the improbable.
So, if we've found the way to have a probable apple,
maybe, in the future,
we'll also find the way to get a Star Wars apple.
Putting all these pieces together,
we managed to get an R1 which learns about flowers from the web.
These are all objects
the robot has already learned to recognise,
and these bars you see there, that rise and fall,
measure the confidence the robot has in what it sees.
In naming what it sees.
Now it detects the oven mitt.
And, you can see - sometimes it hesitates.
And now we show it the flower.
- [English]: What is this? - I think this is a cellphone.
Mmm.. no.
- Learn what is a flower.
So it needs to learn what a flower is.
- OK, let me search the flower images on the web.
And so it looks for images on the web similar to what it saw.
And it finds lots of them.
It also has the name "flower", which can help it.
And what we do to help the robot
contextualise this knowledge into its experience of the world,
whenever it needs to use this model,
is that we take all these flowers, we take off the photo, automatically,
online, everything but flowers,
and we put it in a scenario
which is very similar to the one the robot sees around it,
to help it recognise the flowers it can see around it.
- [English]: OK, now I know the flower.
Here's the flower model.
- What is this? - I think this is a flower.
And so it has learnt.
We are still a long, long way from being able to say
that we can recognise any object,
but we're making small steps in this direction.
When these little steps take us all the way to the end of the journey,
we will have robots which,
at our children's birthday parties,
will take the lights off the Christmas tree
and try to dress up for the party.
You're all invited.
Thank you.
(Applause)