Transcriber: Beto Maya Reviewer: Tatjana Jevdjic
Hi! I have two main passions in life:
one is music, I am an enthusiastic saxophone player,
though not a very good one,
and second is the brain, and how it functions.
In the last two years, I'm combining these two, in one project.
In this project, we are trying to teach blind individuals
to see using music and sounds.
But before we'll dive into it –
It's a little bit similar like how dolphins and bats are seeing
using sounds in dark caves or in dark oceans,
but before we'll dive into it,
let's talk a little bit about how vision works.
On the surface, it seems like vision works a little bit like a camera.
And indeed there are many similarities:
the lenses, the shutter in the iris,
and so on, and so forth.
But actually it is a little bit more complicated,
because we see with our brain, not less than we see with our eyes.
Let me give you one example:
I like this example, because it happened to me quite spontaneously.
I used FedEx hundreds of times.
You know, you send a paper, it's been rejected,
you send it again,
and one day, after these hundreds of times,
I suddenly saw an object between the "E" and the "x"
in the FedEx logo.
Now, did they put it there on purpose?
What's going on there?
So actually, try to look now between the "E" and the "x".
How many of you see for the first time in their lives a white arrow there?
Please raise your hand.
So I see at least half of the audience see this arrow for the first time.
What I would like to suggest is that your brain changed now forever!
So, what is going on here?
The FedEx logo did not change.
The image that falls on your eyes did not change.
But the image that you see in your brain -
and this is the image that counts - changed forever.
Now each time you cross the street and see a FedEx truck,
you cannot avoid this ever anymore.
And probably, I changed your way also in this way
that you would always remember me. (Laughter)
Ok, so we see with our brain not less than we see with our eyes.
Now, you have to remember here
that most of the blind individuals, the huge majority,
are blind because of a problem in their eyes, or their visual pathways.
But their brain functions very well.
So what if we found a bypass to deliver the visual information in their brain?
Bypass the problems in their eyes.
Max is demonstrating here.
Max is having a tiny camera,
or not so tiny actually in this case, but it can be tiny –
that looking on this plate of apple,
the image is being scanned by the camera from left to right,
and then being transformed by a computer, a smartphone,
into sounds and music that is being delivered
into an existing sense, into the hearing sense.
What Max's brain needs to learn now, is this conversion algorithm.
Somehow he needs to understand
what is in front of him using the sounds.
And this is very important for Max,
because Max really likes red apples, but he hates green apples.
So his brain really needs to learn this new sensory language.
But never mind about Max's brain,
let's talk about your own brain.
In the next couple of minutes
I will try and teach you how to see with your ears.
But, pay attention, because there's going to be an exam.
You know, I am teaching at the University, you must have an exam.
Ok, let's start with something simple:
(Sound)
The image is scanned from left to right, OK.
And when there is darkness it's quiet,
when there it is an image you would hear a sound.
(Sound)
Now, if the horizontal line is a little bit lower,
(Sound)
you here the line a little bit lower,
(Sound)
and a little bit lower.
(Sound)
So actually now, you hear stairs,
and you can create the image of the stairs in your mind eye.
(Sound)
What about a vertical line?
This is a little bit like pressing all the keyboards of the piano
in the same time, for a very brief time.
So the sound, let's hear it again, it's a little bit dirty:
(Sound)
and very brief in time.
Now, lets add the horizontal line:
(Sound)
So, one is short, one is longer and purer. Let's hear it:
(Sound)
Now, let's make the lines thicker and closer to each other:
(Sound)
and:
(Sound)
And now actually, you are ready to hear your first word,
to read your first word using sound:
(Sound)
OK?
(Sound)
OK, now it's test time. (Laughter)
So, test youself.
This is a word in English.
I will give you one clue:
the first two letters are actually the "Hi" that you saw,
that you saw with your ears a second ago.
And I will play it several times,
and try to create the image in your mind eye.
Don't shout it, I always dreamed to be a conductor,
so when I give you the cue, those of you who managed to get it —
shout it loud. OK?
OK.
(Sound)
OK, second time.
(Sound)
Now, try to think how many letters there are.
(Sound)
Try to hear, if there is an a — don't shout it loud,
try to hear if there is a repeating word.
(Sound)
Ok, and finally try the actual sentence.
(Sound)
OK, how many of you think they got it? Raise your hands.
Fantastic! So when I give the cue, shout it up!
(Audience): Hill.
Fantastic audience! Ok, let's hear it.
(Sound)
Ok? Now, let's spice it up. (Applause)
Thank you very much.
Now, let's spice it up and add a little core here.
(Sound)
OK, this starts to be more groovy.
Now but, is it just reading letters, because this is only one visual ability.
How about if I want to draw a hill?
(Sound)
This is how we can draw the hill with music.
Now let's add a little sunshine:
(Sound)
And maybe a little lake in the bottom:
(Sound)
OK, so if you managed to do it, well done!
And if you didn't manage to get it, don't be hard on yourself –
it takes much more than a couple of minutes to really master it,
and this is what we are doing in our team.
So, the key here, is the algorithm.
The translation key that keeps the visual information
inside the sounds and the music.
This is one aspect.
The second aspect is the training program:
How you train your brain to learn this procedure.
And all this is being done by these fantastic people in the team,
in addition to scanning the brain and trying to understand it better.
So this is what we will see in the next couple of slides.
(Sound) So what can blind do after certain amount of training?
After one hour, for example, they learn very simple shapes,
(Tapping)
and reach for the blue square.
I guess I'm blind now.
After 10 hours – you grab your shoes.
(Video): Wow. (Laugther)
And then, after seventy hours,
they can start to learn the faces of people,
and extract relevant information from them.
(Whistling)
And I really like this movie because it shows all the steps.
(Video) Woman: Ok, very good. Man: And you can hear...
the two nostrils. W: How do you hear the two nostrils?
M: Like two clicks.
Amir Amedi: This is Shlomo, a congenitally blind individual.
Shlomo: Wow, I think I hear the eyes and the nose. Woman: Good!
W: And can you maybe hear anything else?
Shlomo: Wait a second...
I think there are also eyebrows.
Woman: Excellent! Great job!
Try now, please.
Shlomo: Here, the sad one is second. Woman: Very good! Great job!
Amir Amedi: OK, this is not bad, but now after each lesson
we want to move to the real world.
Now, a blind, in this case Hamutal, also blind from birth,
will want to enter a room, look at someone
and do exactly the same.
And again, she's looking with the camera,
and hearing these sounds and music in her ears.
(Video) (Sounds)
Man: What do you hear now?
Hamutal: Something round... That means she is surprised.
Woman: Excellent! Now she's surprised. And now?
(Sounds)
Hamutal: She's smiling.
Woman: Excellent! And now to a different face.
(Sounds) (Indistinct talking)
Hamutal: I think she is angry. Woman: Excellnet! Amazing!
Amir Amedi: OK. (Applause) Thank you.
Now, what happens in the brain when the blind learn
all these amazing abilities?
According to the classical view of how the brain is organized –
the brain is a sensory machine.
We have, for example, the red areas:
the visual system in the back of our brain;
we have the blue areas: on the side of our brain for hearing;
we have the green areas: on the top of our brain for touch;
and so on and so forth.
If we add up all these different areas,
we come up to a huge numbers –
there [are] actually billions and billions of neurons.
In fact, it's similar to the number of stars in our Milky Way Galaxy.
Of these huge number of neurons about quarter is dedicated to vision.
So, what happend in the blind –
What happened with these neurons, with all these billions of neurons?
Are they dead, unemployed? Sit at home and wait?
Or, have they changed function?
The answer is actually quite surprising:
what you see here is a map of activation from a blind individual
that is doing this,
it's listening to sounds and creates their images.
And, as you can see here, and it can be
looking at an apple, reading the "hill", etc, etc.
So, what is happening here, is that quite naturally
when they listen to the sounds,
the blue parts in the brain in the side of the brain – light up.
But also, we have huge activation in the back of the brain,
in the visual system,
which is not less than the activation in their auditory cortex,
in some cases even more.
And the picture is even more complex, there is an order here:
first, the hearing system lights up,
and then the information travels back to the visual system. OK?
We can see it in the next movie –
Pay attention to the activation, which is highlighted in yellow.
The activation flows back into the visual system.
(Sound)
Hearing system –
(Sound) from the back – visual system.
And actually the picture is even more specific
and more accurate if you are listening to the apple:
the information will travel to the visual object recognition center
and pick there the LOC area.
And quite interestingly, in some late blind,
they report that when they hear this apple,
and they build this image in their mind,
the actual visual experience pop in their mind.
This is very surprising and actually under investigation now.
Now, if instead of looking at the apple,
they would read the word "hill",
the information won't go to the LOC,
it will travel lower to another visual area, the visual word-form area
which is activated by reading in all cultures,
and each and everyone of you actually,
in any reading system, but also in this new reading system.
So, maybe we shouldn't think about these areas as visual entities.
This is actually an illusion:
they look visual, and it looks like a sensory brain
because vision is so good [at] this.
But even in blind individuals, that where born blind,
that didn't see even one photon of light in their lives,
we get exactly the same activation there.
So maybe the brain is a task machine, rather than a sensory machine.
It's a new way to look at brain organization.
So, we saw that we can teach blind to see with their ears
in a cheap, relatively easy, and noninvasive –
doesn't require any surgery.
I also suggest that we can use this approach
to awaken the visual system and the visual brain to see again,
after invasive surgeries,
for example, like the bionic eye surgeries
that is under development now.
Finally, I want to highlight the fact
that this type of work, and this type of projects –
You can imagine how it can have implication into sensory augmentation.
Let me give you one quick example.
And in sensory augmentation, I'm talking about adding a new ability
to the abilities that we already have.
Now, if you're guys, or even if you are girls,
imagine for a second you are James Bond:
you are walking in this corridor,
you have a little heat sensor that creates a thermal heat map
from the sensor, and this, since you are James Bond,
is being delivered to your ears only this time,
and you learn this to code this heat map
just like we did with the blind.
James Bond can now walk in this corridor,
see the environment with his eyes, and also in one of his ears,
see beyond the walls the thermal image,
a technology we already have available.
So now, the sky or your imagination is the limit, and I invite you
to start thinking about new ways to create this super human abilities.
Now, thank you very much for lending me your eyes, your ears, and your brains.
(Applause)
Thank you, thank you.