>> SCHMIDHUBER: I'm going to talk about now--about the 2nd Neural Network Renaissance which is
currently ongoing and I'll try to show you how you can win all kinds pattern recognition
competitions using our special breed of deep or recurrent neural networks.
AGI is driven by two types of progress, at least.
On the one hand we have this newer millennium results on optimal universal problem solvers
that are theoretically unbeatable and mathematically optimal.
But then we also have another type of progress driven by hardware advances.
Computers are still getting faster by a factor of 100 or 1000 per decade percent, and this
is what we are going to use in this talk.
A long time ago, 20 years ago, my first student [INDISTINCT], he showed that you cannot really
train deep networks with many non-linear layers or a deep--or a recurrent networks which you
can translate into deep [INDISTINCT] networks which have as many layers as there are time
steps in the sequence that you are processing because there's this vanishing gradient, our
problem as you--as you are increasing the number of layers, modeling your layers to
which you are propagating in these networks.
Every time there's a new layer or a new time step, you decrease the error signal by, at
most, a particular constant factor which means that it vanishes exponentially as you are
adding layers and time steps which means that you cannot really propagate far down in traditional
networks like this.
This prompted one of the pioneers of neural networks to say that nobody in his right mind
would take a deep multi-layer perception and train it by [INDISTINCT].
I won't mention this pioneer by name, except to say that he is Geoff Hinton.
So nobody in his right mind, that seems like an accurate description of our neural computer
vision team, here we see Dan, Ueli, and Jonathan and Alex, and so we took a deep, multi-layer
perception with many layers and trained it by [INDISTINCT].
Although my own lab showed that you shouldn't really do that.
However, something has changed because now computers are a million times faster than
they used to be 20 years ago.
And suddenly we can propagate hours down a little bit further than we used to, and now
we can have as many layers in the deep network as are connecting our retinas with the visual
cortex, 789, no one knows exactly, something like that.
And we applied that to the most famous benchmark of machine learning which was developed in
the lab of another pioneer of neural network of [INDISTINCT] in New York.
And there's a long history of broken benchmark records on this problem, and many people use
all kinds of support like the machines and whatever.
However, our approach achieves a new world record just last year.
And then we greatly profit from [INDISTINCT] cards which makes everything 50 times faster
than--on traditional CPUs.
Now you can further improve this result if you take not only one deep network like that,
but you can take seven or 10 and then each of them is slightly differently initialized
and maybe has slightly different pay processing and you get different X cards automatically
as a byproduct.
There's a random byproduct of that and then you do a democratic committee which just takes
the average of all these--of all these predictions of the different networks.
And the result is better than any individual network and this led to a new world record
on the same data set which is now 0.31%.
And what it's really--what you really should do is take not a plain backdrop but something
which is combining all kinds of ideas that range back to Fukushima, the Neil [INDISTINCT],
and max pooling which is a way of down sampling from one layer to the next, and [INDISTINCT]
weight sharing where you have not as many free parameters as you would have in a plain
deep network and then the current world record on that in this problem is 0.27%.
You can also do other images with that.
For example, this is the CIFAR object recognition problem of Geoff Hinton's group in Toronto.
And that we also just, a while ago, broke the record on that one.
And the current one which is not even published yet is now13% on this one here.
There you have to distinguish [INDISTINCT] from cars and horses and boards.
I have a lot of problems with the--with the frogs.
And just awhile ago, Ueli and Don, they--they were instrumental in winning the Chinese handwriting
recognition competition which--where we used exactly the same approach.
Now, that's an interesting problem because unlike with [INDISTINCT] for example, you
have not only ten different patterns but 40--4,000.
And then we got the first and the second ranks.
That was just two months ago.
And if you look in at the details of these--these feature detectors that are being learned automatically
as a byproduct of these deep networks, you see the stuff that you know from vision of
decades ago.
You get automatic--automatically you get garbled filters and on--of surround detectors and
orientations, sensitive body detectors in the lower layers.
And as you're looking at feature detectors in higher layers, they get more and more opaque
and intransparent and less and less clear what's going on there, just like in human
brains.
Now we apply the same thing to the NORB object recognition benchmark and also set a new record.
So--I'm sorry, my slides are going to remain boring like that for a way, but then they
will become boring in a different way.
Yes.
The traffic sign competition that took part in January; that was an online traffic sign
competition and our toughest competitors usually are those from [INDISTINCT] lab.
It's interesting to see that only neural networks are in the--are now playing a part in these
competitions and no support vector machines, no boosting and all these things.
It's a competition mainly between networks and all which is kind of unusual or unexpected
because 10 years ago, neural networks almost vanished from the scene because support vector
machines were so much better in many applications.
And for a long time, it was not possible to publish anything at the leading neural network
conferences that had the word "neural" in the title.
And there, on the last day of the competition where we thought we had a comfortable lead,
then suddenly [INDISTINCT] lab, he had an even better result.
And then Don and Ueli and Jonathan worked late night to re-establish a correct order.
And then the best teams of that time, they were invited, just a couple of days ago, to
the IJCIN Conference where the goal was to take the new data set with more data and then
participate in the final--in the final competition, and down there we see Don with the Number
One Traffic Sign that you can see in his hands.
There the--the differences were unexpectedly high, so PSM [INDISTINCT] this time of New
York University of [INDISTINCT] lab achieved 1.69% correct, then the second best were the
Humans.
Humans are able to get 1.16% correct according to the conference organizers and first place
for our team with 0.56% error.
Now this might be of interest to everybody who is doing self-driving cars and of course,
we all remember the--this incredible keynote of [INDISTINCT] who is not using GPS for self-driving
cars but is doing it in the human-like way.
And then 1995, Audi used these active vision systems to drive from Munich to Denmark and
back in traffic with up to 12 vehicles surrounding the car and tracking them all using these--how
many people saw this talk and how the cameras riddled along there, the active vision camera.
I think this will also be the future of self-driving cars in addition to GPS and radar and [INDISTINCT].
But to do fast driving, you just have to ask [INDISTINCT] you need these--these active
vision techniques for getting high resolution input of stuff that is far ahead.
As you're driving fast, you need to look far ahead to be able to brake in time if there's
an obstacle on the street, and so [INDISTINCT] or something like that is not sufficient.
Now, once you have identified salient points then you still have to do the traffic sign
recognition or a pedestrian recognition or whatever you want to recognize there and there
at the moment, I think I can recommend this technique.
Now, of course, in the long run, we don't just want to recognize patterns that are--single
patterns by few coordinates, no, we have streams of patterns.
Videos essentially where there's no segmentation, where no--nobody segmented the stream into
relevant parts and to do that, we cannot use coordinates [INDISTINCT].
We really have to use recurrent networks which are general computers.
And one of the examples of applications is connected handwriting.
Connected handwriting, there somebody scribbles something and we're supposed to translate
that into plain text which means you have a rather short label sequence.
However, you have a long real value input sequence of pixels that you are sequentially
observing and there you need a recurrent network and preferably a long short-term memory network
that can really deal with these long delays between relevant events.
And Alex [INDISTINCT] our--my former PhD student and then postdoc who is now going to Geoff
Hinton's lab actually, he used this approach in conjunction with technique called Connectionist
Temporal Classification which maximize the probability of the label sequences given these
long real value input sequences without any pre-segmentation and then apply that to various
handwriting recognition competitions, connected handwriting.
And so that's how we won the French Handwriting Connected Handwriting Competition and the
FARCI Connected Handwriting Competition and the Arab Connected Handwriting Recognition
Contest without speaking any word of Arabic.
You may have guessed that my--that the Swiss [INDISTINCT] is in Switzerland.
And in Switzerland, traffic signs are really important.
If you don't obey a traffic sign in Switzerland, you go to jail.
However, across the border, there is Italy.
And in Italy, they also have traffic signs in the street, however only for decoration.
While we are currently trying to write a book about things like that, and I think that's
the end of my talk.
Thank you.