Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
AI research has come a long-long way in the last few years.
I remember that not so long ago, we were lucky if we could train a neural network to understand
traffic signs, and since then, so many things happened: by harnessing the power of learning
algorithms, we are now able to impersonate other people by using a consumer camera, generate
high-quality virtual human faces for people that don’t exist, or pretend to be able
to dance as a pro dancer by using an external video footage and transferring it onto ourselves.
Even though we are progressing at a staggering pace, there is a lot of debate as to which
research direction is the most promising going forward.
Roughly speaking, there are two schools of thought.
One, we recently talked about Richard Sutton’s amazing article by the name, “The Bitter
Lesson”, in which he makes a great argument that AI research should not try to mimic the
way the human brain works - he argues that instead, all we need to do is formulate our
problems in a general manner, so that our learning algorithms may find something that
is potentially much better suited for a problem than our brain is.
I put a link to this video in the description if you’re interested.
And two, a different school of thought says that we should a good look at all these learning
algorithms that use a lot of powerful hardware and can do wondrous things, like playing a
bunch of Atari games at a superhuman level.
Note that they learn orders of magnitude slower than the human brain does, so it should definitely
be worth it to try to study and model the human brain, at least until we can match it
in terms of efficiency.
This school of thought is what we are going to talk about in this video.
As an example, let’s take a look at deep reinforcement learning in the context of playing
computer games.
This technique is a combination of a neural network that processes the visual data that
we see on the screen, and a reinforcement learner that comes up with the gameplay-related
decisions.
Absolutely amazing algorithm, a true breakthrough in AI research.
Very powerful, however, also quite slow.
And by slow, I mean that we can sit for an hour in front of our computer and wonder why
our learner does not work at all, because it loses all of its lives almost immediately.
If we remain patient, we find out that it works, it just learns at a glacial pace.
So, why is this so slow?
Well, two reasons.
Reason number one is that the learning happens through incremental parameter adjustment.
What does that mean?
If a human fails really badly at a task, the human would know that a drastic adjustment
to the strategy is necessary, while the deep reinforcement learner would start applying
tiny, tiny changes to its behavior and test again if things got better.
This takes a while, and as a result, this seems unlikely to have a close relation to
how we, humans think.
The second reason for it being slow is the presence of weak inductive bias.
This means that the learner does not contain any information about the problem we have
at hand, or in other words, has never seen the game we’re playing before and has no
other previous knowledge about games at all.
This is desirable in some cases, because we can reuse one learning algorithm for a variety
of problems.
However, because this way, the AI has to test a stupendously large number of potential hypotheses
about the game, we will have to pay for this convenience by a mighty inefficient algorithm.
But is this really all true?
Does deep reinforcement learning really have to be so slow?
And what on earth does this have to do with our brain?
Well, this paper proposes an interesting counterargument that this is not necessarily true and argues
that with two well thought out changes, the efficiency of deep reinforcement learning
may be drastically improved, and get this, it also tells us that these changes are also
possibly based in neuroscience.
So what are the two changes?
One is using episodic memory, which stores previous experiences to help estimating the
potential value of different actions, and this way, drastic parameter adjustments become
a possibility.
And it not only improves the efficiency, but there is more to it, because there are recent
studies that show that using episodic memory indeed contributes to the learning of real
humans and animals alike.
And two, it is beneficial to let the AI implement its own reinforcement learning algorithm,
a concept often referred to as “learning to learn” or meta reinforcement learning.
This also helps obtaining more general knowledge that can be reused across tasks, further improving
the efficiency of the agent.
Here you see a picture of an fMRI, and some regions are marked with yellow and orange
here.
What could these possibly mean?
Well, hold on to your papers, because these highlight neural structures that implement
a very similar meta reinforcement learning scheme within the human brain.
It turns out that meta reinforcement learning, or this “learning to learn” scheme may
not just be something that speeds up our AI algorithms, but may be a fundamental principle
of the human brain as well.
So these two changes to deep reinforcement learning not only drastically improve its
efficiency, but it also suddenly maps quite a bit better to our brain.
How cool is that?
So, which school of thought are you most fond of?
Should we model the brain, or should we listen to Richard Sutton’s Bitter Lesson?
Let me know in the comments.
Also, make sure to have a look at the paper, I found it to be quite readable, and you really
don’t need to be a neuroscientist to enjoy it and learn quite a few new things.
Make sure to have a look at it in the video description!
Now, I think you noticed that this paper doesn’t contain the usual visual fireworks, and is
more complex than your average Two Minute Papers video, and hence, I expect it to get
significantly fewer views.
That’s not a great business model, but no matter, I made this channel so I can share
with you all these important lessons that I learned during my journey.
This has been a true privilege and I am thrilled that I am still able to talk about all these
amazing papers without worrying too much whether any of these videos will go viral or not.
This has only been possible because of your unwavering support on Patreon.com/TwoMinutePapers.
If you feel like chipping in, just click the Patreon link in the video description.
If you are more like a crypto person, we also support cryptocurrencies like Bitcoin, Ethereum
and Litecoin, the addresses are also available in the description.
Thanks for watching and for your generous support, and I'll see you next time!