Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Before we start, I’d like to tell you that this video is not about a paper, and it is
not going to be two minutes.
Welcome to Two Minute Papers!
This piece bears the name “The Bitter Lesson” and was written by Richard Sutton, a legendary
Canadian researcher who has contributed a great deal to reinforcement learning research.
And what a piece this is!
It is a short article on how we should do research, and ever since I read it, I couldn’t
stop thinking about it, and as a result, I couldn’t not make a video on this topic.
We really have to talk about this.
It takes less than 5 minutes to read, so before we talk about it, you can pause the video
and click the link to it in the video description.
So, in this article, he makes two important observations.
Number one, he argues that the best performing learning techniques are the ones that can
leverage computation, or, in other words, methods that improve significantly as we add
more compute power.
Long ago, people tried to encode lots of human knowledge of strategies in their Go AI, but
did not have enough compute power to make a truly great algorithm.
And now we have AlphaGo, which contains minimal information about Go itself, and it is better
than the best human players in the world.
And, number two, he recommends that we try to put as few constraints on a problem as
possible.
He argues that we shouldn't try to rebuild the mind, but try to build a method that can
capture arbitrary complexity and scale it up with hardware.
Don’t try to make it work like your brain - make something as general as possible and
make sure it can leverage computation and it will come up with something that is way
better than our brain.
So, in short, keep the problem general, and don’t encode your knowledge of the domain
into your learning algorithms.
The weight of this sentence is not to be underestimated, because these seemingly simple observations
sound really counterintuitive.
This seemingly encourages us to do the exact opposite of what we are currently doing.
Let me tell you why.
I have fond memories of my early lectures I attended to in cryptography, where we had
a look at ciphertexts.
These are very much like encrypted messages that children like to write each other at
school, which looks like nonsense for the unassuming teacher, but can be easily decoded
by another child when provided with a key.
This key describes which symbol corresponds to which letter.
Let’s assume that one symbol means one letter, but if we don’t have any additional knowledge,
this is still not an easy problem to crack.
But in this course, soon, we coded up algorithms that were able to crack messages like this
in less than a second.
How exactly?
Well, by inserting additional knowledge into the system.
For instance, we know the relative frequency of each letter in every language.
For instance, in English, the letter “e” is the most common by far, and then comes
“t”, “a”, and the others.
The fact that we are not seeing letters but symbols doesn’t really matter, because we
just look up the most frequent symbol in the ciphertext and we immediately know that okay,
that symbol is going to be the letter “e”, and so on.
See what we have done here?
Just by inserting a tiny bit of knowledge, suddenly, a very difficult problem turned
into a trivial problem.
So much so that anyone can implement this after their second cryptography lecture.
And somehow, Richard Sutton argues that we shouldn’t do that?
Doesn’t that sound crazy?
So, what gives?
Well, let me explain through an example from light transport research that demonstrates
his point.
Path tracing is one of the first and simplest algorithms in the field, which, in many regards,
is vastly inferior to Metropolis Light Transport, which is a much smarter algorithm.
However, with our current powerful graphics cards, we can compute so many more rays with
path tracing, that in many cases it wins over Metropolis.
In this case, compute reigns supreme.
The hardware scaling outmuscles the smarts, and we haven’t even talked about how much
easier it is for engineers to maintain and improve a simpler system.
The area of Natural Language Processing has many decades of research to teach machines
how to understand, simplify, correct, or even generate text.
After so many papers and handcrafted techniques which insert our knowledge of linguistics
into our techniques, who would have thought that OpenAI would be able to come up with
a relatively simple neural network with so little prior knowledge that is able to write
articles that sound remarkably lifelike.
We will talk about this method in more detail in this series soon.
And here comes the bitter lesson.
Doing research the classical way of inserting knowledge into a solution is very satisfying
- it feels right, it feels like doing research, progressing, and it makes it easy to show
in a new paper what exactly the key contributions are.
However, it may not be the most effective way forward.
Quoting the article.
I recommend that you pay close attention to this.
“The bitter lesson is based on the historical observations that 1) AI researchers have often
tried to build knowledge into their agents, 2) this always helps in the short term, and
is personally satisfying to the researcher, but 3) in the long run it plateaus and even
inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing
approach based on scaling computation by search and learning.
The eventual success is tinged with bitterness, and often incompletely digested, because its
success over a favored, human-centric approach.“
In our cryptography problem from earlier, of course the letter frequency solution and
other linguistic tricks are clearly much, much better than a solution that doesn’t
know anything about the domain.
Of course!
However, when later, we have a 100 times faster hardware, this knowledge may actually inhibit
finding a solution that is way, way better.
This is why he also claims that we shouldn’t try to build intelligence by modeling our
brain in a computer simulation.
It’s not that the “our brain” approach doesn’t work - it does, but on the short
run.
On the long run, we will be able to add more hardware to a learning algorithm, and it will
find more effective structures to solve problems, and it will eventually outmuscle our handcrafted
techniques.
In short, this is the lesson: when facing a learning problem, keep your domain knowledge
out of the solution, and use more compute.
More compute gives us more learning, and more general formulations give us more chance to
find something relevant.
So, this is indeed a harsh lesson.
This piece sparked great debates on Twitter, I have seen great points for and against this
sentiment.
What do you think?
Let me know in the comments, as everything in science, this piece should be subject to
debate and criticism.
And therefore, I’d love to read as many people’s take on it as possible.
And, this piece has implications on my thinking as well — please allow me to add a three
more small personal notes that kept me up at night in the last few days.
Note number one, the bottom line is whenever we build a new algorithm, we should always
bear in mind which parts would be truly useful if we had a 100 times the compute power that
we have now.
Note number two, a corollary of this thinking is that arguably, hardware engineers who make
these new and more powerful graphics cards may be contributing the very least as much
to AI than most of AI research does.
And note number three, to me it feels like this almost implies that best is to join the
big guys where all the best hardware is.
I work in an amazing, small to mid-sized lab at the Technical University of Vienna and
in the last few years, I have given relatively little consideration to the invitations from
some of the more coveted and well-funded labs.
Was it a mistake?
Should I change that?
I really don’t know for sure.
If for some reason, you haven’t read the piece at the start of the video, make sure
to do it after watching this.
It’s really worth it.
In the meantime, interestingly, the non-profit AI research lab, OpenAI also established a
for profit, or what they like to call “capped profit” company to be able to compete with
the other big guys like DeepMind and Facebook Reality Labs.
I think Richard has a solid point here.
Thanks for watching and for your generous support, and I'll see you next time!