Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Oh my goodness.
This work is history in the making.
Today we are going to have a look at AlphaFold, perhaps one of the most important papers of
the last few years.
And you will see that nothing that came before even comes close to it, and that it truly
is a gift to humanity.
So, what is AlphaFold?
AlphaFold is an AI that is capable of solving protein structure prediction, which we will
refer to as protein folding.
Okay…but what is a protein and why does it need folding?
A protein is a string of amino acids, these are the building blocks of life.
This is what goes in, which in reality, has a 3D structure.
And that is protein folding.
Letters go in, a 3D object comes out.
This is hard.
How hard exactly?
Well, let’s compare it to DeepMind’s amazing previous projects, and we’ll see that none
of these projects even come close in difficulty.
For instance, DeepMind’s previous AI learned to play chess.
Now, why does this matter, as we already have Deep Blue, which is a chess computer that
can play at the very least as well as Kasparov did, and it was built in 1995.
So, why is chess interesting?
The space of possible moves is huge.
And Deep Blue in 1995 was not an AI in the strictest sense, but a handcrafted technique.
This means that it can play chess and that’s it.
One algorithm, one game.
If you want a different game, you write a different algorithm.
And, yes, that is the key difference.
DeepMind’s Chess AI is a general learning algorithm that can learn many games, for instance,
Japanese chess, Shogi too.
One algorithm, many games.
And, yes, chess is hard.
But these days, the AI can manage.
Then, Go is the next level.
This is not just hard, it is really hard.
The space of possible moves is significantly bigger, and we can’t just evaluate all the
long-term effects of our moves, it is even more hopeless than chess, and that’s often
why people say that this game requires some sort of intuition to play.
But DeepMind’s AI solved that too and beat the world champion Go player 4 to 1 in a huge
media event.
The AI can still manage.
Now, get this, if Chess is hard, and Go is very hard, then protein folding is sinfully
difficult.
Once again, a string of text encoding the amino acids go in, a 3D structure comes out.
Why is this hard?
Why not just try every possible 3D structure and see what sticks?
Well, not quite.
The search space for this problem is still stupendously large, perhaps not as big as
playing a continuous strategy game like Starcraft 2, but the search here is much less forgiving.
Also, we don’t have access to a perfect scoring function, so it is very difficult
to define what exactly should be learned.
In a strategy game, a win is a win, but for proteins, nature doesn’t really tell us
what it is up to when creating these structures.
Thus, DeepMind did very well in Chess and Go, and Starcraft too, and challenging as
they are, they are not even close to being as challenging as protein folding.
Not even close.
To demonstrate that, look, this is CASP.
I've heard DeepMind CEO Demis Hassabis call it the Olympics of protein folding.
If you look at how teams of scientists prepare for this event, you will probably agree that
yes, this is indeed the Olympics of protein folding.
At about a score of 90, we can think of protein folding as a mostly solved problem.
No need to worry about definitions though, look, we are not even close to 90.
And it gets even worse, look.
This GDT score means the global distance test, this is a measure of similarity between the
predicted and the real protein structure.
And, wait a second.
What?
The results are not only not too good, but they appear to get worse over time.
Is that true?
What is going on here?
Well, there is an explanation.
The competition gets a little harder over time, so even flat results mean that there
is a little improvement over time.
And now, hold on to your papers, and let’s look at the results from DeepMind’s AI-based
solution, AlphaFold.
Wow, now we’re talking.
Look at that!
The competition gets harder, and it is not only flat, but can that be?
It is even better than the previous methods.
But, we are not done here, no-no, not even close.
If you have been holding on to your papers, now, squeeze that paper, because what you
see here is old news.
Because only two years later Alphafold 2 appeared.
And just look at that.
It came in guns blazing.
So much so that the result is…I can’t believe it!
It is around the 90 mark.
My goodness, that is history in the making.
Yes, this is the place on the internet where we get unreasonably excited by a large blue
bar.
Welcome to Two Minute Papers!
But what does this really mean?
Well, in absolute terms, AlphaFold 2 is considered to be about three times better than previous
solutions.
And all that in just two years.
That is a miracle right in front of our eyes.
Now, let’s pop the hood, and see what is inside this AI, and...hmm!
Look at all these elements in the system that make this happen.
So, where do we start?
Which of these is the most important?
What is the key?
Well, everything…and nothing.
I will explain this in a moment.
That does not sound very enlightening, so, what is going on?
DeepMind ran a detailed ablation study on what mattered and the result is the following:
everything mattered.
Look.
With few exceptions, every part adds its own little piece to the final result, but, none
of these techniques are a silver bullet.
But to understand a bit more about what is going on here, let’s look at three things.
One, AlphaFold 2 is an end to end network that can perform iterative refinement.
What do these mean?
What this means is that everything needed to solve the task is learned by the network,
and that it starts out from a rough initial guess, and then, it gradually improves it.
You see this process here, and it truly is a sight to behold.
Two, it uses an attention-based model.
What does that mean?
Well, look!
This is a convolutional neural network.
This is wired in a way that information flows to neighboring neurons.
This is great for image recognition, because usually, the required information is located
nearby.
For instance, let’s imagine that we wish to train a neural network that can recognize
a dog.
What do we need to look at?
Floppy ears, black snout, fur, okay, we’re good, we can conclude that we have a dog here.
Now, have you noticed?
Yes, all of this information is located nearby.
Therefore a convolutional neural network is expected to do really well at that.
However, check this out.
This is a transformer, which is an attention-based model.
Here, information does not flow between neighbors.
No sir!
Here, information flows everywhere!
This has spontaneous connections that are great for almost anything if we can use them
well.
For instance, when reading a book, if we are at page 100, we might need to recall some
information from page 1.
Transformers are excellent for tasks like that.
They are still quite new, just a few years old, and are already making breakthroughs.
So, why use them for protein folding?
Well, things that are 200 amino acids apart in the text description can still be next
to each other in the 3-D space.
Yes, now we know that for that, we need attention networks, for instance, a transformer.
These are seeing a great deal of use these days, for instance, Tesla also uses them for
training their self-driving cars.
Yes, so these things mattered.
But so many other things did too.
Now, I mentioned that the key is everything…and nothing.
What does that mean?
Well, look here.
Apart from a couple examples, there is no silver bullet here.
Every single one of these improvements bump the score a little bit.
But all of them are needed for the breakthrough.
Now, one of the important elements is also adding physics knowledge.
How do you do that?
Typically the answer is that you don’t.
When we design a handcrafted technique, we write the knowledge into an algorithm by hand.
For instance, in chess, there are a bunch of well-known openings for the algorithm to
consider.
For protein folding, we can tell the algorithm that if you see this structure, it typically
bends this way.
Or we can also show it common protein templates, kind of like openings for chess.
We can add all this valuable expertise to a handcrafted technique.
Now, we noted that scientists at DeepMind decided to use an end to end learning system.
I would like to unpack that for a moment, because this design decision is not trivial
at all.
In fact, in a moment, I bet you will think it’s flat out counterintuitive.
Let me explain.
If we are a physics simulation researcher and we have a physics simulation problem,
we take our physics knowledge, and write a computer program to make use of that knowledge.
For instance, here, you see this being used to great effect, so much so that what you
see here is not reality, but a physics simulation.
All handcrafted.
Clearly, using this concept, we can see that human ingenuity goes very far, and we can
write super powerful programs.
Or, we can do end to end learning, where, surprisingly, we don’t write our knowledge
into the algorithm at all.
We give it training data instead, and let the AI build up its own knowledge base from
that.
And AlphaFold is an end to end learning project, so almost everything is learned.
Almost.
And, one of the great challenges of this project was to infuse the AI with physics knowledge
without impacting the learning.
That is super hard.
So, training huh?
How long does this take?
Well, get this, DeepMind can train this incredible folding AI in as little as 2 weeks.
Why is two weeks little?
Well, after this step is done, the AI can be given a new input and will be able to create
this 3D structure in about a minute.
And, we can then reuse this trained neural network for as long as we wish.
Whew, so… this is a lot of trouble to fold these proteins.
So what is all this good for?
The list of applications is very impressive, I’ll give you just a small subset of them
that I really liked: it helps with us better understand the human body, or create better
medicine against malaria and many other diseases, develop more healthy food, or develop enzymes
to break down plastic waste, and more.
And that’s just the start.
Well, you are probably asking, Károly, you keep saying that this is a gift to humanity.
So…why is it a gift to humanity?
Well, here comes the best part.
A little after publishing the paper, DeepMind made these 3D structure predictions available
for free for everyone.
For instance, they have made their human protein predictions public.
Beyond that, they already have made their predictions public for yeast, important pathogens,
crop species, and more.
And I have already seen followup works on how to use this for developing new drugs.
What a time to be alive!
Now note that this is but one step in a thousand-step journey.
But one important step nonetheless.
And, I would like to send huge congratulations to DeepMind.
Something like this costs a ton to develop, and note that it is not easy, or maybe not
even possible to immediately make a product out of this and monetize it.
This truly is a gift to humanity, and a project like this can only emerge from proper long
term thinking that focuses on what matters in the long term.
Not just thinking about what is right now.
Bravo.
Now, of course, not even AlphaFold 2 is perfect.
For instance, it's not always very confident about its own solutions.
It also performs poorly in antibody interactions.
Both of these are subject to intense scrutiny and followup papers are already appearing
in these directions.
Now, one last thing.
Why does this video exist?
I got a lot of questions from you asking why I made no video on AlphaFold.
Well, protein folding is a highly multidisciplinary problem, which, beyond machine learning, requires
tons of knowledge in biology, physics, engineering.
And my answer was that I don’t feel qualified to speak about this project, so I better not.
However, something has changed.
What has changed?
Well, now I had the help of someone who is very qualified.
As qualified as it gets, because it is the one and only John Jumper, the first author
of the paper, who kindly agreed to review the contents of this video to make sure that
I did not mess up too badly.
Thus, I would like to send a big thank you to John, his team, and DeepMind, for creating
AlphaFold, and helping this video to come into existence.
It came late, so we missed out on a ton of views, but that doesn't matter.
What matters is that you get an easy to understand and accurate description of AlphaFold.
Thank you so much for your patience!
Thanks for watching and for your generous support, and I'll see you next time!