Fellow Scholars, I think you are going to love this. This is a follow up to an amazing project
where AI agents learn to play football/soccer. Initially, they start out like this. Well,
learning is a strong word to use here. But now, let’s jump 5 years in time, and look! They are
now competent players, but wait a second, was this really 5 years of training? Yes it was,
however, not in our time, but in their time. You see, if you have a powerful computer,
you can simulate these little AIs much, much faster than real time.
And now, we are going to look at a similar, really exciting sim2real project,
which means that first, the agents learn to play in a simulation, in a computer game if you will,
and then come out into the real world as real robots to shoot a real ball. We talked a bit
about football robots too before, but the new results are truly something else. However,
I am a little worried. I hear you asking, Károly, why are you worried? Well, look
at this. In this earlier software project, there was no referee, and therefore there
was no penalty of basically completely destroying each other, and look at that.
Absolute pandemonium ensued. Now I am worried that these little Robot Scholars are also going to beat
up each other really bad. So, do they? I will tell you in a moment why this is not the case.
Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
So, first, they learn in a video game world, and I already see that this is going to be
quite tough. Look, 20 degrees of freedom. That means 20 controllable joints. That gives us at
least 20 things that can go wrong. Ouch. This is not going to be glorious, not in the slightest!
This means that we have to start out from the humble beginnings. That controlling the arms
and the limbs of this robot will not be easy, so much so that they first need to learn to stand,
walk, and of course, get up after falling. Then, basic training in football. Wow,
this is not looking great. I wonder if enough learning can happen here so we get to see anything
interesting? Not sure. Then, soccer training against an opponent. This is not too impressive
at this point, and it will probably get even worse upon uploading this AI into a real robot.
Especially that these poor little robots barely see anything, and when they start running,
oh my goodness. It gets even worse. So now we see that this is a super difficult task,
so let’s see if it can be done at all.
First, a penalty kick is simulated in the video game world, and now, hold on to your
papers Fellow Scholars, and…yes! That was a good kick! Finally! Learning is happening!
But this is when things are going well. So what if things are not going well at all? Well, after
a bit more learning, look how robust it became against perturbations. What does that mean? Well,
this is a fancy way of saying that there was lots of fun to be had in the lab that day. And
it can recover from all this. Wait! Not only that, but it can even get up and score! Bravo!
But it gets worse. I know that for a fact. Why do I know that for a fact? Because I had the
huge honor of recording this piece of footage myself in person at the Google DeepMind lab.
Look at that! Ouch! I did not expect that at all, and it really shows how these robots can
fail in the most spectacular manner. Wow. And, a big thank you to Google DeepMind for the trip.
So, are they destroying each other? Not quite. Why not? Because even though there we don’t have
a referee in person, they were told that the rule is that if you get too close to the other robot,
you get a penalty. Whew! So we can expect peaceful matches after this insanity from a previous work.
And then, something incredible happened. After playing more and more against themselves,
they learned new things, and now they have 7 absolutely incredible new skills.
One, they were also advised to avoid high knee torques during the video game phase. Essentially,
they were taught to take it easy or otherwise that knee is not going to
make it. I love that as soon as they are locked into a somewhat human-like body,
they now have knee pain of sorts. Welcome to the real world, little AI!
And, over time, yes! They learned to walk and run a bit softer. So good!
Two, they can now kick a moving ball. Three, they learned to block the other robot’s shot
with their bodies, that is amazing. Four, can now turn much better, and look. Five, they can also
do something that I did not expect at all. And that is getting up…but from the back. So cool!
Six, they can also anticipate what is about the happen and position themselves
accordingly. And seven, their ball control skills are also a thing of beauty. Bravo!
And now, here comes the absolute crown jewel result. The manufacturer of this robot has
little handcrafted scripts, programs that control the robot for these movements. This
is what it looks like. These are carefully crafted by engineers who know these models
really well. And now, let’s compare it to their learned behavior in this project.
Oh my goodness. This is so much better! And they learned all this by themselves.
The AI agent now walks and turns 2-3 times faster,
takes only close to half as much time to get up, and get this Fellow Scholars,
it even kicks a ball 34% faster than this manufacturer baseline. Holy mother of papers!
This is an absolutely incredible paper and I feel honored to witness it together with you. Wow.
And note that this training could not have been done entirely in the real world,
but first only in a video game world because it would have taken too long,
and the robots could also have hurt themselves in the process. But,
after learning in a video game, we get this absolute miracle. And remember, in the game,
they can put in years and years of training in just a few hours. What a time to be alive!