Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to see whether NVIDIA’s modern AI system they call MineDojo
can learn to play and build things in Minecraft and more.
In an earlier episode, we explored a previous project that goes by the name GANCraft,
which performed world to world translation. What is that? Well, simple, we give an AI a
very rough draft of a virtual world, and out comes a much more detailed and beautiful one.
This almost seems like science fiction. Really, just dream up something, and it makes it happen.
And, it gets better, it created water, it understands the concept of an island,
and it created a beautiful landscape,
also, with vegetation. Insanity. It even seems to have some concept of reflections,
although they will need some extra work to get it perfectly right. It even supported interpolation,
which means that we can create one landscape and ask the AI to create a blend between different
styles. We just look at the output animations, and pick the one that we like best. And, with this,
finally, we also have some artistic control over the mood of the results. Absolutely amazing.
But then they thought, why stop there? For instance, DeepMind has a long history of
using AI systems to master all kinds of games, from Chess, to Go, to StarCraft 2,
OpenAI has a DOTA2 AI project that is able to challenge a world champion team,
so, NVIDIA thought, why not try their hands at Minecraft? Minecraft is an open world game,
one of the most played games in the world, which is kind of like a virtual sandbox.
Here, you can build things, explore, but really, do whatever you wish. It’s a sandbox after all.
So scientists at NVIDIA thought, let’s train an AI to play this game and see what happens. Okay, but
how? Well, most AI systems of today require lots and lots of training data. Wait a minute.
There is no shortage of that out there in the internet, that’s for sure. Let me explain.
We can have this little AI sit down and watch hundreds of thousands of Minecraft
tutorial videos on Youtube, 33 years of footage in total, my goodness, then, have it read over 7
thousand wiki pages, and then also becomes the ultimate Reddit lurker. Wow. That makes a ton
of sense. Just think about it. It can learn practical knowledge from the tutorial videos,
encyclopedic knowledge from the wiki, and it can learn what makes the best creations
the best that are shared on reddit. That sounds fantastic…on paper. But, there is so much to do
and to learn about this game, can an AI really get an understanding of all this? Now,
hold on to your papers, and let’s see together what this AI could learn from all this data!
But wait a minute! This is a gamer AI that uses the controller to move around in
a virtual world. But how do we instruct it? Do we need to speak robot? If not, well,
it doesn’t even understand English text. What do we do with that? Well, remember OpenAI’s GPT-3,
which is a neural network model that has read almost the entirety of the
internet. This one has proper English knowledge, so, we plug that in, and bam,
now you can read that wikipedia, and it gets better, because we can now even give it text
instructions. Now let’s see how well it fares through 5 of my favorite examples.
One, we can ask it to explore an ocean monument. I love this one because the text description
is sufficiently vague. How could a machine understand what exploring means? Well, humans do,
and this learns from humans as there must be tons of tutorial videos out there on exploration.
And, it seems to me that it was able to learn the concept correctly. Loving it.
Two, it can encircle these llamas with a fence.
That is excellent. It understands what object it needs to use, where to move, where to look,
and that not even a tiny gap is allowed. Very impressive.
Three, this will be a more dangerous quest. Now scoop a bucket of lava. Oh my goodness.
Be careful there! Do not fall! We are getting there, and…got it!
Four, we can even ask it to build a nether portal. It builds the correct portal
frame and, did it use the correct materials? Does it work? Yes and yes! Good job little AI!
And, five. It is final boss time. Literally. Oh yes. Someone was brave enough to ask the AI
to fight an ender dragon, essentially the final boss of the game,
and the AI was brave enough to try it. Well, apparently, this is a limited example
as it does not appear to be charging at the AI, but, the AI seems to know what we are looking for,
and what fighting entails in this game. Starting this game from scratch
and building up everything to defeat such a dragon takes a long time horizon, and will be
an excellent benchmark for the next AI one more paper down the line. I’d love to it perform this
start to end. Make sure to subscribe, if such a paper appears, I’ll be here to show it to you.
While we are looking at more examples of what it could do, I have to note that we have
barely scratched the surface here. For instance, it understands text,
yes, but if we attach a speech recognition AI to this agent, we don’t even need to write.
It can essentially be a little virtual friend. How cool is that? What a time to be alive!
Okay, so this is MineDojo, an AI agent that understands English,
and can learn to navigate these virtual worlds so well, we can give it a task, and they would
execute it. And not just some simple ones, we are talking a wide variety of complex tasks.
Now, all this footage looks great, but how do we know if this is really
performing these tasks correctly? Well, if you have been holding on to your papers, now,
squeeze that paper, because they had an experienced human evaluator look at these results,
and agreed with the AI’s solutions about 97% of the time. Wow. That is an incredible result.
And don’t forget, NVIDIA is amazing at democratizing these works and putting them
into the hands of everyone, and this one is an excellent example of that.
If you have some programming knowledge, you can give it a try right now.
So, does this get your mind going? What would you use this for? Let me know in the comments below!
Thanks for watching and for your generous support, and I'll see you next time!