Goodness. DeepMind’s new work might be one of the best papers of the year! So what is going on here?
Well, today, we can use AI techniques to generate images from text, videos from text, but wait. Are
you thinking what I am thinking? It is great to look at all this, but this is a game. I don’t just
want to look. I want to play! And DeepMind’s amazing new paper is about exactly that.
Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Now believe it or not, we already looked at an earlier NVIDIA paper
that did something like this. So is this a solved problem? What did it do
exactly and what does DeepMind’s new work do that’s perhaps even better?
Well, this is NVIDIA’s GameGAN. Normally, if we wish to write a computer game, we first,
envision the game in our mind, then, we sit down and do the programming. But this paper did this
completely differently. It first looked at someone playing the game, and then, it was able to code up
the game so that it not only looks like it, but it also behaves the same way to our keypresses. You
see it at work here. Yes, this means that we can even play with it and it learns the internal rules
of the game and the graphics, just by looking at some gameplay. We don’t need access to the source
code or the internal workings of the game, as long as we can just look at it, it can learn the rules.
And scientists at DeepMind just put out a paper that made my jaw drop.
I mean, look at the list of authors,
this is essentially a supergroup. Wow. I am very excited. So what does this do?
Well, it doesn’t even need to look at an already existing game, because it makes
a game from scratch. Oh my, this sounds not like text to image, not even text to video,
but text to video game. So here’s the promise, in goes a piece of text. The text goes into a
text to image AI that produces an image, and now, hold on to your papers as we can
now start playing with that image. Wow, just look at that! A fully AI-assisted workflow.
It recognizes who should be the playable character and which of this is the environment, creates the
controls for this character, like moving around and jumping. It also learned the parallax effect,
so it knows which the foreground and background is, how far away they are, and how quickly
they should move compared to each other. Bravo, this is already incredible. But it gets better.
So far the input was eventually an image, and that image can also be
a photo from the real world. You add the photo and out comes a playable game. Yes,
we will talk about the fact that this is quite pixelated in a moment.
But you see, we don’t even necessarily need a photo
from the real world - we can also use a sketch! Just draw something,
and you get a game out of it. My goodness - isn’t that the dream? So good!
And it does all this after watching videos on the internet and, let’s have a look at how it
relates to previous techniques. Those required additional information, for instance, they needed
to know the buttons that were pressed. But this one, look! This one is completely unsupervised.
That means that we don’t even need to label the videos and show which is the playable character,
and what buttons were pressed. Nothing like that. Just chuck in the videos, and now,
little AI, it is your time to figure it all out. And so it does. Lovely.
Now, this does not go as quickly as you see here, this runs at one frame per second. And
as all of you Fellow Scholars can see, it is also heavily pixelated. But we are wise
Scholars here so we are not fooled by the low pixel count. It is not like the super
high resolution, detailed output like OpenAI’s Sora can produce. However,
I feel that we are currently at a DALL-E 1 moment of text to game. What does that mean?
Well, this is what DALL-E 1 was capable of creating. And approximately a year later,
version two could do this. It was a huge leap in capabilities. And just imagine what this
technique will look like by the time it will go to version 3. I cannot wait to have a look at that.
Just imagine using this together with the miracles that computer graphics researchers are capable of
today. For instance, yes, what you are seeing here is not reality. This is a completely made up scene
in a computer simulation, written by hand, and it computes how the water and everything should move,
and a ray tracing-based technique simulates what this should look like. And now, Fellow Scholars,
imagine two more papers down the line this AI looking at footage like this,
and then, we would play with that. Oh my goodness. What a time to be alive!
I have written a Comment paper on this in Nature Physics, it is very short and
an easy and fun read, and it is freely available for all of you Fellow Scholars.
It is about simulations that look like reality. The link is available in the video description.
And every good paper goes beyond its own scope a little. And it is no different for this new
work. Yes, this paper may also help train robots that move around in this world. That
is fantastic because robotics has a huge data problem, and this work might give this research
area just what it needs to improve quicker. And it can also learn a bit about deformations. We
can play with this bag of chips and it knows what is supposed to happen. So, it creates
the games of today, and it trains the robots of tomorrow. What an excellent paper. Bravo!