Well here's a demo, and it's also just me asking for advice, because I'm
sure there're ways to make this thing better. And what is this thing? Well, what I've done
is, uh, trained a neural network to predict what the Atari 2600 should be displaying
just based on the contents of its memory. So this lets you play a video game, play an Atari game,
not looking at the screen, but looking at what the neural network thinks the screen should be
displaying based on what the contents of RAM currently are. So I guess to get a feeling for
this, I mean the first thing to note is that the Atari 2600 only has 128 bytes of RAM. So it's
not a lot of memory! So we can actually play some Atari games and just look at the contents of
memory and just see: does it even seem plausible that you could play the game just by looking at
at the memory dump? So let's take let's take a look at that. We'll try the game Freeway by Activision.
Okay here I've got the game freeway, and in this game you've got to get the chicken across the
road. So you can move, uh, up and down and try not to get hit by the cars. If you,
If you do get hit by the car you're, uh, you're sent back. And if you make it all the way across,
(almost!) then i get a point. I can try it again. So uh we can look at the at the memory
dump while this game is uh is being played. Okay so here's, uh, here's that video of me playing
this game again, and you can see that this particular location in memory is going
up as my chicken player is moving up the screen. Right, and as i'm sent back down that number is,
uh, is being decreased. So the cartridge (maybe) is using that location in memory
to store the position of my player, the the y coordinate for my player. So here's what we're
going to do. We're going to have the computer play this game Freeway just thousands of times, mashing
the keys randomly. That'll present the computer with thousands of examples of, uh, the memory on the
Atari, the 128 bytes of RAM, and what the screen is supposed to be displaying. And then through
the magic of generative adversarial networks (GANs) the computer will learn a relationship between
the memory contents, the 128 bytes of RAM, and what's supposed to be on the screen and then we'll
be able to play the game, not by looking at the screen, but by looking at what the neural network
predicts the screen is supposed to look like based on the contents of memory. Okay so let's try that
Okay so we've got this game Freeway loaded. On the left hand side of the screen, you see the truth,
the the thing the Atari is actually displaying on its screen. And on the right hand side of the
screen, what you see is the prediction. You're seeing what the neural network is predicting
the Atari screen should look like just based on the contents of its memory
after having played thousands of games of Freeway. So here we go. So i will attempt to play the
game Freeway just by looking at the output of the neural network. So i'm not able to see
the true screen output -- I'm just looking at the neural network output.
And moving up... whoops gotta back up a little bit here, don't wanna hit those cars of course.
I'm gonna go up, ah, it's disappeared... well part oh! Okay part of the trouble is that when it was
playing this game randomly... i'll just run across the screen and hope that I make it. Did I make it
across? I can't even... okay I'm back at the bottom again. So part of the trouble is that when the,
when the thing was playing this game of course it didn't it didn't necessarily make it very far.
So the neural network has only been trained on examples [laughter] where it hasn't made it very far across
the road, so it's kind of... oh i'm not even sure where i am anymore. Rats! Well that that didn't work
as well as i might have hoped. Well here's another way to play with this stuff: we'll freeze RAM and
then we'll dump that frozen RAM into the model to get a prediction, but then we'll wiggle individual
bytes of memory to see the effect of those memory locations on the prediction. We'll try this on
Freeway. Okay so we'll start perturbing memory, start wiggling individual memory locations, just
starting at the top of memory here. And you know it doesn't seem to be doing too much. We can move,
move to the location that seemed to be related to the player's position before, and yeah I can
see that the player is kind of pulsing. Looks like that location in memory has something to
do with the with the player's position, which was what we saw uh earlier as well. Ooh now if
I go if I go farther in here it looks like that location definitely has something to do
with the with the player's position. Definitely makes the player sprite move. Oh and these, these
locations seem to have something to do with the positions of the cars on the freeway.
They certainly seem to be moving when i wiggle those those memory locations, and that location is
definitely moving the player. Looks like the player is is moving a bit when i move that, when i change
that memory location. What's surprising to me, I guess, is, is just how stable this stuff really
is. I mean wiggling individual memory locations doesn't change the predicted output by very much.
So i'd love to have ideas about how to make this better. I think this is a really cool idea and
clearly I could improve uh how well it works. Let's try it with some other games. Let's try the game
River Raid, which was certainly one of my favorites when i was a kid.
Okay well here is, here's River Raid. So here we go, boom! Okay oh but now see now i'm
actually looking at the true, the actual screen this time, which makes it quite a bit easier to
play. Let me switch over to just looking at the, um, at the prediction... okay, so now I'm only looking
at the prediction. Oh yeah I can't even really see where my ship... well I guess I must have died.
So I definitely played around with this some more... oh no I can't really see where my ship is
supposed to be. It's so i tried, uh, to try to improve the sampling in this game. I did
something a little bit different in this game. That I, uh, I just messed with, uh, memory locations as
the game was being played to move the uh the ship around randomly during training. I was hoping that
would at least get it to display... I'm... it's it's essentially impossible to play this game
just from the point of view of the, uh, of the learned output. But okay! So let me just try to
play this level looking at the actual screen, because I can actually play this game.
So let's, um, let me try doing that. So i'm going to fly through this to this level, since I
do want to demonstrate one kind of neat thin here... going through this level, getting some fuel...
So I trained this thing a little bit differently than I trained Freeway, just in the sense that I
randomly messed around with um whoops! With the memory locations, but you can see it
has learned some of the, uh, some of the actual scenery in in this level, sort of, i mean it
depends on how far this thing actually managed to play I guess during, uh, during training, but it's
definitely learned something about how River Raid draws its, uh, draws its levels.
Whoops! i lost again... okay well let's let's play [laughter] let's play another game!
Okay, here's Pitfall, another classic game. And you can see the prediction as I'm moving
my character around, it's not even... oh for a second there... let me see. Yeah you can. It is the prediction
is actually drawing the player on the ladder but not [laughter] not at any other time apparently.
Better run away from that i guess... but it's doing an okay job of... at least it's figured
out the, uh, the backgrounds, which is, which is something certainly. Yeah here it's looking...
oops! Oh for a second there... and yeah and the ladder. Definitely knows about ladders. But not,
not a lot else. So... so i don't know, still not a, not a fantastic job, I would say with with Pitfall.
There are some crazier things that you can do as well. Here I've got Pitfall playing on the
Atari, but i'm viewing the output of Pitfall through the model that was trained on on River
Raid, and this... you know, it doesn't, it doesn't do too much, but it is kind of fun, I mean, to see
what would, uh... what would the memory of Pitfall look like if it were being interpreted by
by River Raid [laughter] and uh well... not, not, not too good. You can kind of see, see if I
yeah, I can't even really tell when I'm, when I'm moving between screens on, uh,
on [laughter] River Raid too, too well.
Yeah look at that. That's, that's just a mess. Well it doesn't work too well, but the fact
that it works at all is amazing! Right, that's, I think it's surprising. Surprising to me anyhow.
If you want to play this yourself, down below in the description I'll put a link to a GitHub
repository with the Python code and also a link to some of the trained models, so you can try those
those models for yourself. I certainly would appreciate ideas about how to make this thing work
work better. It's been really cool i think being able to use PyTorch, just an amazing
kind of environment for setting up the kind of neural network stuff, and also the Arcade Learning
Environment, it's been really cool. It's kind of a playground for loading Atari games and being
able to play around with them in this sort of machine learning environment. i mean, both
both PyTorch and this Arcade Learning Environment (ALE), both are really cool. I just think it's crazy, you
know, I mean given that I played these games as a kid, uh, you know sort of messing around with them
from this point of view. I mean, it's kind of surreal just to think about what's,
just what's possible, and just the scale of my current computer. I mean
128 bytes of RAM is not very much. The models are quite large in comparison you know, and it's, uh,
it's just funny, i think, to think about how much computing power you know we
have access to today compared to what was on that Atari 2600 so many years ago.