Hello, everybody and welcome to yet
another python plays a
GTA5 tutorial
In this tutorial what we're going to do is just kind of build not the last one the last one we proved that yes
We can actually stream this game to open CV/python at a rate of about 10 frames a second
Also, just in case that's not you like for example. If you're on like a prodigal. I don't know if you're on any computer
they can play Gta 5 I feel like you should unless your computer's horribly imbalanced have a
processor that can stream that image back, so
So I'm playing Grand Theft Auto 5 and then it's 800 by 600
so
So that as long as you have a computer that can normally play grand theft auto 5 at full screen on like high settings
You should probably have a processor that can stream Grand theft auto 5 at like 10 frames a second which to be honest
It's actually pretty impressive now that I'm like thinking about it
I'm almost positive that all of openCV is actually on the CPU
So that the CPU can play just about 5 at 10 frames a second. It's actually kind of cool
anyway
Yeah, let's the next thing that I want to cover here is
okay, so we used open CV to
Actually we need all we've done is actually used open CV to we did convert the color
But we also are really just using it to show the window so so we haven't really seen that
Sure, we could do some manipulations on this and not sacrifice like five frames a second okay
so the next thing we need to really test is whether or not we can actually do open CV stuff on this data, so
Just in case you're not familiar. You don't have to go through this series, but if you want to reference
Some stuff with open CV. You can go to pythonprogram.net
Data analysis, image and video analysis
and put a boom and then this is just a bunch of stuff [that] we can do with open CV
This is just a bunch of examples
But if you want to kind of get comfortable with doing things and especially like masks and regions of interest
You might want to check out
Check out open CV. The tutorials anyway
So the first thing we're going is just kind of do some typical processing of the the actual screen that we're pulling here, so
I think what I'll do is eventually we'll get rid of this
Print just because actually the act of printing especially an idle is
Especially over a long period of time is going to negatively impact our frames a second
so
You won't want to always have that there is
especially if you're running an idle if you're running through like a command prompt or terminal or something like that it won't be as
Impactful and you could leave it there for like debugging purposes
anyway
Let's go ahead and do some real [openCv] tasks, so I'm going to Define process underscore image
[and] then we'll pass image into it and then we're just going to say to you original
Image equals image, so don't reason we're doing this is
And in fact, we probably could just leave it as image, but I like the name original image, so it's clear
but we are kind of duplicating things ah
aah.
Get called this original image. I could change
Change that we can do a original image like [that], then we're not copying that image actually
Okay, we'll do that. So we'll call it a original image now. What we're going to say is processed image
So why are we doing this so?
Eventually what happens when you hit a typical kind of workflow with [opencv] is you have an original image?
And then you dumb that image down crazy to like do something like find edges or even more dumb like in our case
What we're going to do is we're going to attempt to use [openCv] to find lanes in a road
so as you umm.
Do that you're going to simplify that image so much so that eventually you just have two lanes, right?
But then what we kind of want to be able to do is take those two lanes and put them back on the real
Original image, so you always kind of want to save the [original] image
[so] anyway original our process image now will be [C] [V2] dot
Converts color we want to convert the color of
Original image, I'm pretty confident that doesn't happen in place. We're going to find out, but anyway
Hopefully, it doesn't anyway see b 2 dot color
And then this will be BGR [to] gray
We're converting the curtain to gray we're converting to gray
we're converting to gray because we want to have much simpler Data, so
BGR is
you know it's it's a
[array] [of] values from 0 to 255 right so it's [three] values per pixel whereas with Gray
Gray is simply a you know each pixel is a single value
It's obviously much simpler and we can feed that directly through you know a convolution neural network or something if it comes that
initially though my plan is to try to use open CV and
More like rules first before we try to you know throw it you can't just throw Neural networks at things
That's not how it works so so so we really I'd like some sort of way to
Determine whether or not we are in a lane and
Even if we're [not] a hundred percent like even if we don't have a hundred percent
Perfection at that what we can do is every time we have two lanes that work very?
confident or two lanes we can take that data and save that data to later train a neural network, so
Anyway, that's why we're doing that the other option [to] [is] to just play grand theft auto for like a few hours
Especially like we just drive the same road
over and over
and then take the
image data and then the input that we made as we drove that road and train a neural network off that that's
Possible to do and in fact just recently I saw an video released a thing where you can actually do that like [NVIdia] will log
Basically what you've done, and then you can train a model to play the game like you play it
So I'm very curious about that
I wonder how much training data it really takes, but interesting concepts so anyway back to what we're doing
Process image, so we converted it to gray
Now what we're going to go [ahead]
and do is just do like some typical canny Edge Detection so we can say we're going to just keep the process and
process image
equals cv2 dot canny
in here we pass the process underscore image, and then you have two thresholds
So we'll say threshold [1] equals
200 and then threshold 2
equals 300 you can tweak these numbers
Probably a lot it's going to depend on the game a lot will depend on just so many things so
This will be tweakable for sure
[now] let's go ahead and return the process underscore image
And now what we're going to do is
Under screen will keep we'll keep that happening
[in] fact under screen will do this, so we're going to say now new screen equals process image
original image and
then now
Rather than showing the [original] screen this one will [just] comment this out
Eventually we are going to show
Pretty much the original screen with some new things drawn on it. So I want to keep that there
[but] for now, we're actually just going to do CV [to] damn show, and we'll call this I don't know well
We can call this window and let's just call this window to before we forget
[Window] and then CV2 dot
Actually, [we] don't need to convert anything. Let's just do a new screen
Cool, all right. Let's save that and let's run that and see [how] we're doing up to this point
Move this out
These screen processed image original image [note]. We need to [process] screen
there again
Beautiful so here. We have the edge Detection, so this is a little unfortunate
I wish I could make this [just] [for] why this is so big
Maybe we'll have to resize this to 8 I guess eight-six
What's happening is my monitor is a 4k monitor, so this
This is real 800 by 600 this [I'm] sorry
This is a scaled 800 by 600 to like the [the] monitor does auto-scaling. Oh
Dear what have I done?
Come back. Oh, we've been trolled by windows shake
[let] me pull up the
Script again there we go and again actually we're still seeing ten frames a second. Which [is] actually really exciting for me
I'm very glad that we're still hitting that anyway we have edges
beautiful beautiful
Can you imagine playing GTA just with Edge Detection ah?
That would be pretty brutal, but yeah
We still are operating a really good frames a second so very cool very cool
Now the last thing that we need to do is the things that [I] kind of put off initially and again
I've already I know the solution, right?
and that is
Sending input to the game