(music)
>> So welcome everyone to part two
of my reinforcement learning on Azure series.
Tonight, we're gonna be focusing a little bit more
on the reinforcement learning side of things,
getting our first scenario up and running.
Today basically, I'm gonna review the Azure settings
that everybody should have set up.
We'll go through setting up like a really basic scenario,
and also kind of a basic script
for running your very first
Doomguy reinforcement learning agent.
All right, so, little bit of a review from last time.
So you should have a virtual environment set up
with all of your Azure ML core
and developer stuff installed.
You also need to make sure
that you have Azure management network.
If you are in Unix, you can use PIP install ViZDoom.
So the things that you're gonna need,
you're gonna need a workspace,
you're gonna need experiment,
you're gonna need these resources,
you're gonna need to like have those all set up,
and you're gonna need your config JSON file
with all of that information about the specific code
for your subscription.
And then you need to also set up a virtual net,
and a sub-net.
We did all of that.
Set up your security parameters,
and then you need to set up your head compute.
That's right, we did that.
And then you're gonna need
to set up your worker compute as well.
And these are what is going to set up the environment,
and kind of handle the environment inputs and outputs.
So those are the things that we set up last time.
Let's just make, we'll call it our simple scenario
that we're gonna use
for our first reinforcement learning adventure,
and we are ready to get started.
So the first thing that we're going to need to do
before we even start to run the script
that controls our agent
and allows our agent to make decisions,
and has those like environmental parameters,
we need a configuration file.
We need a scenario for our script to utilize.
And so for that, what you're gonna do
is you're gonna create,
let's just call it simple.config.
We're gonna use the basic wad that they gave us
because this is like super, super basic.
And we're also gonna use a map that they gave us
which is just the first map.
I'm going to copy out the rendering options that I used.
And then we want to give the Doom Slayer
our reinforcement learning agent
a certain number of actions for him to try
before we're like, okay, you're your cutoff.
Like, you're probably not doing so great.
We're going to start with really basic.
We're gonna start with move left, move right, and attack.
Really, really basic.
We're not even gonna allow forward and backward movement.
In reinforcement learning,
we're gonna have combinations of environment variables
and actions.
So we've just set what we're gonna allow our agent
to choose.
Now we need to show what env variables
or environment variables we're going to allow
to be part of our simulation.
So this is all that you need
for your basic configuration.
What wad you need, what map you need,
rendering details,
actions you're gonna allow your agent to use,
what variables you're gonna expose the environment to,
the mode and your Doom skill.
Now that we've got all of that set up,
we need to write our script.
Let's see, how do we want to set this up?
So we need to create a game instance.
Then what we're gonna wanna do
is we're gonna want to initialize the game.
We'll have like true false false.
There's one action, move left.
So now we have our action matrix.
Obviously we could create more combinations.
We could have the Doom Slayer move left, and shoot
or move right, and shoot.
But I'm keeping it like real basic.
We wanna start simple.
This is honestly like the hill that I will die on
when it comes to machine learning, start simple.
Because you need to be able to benchmark
your simplest models.
So you need to be able to justify sacrificing simplicity,
and straightforwardness for performance gains.
If you don't have performance gains,
you've got no right making your model more complex.
The next thing that we want,
we need to put in like a pause,
or a sleep time so that visually we can follow
what our reinforcement learning agent is doing.
I'm just gonna set this up.
We need to set up our episodes.
So if the episode is finished,
we need to follow our typical pattern
for reinforcement learning.
And get the state, get the environment state.
Once we have the state,
we need to kind of extract some information.
We're gonna start out with a random action from our matrix,
our action matrix.
We're gonna pick something random.
All done, let's close the game.
All right, let's see if this works
or if I screwed everything up.
I screwed something up.
Really?
So actually I forgot a couple of things.
Good job me.
All right.
I don't think that was the issue, but let's see.
Nope, that was.
All right.
So our Doom Slayer is not very good.
He's trying.
Oh, oh.
But a toddler I think would do a little bit better,
but to learn our agent needs to be refining the connection
between actions and rewards.
It needs to be mapping action state combinations to reward.
So this is where you literally make a table
of all of the possible action state combinations,
and then the value of that particular action
weighted by future states.
Then you have a chance to also pick an action
that maximizes your reward, exploitation.
Here we are only picking random actions.
There's no way to learn.
What we're gonna need to actually make this agent learn.
We're gonna need some idea of memory.
We need our agent to remember.
There are so many different approaches to this.
You can have a way to distribution
or a weighted sampling of memories
or action state reward combinations,
you can have based on chronology.
What else?
Those are the big ones that I've seen.
It depends on your reward function as well.
And so there are reward functions
that are kind of like MCMC
in that they're only considering the immediate reward.
I like to think of this as a greediness.
And I don't mean this as like Epsilon greedy,
metaphorical greediness.
If you have an agent
that is always prioritizing their immediate reward
instead of considering long-term rewards,
you might find a local maximum in your reward function,
but you won't find the global maximum
because it might take early game sacrifices
that end up putting you in a bad place,
that end up actually leading to the best overall outcome.
But you can also have the agent of play itself.
And this is called self play.
And this is where a lot of times
you'll have one or more models
that are either competing or playing against each other.
And that's how the agent can learn as well.
So our Doom Slayer is not very good at killing enemies.
That's where we're at, but it's intentional.
All right, lovelies on Twitch.
Please check out documentation
if you'd like to learn more about what we did tonight,
I'll see you all tomorrow.