Hello World! Its Siraj
point A to point B using a special type of reinforcement learning called Q
learning reinforcement learning means learning by interacting with an
environment through positive feedback or reinforcement it's similar to how you
give a dog a treat but only if it rolls over and it's evolved over the past few
decades in the late 1950s an American mathematician named Richard bellman was
trying to solve what he called the optimal control problem this describes
the problem of designing an agent to minimize some behavior of a system over
time eventually he and his colleagues finally discovered a possible solution
to it which was later called the bellman equation it describes the value of a
problem at a certain point in time in terms of the payoff made by previous
decisions and it also describes the value of the remaining decision problems
that result from the initial decisions by involving a systems various states in
this way it broke the problem down into simpler subproblems the bellman equation
is now used in many many fields it helps with minimizing flight time for
airplanes maximizing profits for hedge funds minimizing the beef that soldier
boy seems to have with everyone yeah John episode no no no no no no Suraj it
wasn't b1 though bellman Squad was making waves throughout the math
community but meanwhile a psychologist named Edward Thorndike was trying to
understand how learning works by studying the animal kingdom he came up
with what he called the law of effect which states responses that produce a
satisfying effect in a particular situation become more likely to occur
again in that situation and responses that produce a discomforting effect
become less likely to occur again in that situation thanks Captain Obvious
now is actually pretty important discovery one of his experiments was
putting a cat in a wooden box and observing it while it tried a bunch of
different ways of getting out until it finally hit the lever that opened the
box when he put the cap back in the box it immediately knew to hit the lever to
get out and it was able to do that because of the process of trial and
error which is what Thorndike was essentially describing in the law of
effect a couple decades later a British computer scientist named Chris Watkins
thought that perhaps these two ideas could be combined to create a new type
of learning algorithm the idea of designing an agent that minimizes some
behavior of a system over time like the bellman equation and does
through the process of trial and error similar to the law of effect and so he
invented a novel reinforcement learning technique he called q-learning
so what is this let's say we had five rooms in a building connected by doors
and we'll just think of everything outside of the building as one big room
all of space-time is room five we can think of this system as a graph each
room is a node and each door is a link like room one has doors to both room
five and three so they're connected our goal is to put an agent in any room and
for it to learn how to get to room five through trial and error so the goal room
is room five the set room five is a goal we can associate a reward value with
each door which is the link between nodes so doors that lead immediately to
the goal room get an instant reward of 100 doors not directly connected to our
goal room gets zero reward in q-learning the goal is to reach the state with the
highest reward through a set of actions so if each room is a state each action
is represented by an arrow and the mapping of state to action is the agents
policy it uses the reward value as a signal to improve its policy over time
and it stores what it has learned through experience and what's called the
Q matrix the rows represent the possible states and the columns are possible
actions leading to the next state it updates the Q matrix over time as it
learns the best actions to maximize the reward seems pretty useful right shoe
learnings got to be used everywhere in video games right consumer video game
BOTS need to be good but not so good that a human couldn't beat them the bots
that used Q learning to master games like chess and checkers and most
recently Atari games become insanely good at whatever they play academic AI
learns while consumer game AI generally just makes educated guesses it doesn't
really learn and its actions are all scripted but as different as they are
the two fields are converging as we discover more about machine learning for
example in Forza Motorsport you can create a drive ATAR for yourself it's an
AI that learns how you drive by observing you and can then imitate your
driving style having adapted behavior like this will make games more
interesting and there's a lot of potential for more of it so let's write
out a 10 line high-level Python script that uses cue learning to train a bot to
get from point A to point B this game is a 5 by 5 grid our agent is a yellow
square and the goal is to find its way to the green square or the Red Square to
end the game each cell represents a state the agent can be in and there are
4 actions up down left and right moving a step will give us a reward of neck
point zero for the red cell gives us negative one and the green cell gives us
positive one so we ideally want to get to the green cell every time the game
world is already built for us so we'll just start off by importing that at the
top then in our main function we can go ahead and create a while statement set
to true because we want our agent to run indefinitely
next we'll initialize our bots position from a world class and set it to the
agent variable so now we want our bot to pick the right action to take in the
game world and the question is how do we decide that
I got a grid of squares measured five by five and I'm gonna get to grade in one
piece alive I'm gonna make a cube matrix initialize keeping every single reward
AB line archived then I'ma pick an action straight how to queue then I'm
gonna do it yeah you just fresh and new update queue width reward on B and once
I got that I'll go ahead and repeat yeah we'll use our box position as a
parameter for the max Q function which we'll choose an action from our queue
matrix as well as a potential reward then we can perform that action by
inputting the action as a parameter to the do action method which will return
our bot the action we took the reward we received and the updated bots position
after taking the action now we're ready to update our Q matrix so we'll use the
updated BOTS position as the parameter we'll print out both parameters to
terminal so we can observe the results we'll run the script by typing in Python
Lerner PI into terminal and it'll pop up as a GUI the bot will immediately start
trying out possible paths to get to the green one and we can observe the score
in terminal improving over time this bot in particular gets really good really
fast like in 10 seconds it's already found the ideal path and is just going
to keep doing it so to break it down reinforcement learning is the process of
learning by interacting with an environment through positive feedback
cue learning is a type of reinforcement learning that minimizes the behavior of
a system over time through trial and error and it does this by updating its
policy which is a mapping of state to action based on a reward the coding
challenge for this video is to modify this code so that the game world is
bigger and has more obstacles let's make it harder for our Q learning BOTS to
find the optimal strategy details are in the readme poster github link in the
comments and I'll announce the winner next video for now I've got to optimize
my life so thanks for watching