(upbeat music)
- Hey guys, welcome to a bonus edition
of the AWS DeepRacer F1 ProAm.
- We're going to dive a little bit deeper into some
of the machine learning strategies
that helped us optimize model performance
for the F1 drivers.
- Finally we gotta race on our hands.
(laughs loudly)
- We'll also have some special guests on,
that will share some of the tips and tricks
that made them so successful on the track.
- Hi Rob, hi Brian, how are you guys doing?
- Good to see you guys again.
- Hi guys.
I see you guys have had haircuts.
(all laughs loudly)
- We wanted to be a little bit less shaggy
for the deep dive.
And we're gonna grow the hair through the video.
- Sounds good.
(laughs loudly)
- How's the season treating you Rob?
I see that F1 got started again.
- We've already done three races
I'm back into it with a passion.
- All right, super cool to hear about,
so now let's bring in JJ.
He was the second place finisher
in our head to head racing format.
- RayG , JJ and GT-Dev,
that's your podium.
- Hey JJ, how are you?
(laughs loudly) - Very well thank you.
- We were just chatting about how dominant you were
over the course of the race in May.
And really interested to hear some
of your secret sauce today.
(upbeat music)
One thing that we worked on with the F1 drivers early on,
was this idea of a racing line.
We started as many DeepRacer customers do,
with a simple follow center line reward function.
With Daniel Ricciardo, he immediately jumped up and said,
"Hey, that's not the optimal way to drive a track."
And we started working on this idea of a racing line
and how to cut corners, how to go straight through chicanes,
it sounds to me like that's not the same strategy
that you took.
- That's correct.
The algorithm is powerful enough
to find the optimal solution to these things
given a sufficient amount of time to do that.
So rather than me try
and tell it what I think it should be doing,
it's more efficient to let it figure out
what to do on its own.
- Can you give us any clues
as to how the reward function actually worked?
- Yes, I can.
All I do is stall the progress at each time step,
and look at how much progress it's made
since the last time step
and make the reward proportional to that.
I remember watching one of the early episodes
of DeepRacer TV, and you said the thing we need
to optimize is going around the track in the fastest time.
- The reward is the lap time?
- And I think you hit the nail
(laughs loudly)
right on the head at that point,
because that's exactly what about my strategy is.
- So actually JJ, thank you,
I had the right strategy,
but I was just let down by my engineers.
(all laughs loudly)
- That's something to consider.
- How much time did it take from when you started,
basically from scratch to when you got
that optimal model?
How long did it take you to produce that top time?
- Right over three weeks.
- Your car is following an optimal racing line,
but it learned to do that completely on its own.
And that's the part of this that's really interesting
and that gets me really excited
about reinforcement learning in general.
(upbeat music)
- And now let's welcome back
our inaugural F1 ProAm champion, RayG.
And let's find out exactly how he managed to win this race.
All right, hey Ray, how are you?
- Hi, Brian.
I'm good, very good.
Hi Eddie?
- We're curious to hear from you
if this idea of a racing line
or an optimal racing trajectory,
is something that you incorporate
into your strategy as well?
- Yes, indeed I did.
But the first step I took was basically
to generate a approximate racing line
and that's using a good GitHub repository.
And then I took the coordinates and plotted it
and overlayed it over the track
and over the center line itself
so that I could see what it looked like.
The blue points are basically the standard tread waypoints,
and they represent the center line around the track.
Whereas the orange dots represent my proposed racing line
from a reward function.
So at this point I realized that the racing line,
especially around the bends don't necessarily mimic
what an F1 driver will take
in terms of racing line.
But I just went in assuming
that whatever wasn't efficient about it,
the data would show it to me.
Now I just exported the entire set
of waypoints back into the list in python
and put it into my reward function.
There's certain parts of the track
where the curve is rather steep.
I would actually up sample my waypoints,
so just to get more granular waypoints around them.
I would primarily focus my rewards on steering accuracy
and not reward for speed initially.
- Very cool to hear that through working with DeepRacer,
you learned something like Jupyter Notebooks,
which are really one of the tools of the trade these days
for data scientists for machine learning engineers.
You're really showing kind of the power
of what you can do with the right types
of visualizations into your data.
(upbeat music)
So let's turn our attention
to log analysis here for a minute.
- One particular visualization was the ability
to look at the entropy as rewards
and track progress increases.
I wanted a way to see how much entropy there was
in the neural network itself.
- Can you talk a little bit more about how
we should think about entropy
and how we should interpret it in graphs like this?
- Yeah, so the way I interpret it is,
it's like a brain, an empty brain.
When we start off the brain is empty
and for it to learn it needs
to explore around the environment.
It takes random steering actions
and speed actions.
Then gains rewards for taking the right actions
and gets penalties for not doing the right actions.
It's fast to gain knowledge that neural network
of the brain gets filled with more and more information.
Entropy is basically a measure
of how much randomness there is left in the neural network.
So the smaller the entropy the better,
because that means most of the neural network is embedded
with useful information from past experience.
(soft music)
- JJ, there's a lot of information that's flowing out
of the simulation engine, pretty much on every time step.
- [JJ] This we're looking at here is the number
of steps it's taking to get around successful laps.
Now, what I've done is superimposed the most recent 20%
of evaluations on top of the first 20%.
So you can see the reds are the most recent stuff,
and you can see that distribution has shifted
to the left so we can tell things are getting better.
This was five days in,
that's the same thing, 14 days in,
from 460 to around 420 ish,
maybe it's the center of the distribution.
We could see, it's still improving
as an end, here's 23 days in.
Perhaps the red is slightly improving,
but pretty much it's kind of got
as quick as it's gonna go.
- You're able to learn a lot
of really useful insightful data just from
analyzing the lap times the way you did.
- That's exactly right.
So what I tend to do is look for,
the distribution shift into the left,
or getting narrower than therefore more consistent,
and when it stops doing that,
then it's time to have a look at the hyperparameters
and see if maybe something needs to be changed for them.
(upbeat music)
The main thing I'm doing is just reducing the learning rate.
So start off with the standard learning rates
and just let it run for several days.
When I'm starting to see the progress
not improving in the way that I would like then,
I might chop that in a half.
- Given the training technique that you took,
where you train this out over weeks and periods
of modifying the learning rate has a huge impact.
(upbeat music)
You know over in F1, we actually found
that the discount factor had a huge impact
into what we were doing.
- I tend to use a discount factor 0.95, .96, .97.
So I'm thinking that's going to give me a memory
of 20 odd steps
or something like that to some extent.
- That matches pretty well
what we saw with the drivers.
These first set of graphs that I'm showing here,
these are the exact same reward function,
the exact same configuration of everything,
except that we changed the discount factor
to three different values.
So on the far left, this is showing a discount factor of 0.9
and the middle we're seeing a discount factor of 0.99,
and on the right, we're seeing a discount factor of 0.999.
Which actually happens to be the default value
we provide in the console.
These results aren't gonna be the same
for every reward function
and for every track that you're racing on,
but simply by trying different values
and comparing the results,
each of these was trained for about six hours.
These vertical scales are actually the same
across all three graphs.
And so with a discount factor of 0.9,
we're seeing the reward jump up
to a hundred over the course of that six hours.
I think the important takeaway here is
that those hyperparameter settings are quite important.
(upbeat music)
- So JJ, one of the things that I talked about a lot is
to make sure that we're optimizing around
what we call the performance envelope.
How did you actually go about that?
- With some dimly remembered high school physics
and a bit of googling, plus poking through the code base
to find some of the values for the bits and pieces I need.
The coefficient of friction
and the wheelbase are defined in the code
and we know where gravity is,
so punching those numbers in gives us
a fairly decent estimate of the speed
and what speed you can drive around that angle at the point
where the car is gonna start slipping.
The fewer actions you have,
the easier it is for the machine
to figure out what it needs to do,
in the quicker it will train,
the more actions you have,
then the better it will eventually be able
to optimize given a sufficient amount of time.
- Okay.
- The physics calculations that you used,
you built those into your action space, is that correct?
- That's right, yes.
- So I think that's something
that we didn't necessarily think about
when we were working with the F1 drivers,
but a really kind of interesting way
to actually highly customize the action space
based on those calculations.
- Yes, so I wanna get the most bang
for the buck out of the number of actions I get,
because the technique I use takes long enough
as it is without a whole bunch of superfluous actions.
(upbeat music)
- RayG , can you elaborate on how to get
to the limit of the performance envelope?
- I use this to actually determine the performance
and the loop of my model.
Which is the number of lefts completed within an iteration.
So let's say a batch of 40 episodes for the duration,
how many of those 40 episodes and then complete left.
So I wanted to be able to visualize whether my model as
it trains, gets better
and better in terms of consistency.
So I look at the ratio of laps
that are 100% completed within an iteration
to determine which model checkpoint is suitable
or whether model is still improving.
I also plotted some additional data in this case,
this model does lap times between 31 and 38 seconds.
I know that on average, I will get 33 1/2 for this moment.
And I know that if I submit enough times,
I make it 31 seconds without any further training,
- It clearly works.
(upbeat music)
- When you qualified for the AWS DeepRacer F1 ProAm
and what was going through your mind?
- When I qualified second, I knew I had a chance,
but my money was on JJ actually.
I knew for a fact that his model
was definitely faster than mine.
In the end, I submitted a model
that was slower than my fastest model,
because it was more stable.
I was banking on less off tracks,
in order to have less penalties.
- Do you and JJ ever talk,
or is it like a friendly rivalry
or is it like just totally like
a death match between both of you?
If you were at a restaurant you're walking out,
cause this town isn't big enough for the both of you.
- No, we do talk.
Last year, I raced at the AWS summit in Toronto,
that's where I first met JJ.
We ended up on the podium,
so he was second and I was third.
We were actually encouraging each on.
- Not a lot of trash talking then.
(laughs softly)
- Not yet.
(laughs loudly)
Maybe in the championship cup when we meet again.
(upbeat music)
- So JJ, a major part
of what we do in Formula One is competitor analysis.
If you can somehow
suck some of that R&D into your operation for your benefit,
then why wouldn't you do it?
From the videos that we were putting out,
did you actually glean anything from what we were trying
to do with our FORMULA 1 experience?
- No, I didn't, I just kick the thing off at the start
of the month and just notice through this thing.
But it was very interesting to watch your progress
and see how you were getting along.
- To watch us fail, it was interesting?
(all laughs loudly)
(upbeat music)
- To struggle in the final, I did notice
that as I was coming around
to finish the
second lap I think,
and then catching up with the pack,
it was Daniel Ricciardo's car
that I barreled into the back.
And then that set in train a motion events
which pushed me way down the field.
I was never able to fully recover from that.
(upbeat music)
- Well, Ray, it was an absolute pleasure chatting with you.
Thank you so much for all your contributions
to AWS DeepRacer.
- Massive congratulations, I really admired,
and the guys who were up on top the leader board,
and then hearing about the amount of effort
that you put in.
It was a good lesson for me,
I'd absolutely like to do it again.
I thought it would be something
that they did in their spare time,
it wasn't even, it was almost like a full time job.
Incredible work ethic, really, really well organized,
I think this is the great part of DeepRacer
that we can actually sit down
and talk about it afterwards.
Thank you for being so open about it
that never happens in FORMULA 1 by the way.
(all laughs loudly)
You're always left wondering for decades
is to how somebody was beating you,
but you've been really, really magnanimous
in letting us know how you resoundingly beat us,
particularly Eddie and Brian here.
(all laughs loudly)
I was just along for the ride.
(laughs loudly)
(upbeat music)
- Hey, you were giving the orders, Rob.
We were just following your directions.
(both laughs loudly)
- Yeah, I get that a lot, thanks Eddie.
- Good luck with the rest of the F1 season.
- So this wraps up our inaugural
AWS DeepRacer F1 ProAm event.
But you can join us on a track for a new competition
at the start of each month at DeepRacerLeague.com
(upbeat music)