COLT MCANLIS: In this episode of Cloud Performance Atlas
we take a look at App Engine Scaling Settings
to optimize our instance count.
Will we find the right trade-off?
Stay tuned to find out.
[MUSIC PLAYING]
In a previous episode, we helped the developer of Partly Cloudy
Image Suite reduce her instance count
with some clever shifting of workload
from the server to the client.
While this brilliant solution cut the number of active App
Engine instances significantly, some folks
have pointed out that we really should
have adjusted the scaling values of the App Engine YAML
file instead.
Now in complete honesty, I've never actually messed
with any of the settings in detail,
so I had no idea what they did, why they did it,
or why I should care about that.
But that's what we do here on Cloud Performance Atlas--
we figure this stuff out.
As such, it was time to dig into the Scaling Setting file
and figure out how they influenced the instance
count of our text.
Now I made some nice charts and things,
but before we get to that, I have to take a moment
and talk about how App Engine decides
to start a new instance.
I mean, we can't really start tweaking these settings
unless we know what we're touching.
See, the App Engine's serving algorithms are constantly
deciding on whether it's better to queue up a request
or spin up a new instance.
This takes into account a significant number
of factors for this decision, like request
queue depth, average queries per second, average request
latency, et cetera, et cetera, et cetera.
But to make things more simple, let's
focus on the four primary systems
that you can consider in your configuration file, which
can directly impact the decisions that App Engine makes
about spinning up new instances.
Number one is the number of concurrent requests
per instance.
Number two, how long a request, can sit in the work queue.
Number three, if there's any resident instances available.
And number four, what type of instances you're using.
In order to understand how all of these things
played together, I decided to run a little stress test
to see how many instances we could spin up
on the default setting with four seconds of static response
time.
Now, obviously this isn't a real-world scenario.
Now, when we throw 100 threads of connections
at our application for about 10 minutes,
we end up with a beautiful graph like the Google Cloud Console
has that looks like this.
My completely unrealistic scenario
topped out at about 100 instances
to handle my work-- the blue line right there.
So we can start tweaking some of the scaling settings
and see how that value responds.
Let's start with an easy one--
concurrent requests.
In order to optimize performance versus instance count,
you can control the max number of connections
you have per instance.
The Max Concurrent Request setting
allows you to control this value.
The default for this is 8, which we saw a graph for already.
I changed to 80 and re-ran the test.
You can see that the number of instances
dropped significantly from 100 to around 20 or so,
which is important since this means
my bill has been cut by a significant amount as well.
The trade here is straightforward.
The higher the value, the less number
of instances, you need to spin up connections.
However, if you set it too high, then you're
going to see latency problems as your instance is
strapped for resources.
Or it may cause new instances to be spun up
if you eclipse the available memory of your machine.
Lowering this value can help you find
the sweet spot between the number of connections
per instance, the resources needed
for your request, and the resources
for your particular instance class.
So let's reset that back to default
and see what the Pending Latency settings do.
See, when a request comes in, App Engine's front ends
will place it in a queue until an instance becomes
available to service the request.
If there are no available instances waiting around,
then App Engine has to make a decision.
Either it waits for one of the instances become available
or it starts up a new instance.
The values of min/max pending latency
directly control how the scheduler will allow a request
to wait in this queue before spinning up new instances.
The higher the min pending latency value,
the longer a request will wait in the queue
before triggering a new instance to be spawned.
Changing this from our default value of 30 milliseconds
to 6 seconds, we can see that the number of instances
generally drop compared to base use case.
It looks like we're topping out at about 65 instances
or so for this test.
Now max pending latency, on the other hand,
is the time after which App Engine
must start a new instance, if a request
has been waiting this long.
Setting this value to 4 seconds doesn't really
see much change in the total number of instances
that were spawning, but it might just
be due to the constant load that's being put on the system.
I mean it'll probably change once the workloads become
a little bit more variable.
Now, all of these settings so far
have dealt with the requests themselves,
but we haven't really talked about giving more resources
to the incidents.
It should be no surprise to you that your App Engine instances
are running on some physical piece of hardware
somewhere in the world, and the slower the machine,
the longer it takes to do heavy workloads,
leaving requests in the request queue for longer.
The trick here is to request the least powerful machine
possible that handles your workload
and doesn't force new instances to be created as a result.
The default for the instance class variable is F1.
So let's test an F4 instance, and see
how that changes things up.
We can see that for our particular workload,
changing this from F1 to F4 didn't really
do much to the overall number of instances.
However, I'd expect that to change if we were actually
CPU- or memory-bound instead of just sleeping for 4 seconds.
OK, OK, OK.
So we've tweaked to each setting and we
have a clear picture of what it does
and how it impacts our instance count.
So let's find the sweet spot.
This means that for our 4-second block of work,
we want the average latency for a response
to be as close to 4 seconds as possible.
To do this, let's set Instance Class to F4,
Max Concurrent Requests to 80, Max Pending Latency
to 8 seconds, and Min Pending Latency to 6 seconds.
The result gives us about 18 instances through our test
and only about 400 milliseconds' latency being added
to each request on average.
Now from here we can start tweaking the settings to make
trade-offs between the number of instances
we're willing to pay for and the latency overhead it causes
for our users, which is exactly the spot that we want to be in.
However, this won't solve all of your performance problems
in the Cloud.
That's why you should check out the rest of the Cloud
Performance Atlas videos.
And remember, that when it comes to your users,
every millisecond counts.
[MUSIC PLAYING]