COLT MCANLIS: In this episode of Cloud Performance Atlas,
I help some thankful developers come to terms
with resident instances.
Is their performance cup half full or half empty?
Stay tuned to find out.
I was attending a meetup last week
where I was approached by some developers
at a company called Think Thanks,
a nonprofit whose goal is to improve the quality
and frequency of thanks given to people and organizations.
They were a great group of folks who
built their service on App Engine
and had already done all of the cold-boot optimization
that we could possibly talk about.
Still, they were seeing issues when sudden traffic came in.
Their problem was that the app traffic was extremely bursty.
They would go from zero users for hours
to suddenly thousands.
And in those situations, instances
were legitimately being cold booted,
causing a delay in overall performance
while that instance was being created.
The latency of Think Thanks was taking considerable hit
any time a new instance was being booted up.
So besides just adjusting the scheduler
to boot up less instances, it looks
like they needed something stronger to handle
sudden spikes in traffic.
There was really only one thing left to suggest to them--
resident instances.
As I talked about before, the App Engine serving algorithms
are constantly deciding on whether it's better
to queue a request or spin up a new instance.
This takes into account a significant number
of factors to decide when this should happen.
If your cold boot time is significant, then
during extreme spikes in traffic,
request latency will suffer as those requests
sit longer in the work queue before being serviced.
This could flood your application
with a huge amount of startup costs
and delay the entire system significantly.
This is where resident instances come in.
A resident instance is one that stays alive 24/7
and only receives traffic under a very specific situation
while a new instance is booting up.
Basically, resident instances are spare instances.
While a new instance is booting up,
they handle the excess load that the new instance
would be handling.
Once it's up and running, the traffic
is sent to the new instance and the resident one
can go back to waiting around.
Which is really important because it
means that user requests won't be
subject to sudden spikes in latency that
come from cold-boot performance, which is exactly what I saw
when we changed the main idle instances flag of Think Thanks
to one.
Once the new request load came in,
it was handled by the existing instance
so that latency didn't shoot through the roof
while the new one was booting up.
The takeaway?
Resident instances are the key to rapid scale
and not having your user perception of latency
be impacted.
But after playing around with resident instances a bit,
Think Thanks noticed that there seemed
to be a plus-one tax involved in situations where traffic
was lower than normal.
Basically, if no one had been using their application
for a while, a sudden burst would
cost two instances, the idle instance
as well as the new instance being spun up.
But for infrequent traffic, that really wasn't what they needed.
Think Thanks was wasting an instance.
An alternative solution came from a Google post
where a community member described a similar problem.
His startup process might take multiple seconds
in order to fill a database.
But rather than using resident instances,
he created and used a cron job to ping the instance
and keep it alive indefinitely.
When we tried this on Think Thanks,
we saw pretty much what we expected.
In the lowest volume setting, this setup
allowed their back end to keep from spawning a new instance
when it didn't need to.
Happy with their latency improvement, the Think Thanks
folks finished their meetup-provided pizza
and then went happily along their way.
Afterwards, I talked to a few App Engine experts
who wanted to make it very clear that there's two very distinct
and separate use cases here.
If you have a high frequency, lots of instance
churn scenario and a slow cold boot time,
then resident instances are your best bet.
However, if you have a low frequency traffic
with occasional bursts in a slow cold boot time,
then keeping an always-cron instance around
is ideal since you can handle the initial traffic
load gracefully with a lower number of total instances.
The downside to both of these situations
is that these persistent instances come at a cost.
You're basically paying for an instance
to be alive 24/7, which could add significant costs
to your operating budget.
Which is why I like to say that you should always carefully
balance your tradeoffs before making decisions
to go forward with any performance improvements.
And to understand what types of trade there are,
make sure to check out the rest of the Google Cloud Atlas
videos.
Because when it comes to your users,
every millisecond counts.
[MUSIC PLAYING]