COLT MCANLIS: In this episode of "Cloud Performance Atlas,"
I help some foodies bring home the bacon by profiling
their startup time.
Can their architecture stand the heat?
Stay tuned to find out.
I gave a talk in New York last week,
where I met the people behind FoodEpix,
a social network that only allows
you to share pictures of food--
no text, no audio, no emoji.
Just a picture of food and the geoloc tag
where it was taken at.
Turns out their team had read my previous articles on App Engine
startup performance and wanted to reach out and see if I could
help them with a similar issue.
While talking to their team over a round of free pizza,
I got a clear sense of their architecture.
As a user uploaded a picture of food,
it was sent to a Compute Engine instance, where
it ran some logic and processing on the picture
to make sure it's really an image of food.
Now, I didn't really figure anything was wrong
until they showed me a graph which looked something
like this.
They were seeing about 300-second cold boot
time for their applications while the response
latency for a request was in the 300-millisecond range.
Now, to be fair, by this point I hadn't really
dug in to Compute Engine cold boot performance that much.
So I was happy to use this as an excuse to investigate further.
Now you may see a common trend in my performance debugging
pattern at this point.
When trying to track down performance issues like this,
I first like to remove my code and establish a baseline
to know how badly my code is impacting things.
As such, the first steps in figuring out the problem
was to run a quick test to see if, number one, given
a static configuration, how long Compute Engine instance takes
to boot up, and number two, given changes
in that configuration, how does that
impact the previous number?
So, let's start with a simple test--
just 15 iterations of starting an instance and timing how
long until we can SSH into it.
On average, a two-CPU, 8-gig VM took about 60 seconds
to boot up, which I think does a nice job of establishing
our baseline.
Now, I would assume that improving resources should
see an improvement in cold boot time
since there's more cycles available to help the image get
up and running.
So I doubled the core and memory count
and saw an average of about 63 seconds.
A little worse, but still within margin of error.
This made me wonder if increasing the CPU and RAM
caused boot time to degrade linearly.
To test that, I increased both by 8x
and saw that the average time was about 62 seconds.
Which doesn't exactly conform to my thought
that VM startup time would be linear with resource requested.
More so that these timing fluctuations
had something to do with the small size of my sample set.
That being said, GCE did recently
update the max number of CPUs you can request in a VM to 64.
And when testing, we did see that these
had consistently higher cold boot
time than their less than 32 core counterparts.
So, just to make sure we wrap this all together,
let's do 200 configurations of random CPU and RAM sizes
and see what we get.
After testing this about 20 times,
the average still looked to be about 62 seconds.
And to be clear, those valleys and spikes
don't correlate with a specific configuration.
They move around pretty regularly
between iterations of the test.
This whole thing let me know two important facts.
Number one, cold boot time of Compute Engine VMs
are not correlated with CPU/RAM configurations directly.
And number two, the 300-second cold boot time
that the FoodEpix VMs were seeing
is being caused by something else.
So, let's get some more information about bootup.
Now, obviously, we need to install
the operating system on the VM.
Maybe we were using some weird configuration of image
that I wasn't testing with.
So, let's put them head to head.
Keep the configuration the same, but test a Linux instance
versus a Windows instance and see
if that could be the problem.
Not surprisingly, there was some variation in the results.
Most of the Linux-based builds acted pretty similar.
But the Windows builds were drastically slower.
And, just to get visibility, I also
tested booting clusters of machines.
So seeing how one VM booted versus 20 at the same time
showed some big differences with the Windows VMs.
But FoodEpix wasn't using Windows VMs.
They were using Linux VMs, and only launching
about one or two at a time.
So the choice of an image isn't having anything
to do with their slow cold boot performance.
Now, one nice side effect of profiling things
is that you typically end up learning more
about some part of the system that you didn't have visibility
to before.
Or, rather, you end up adding transparency
to a previous black box system.
For me, running these tests showed off
a few parts of the system I didn't know about.
Namely, that Compute Engine has a distinct latency
in various stages of VM bootup process.
As a request comes in to create a new instance,
GCE will move through the stages with variants' latency attached
to it.
Just to kick the tire, I enlisted the help
of fellow Cloud DA Terry Ryan.
And I was able to get a profile on each
of these stages, which actually didn't
reveal any specific problems.
Other than the specific startup process,
you can expect about seven seconds dedicated
to responding to a request and about 18 seconds dedicated
to provisioning the VM before any of your code
starts running.
Now, at this point, it's unclear what's
causing FoodEpix start time to be so slow.
But, we have eliminated CPU/RAM configurations and image
selection as culprits.
Sadly, though, I only have so much time
in each one of these videos.
So if you want to see what solved the problem,
check out the rest of the "Cloud Performance Atlas" content.
And remember, when it comes to performance,
every millisecond counts.
[MUSIC PLAYING]