COLT MCANLIS: In this episode of Cloud Performance Atlas,
we help a food image sharing service get performance back
on track with custom images.
Like your meals in 30 minutes or less?
Then stay tuned.
Last episode, we got a chance to work with the Foodie Pics
team who make a social network that only allows
you to share pictures of food.
No text, no audio, no emoji, just a picture of food
and the Geolotag of where it was taken at.
Their application uses Compute Engine instances to process
and store the uploaded images.
The issue that they've seen is that their instance boot time
is through the roof, and causing latency issues when
heavy load is on their system.
Now, we've already profiled the startup time stages,
as well as a bunch of random configurations,
and we're able to clearly eliminate the instance boot
process, chosen image, and underlying serving
architecture, as potential bottlenecks for performance,
which means it was time to start digging
into the code running on the instance to find the problem.
To do this, we take a look at the startup script
that Foodie Pics was using.
Once a new instance was created, their very first step
of their app was to execute a startup script, which
installed various services and code libraries that
would allow the instance to function
and be accessible externally.
Their script was significantly more complex,
but it generally looks something like this.
Clearly, though, we needed to figure out
what stage of this script was causing the performance burden.
Now, profiling a startup script is a weird exercise.
A, how do we get the timing data,
and B, how do we return that so that we can test it later?
For the first problem, I'm working in Linux
and dealing with bash scripts.
So the seconds command works just fine,
wrapping it around each one of the sections in the script.
For the second problem of reporting the data,
there's lots of options here.
Chances are that you've got code that already takes advantage
of powerful tool systems like Statsd or Brubeck,
but for my purposes, that would require
pulling down yet another package,
not to mention the time required in starting those services
before I'd be able to use them.
So I'll skip that for now.
On the other hand, the Stackdriver custom metrics API
might work.
See, all GCE instances come pre-installed with access
to write to this API, so that would get rid
of the need to bring down new packages
or host complex back ends.
Sadly, though, writing to the API
requires some complexity, namely authentication,
which I'm not willing to put forth effort into right now.
So we'll skip that.
For this case, I did something much more hack-tastic,
store the timings in a file and serve them as a new endpoint.
This allowed me to pull the endpoint
from an external location and get back my data
without too much heavy lifting.
Here is a graph showing the timings for each section
as 100 iterations of timing the startup script.
It's clear to see in the graph that the time per stage
of the startup script doesn't fluctuate
too much between tests.
It's somewhere between 60 and 75 seconds.
No one stage clearly stands out as a performance problem.
Really, it's the aggregation of all the stages
that's creating an issue.
Doing some more digging with the Foodie Pics' CTO,
it became clear that they only updated those scripts
about once or twice a month at this point, so the content they
were pulling down was mostly static, which pointed almost
immediately to their problem.
See, each time a new instance is spun up,
they're pulling down almost identical content.
The change is on, maybe, a monthly basis.
They're basically wasting a minute of cold boot time
to do the same thing over, and over, and over again,
which means we can leverage custom images.
A custom image, or image snapshot,
is basically a snapshot of the host disk information
after everything has already been installed.
When a target VM is booted, the image information
is copied right to the hard drive.
No need to go grab all that data one at a time
and install thousands of files from the net.
As soon as you boot up the new instance,
you're just paying for the performance cost
of installing the image itself.
For Foodie Pics, this was a huge performance improvement.
They were able to make a custom image with most
of the big packages installed, but still allowed
a small startup script that pulled down
the server front-end code.
The result was moving from a boot time
in minutes down to one measured in subseconds.
And just a note, even though we're using a custom image,
the boot phase of installing their image really
didn't fluctuate much, which is fantastic,
considering that we're not only installing our core operating
system, but also a bunch of other software
which was previously pulled down via network commands.
The Foodie Pics team ran into a very common problem
for most developers.
Once you reach a certain level of complexity in your startup
scripts, the cold boot time of your application
starts to suffer, at which point it's a great idea
to consider using custom images to bring down your boot
time to something normal.
So, if you're interested in more Cloud Performance goodness,
make sure to check out the rest of our videos and remember,
when it comes to performance, every millisecond counts.
[MUSIC PLAYING]