COLT MCARLIS: In this episode of Cloud Performance Atlas,
I help some bacon lovers profile Kubernetes Startup Time.
We'll their performance end up with the perfect crunch?
Stay tuned to find out.
After my work with the foodie picks team,
another food based company approach
me to help with more cold boot performance problems.
Bacon Live is a mobile network for chefs,
where you can watch live video feeds of people cooking food,
annotations included.
The back end for Bacon Live was using Kubernetes,
which is a great tool to help you manage, contain, and deploy
images to your various instances in a seamless and powerful
manner.
Sadly though, reporting back details
on cold boot performance was not something it did easily.
So before we could solve Bacon Live's issues,
we had to figure out how to profile pod startup time.
After digging around, neither the cloud console,
stack driver or heapster surfaced any direct numbers
with respect to the boot time for pods.
There was, however, a single place where some startup time
information was being surfaced, in the Kub Control Get Pod
method.
It turns out that calling this function
will return a JSON string with some time-stamped information
relating to the pod startup phases.
Here you can see that the pod status
emits the scheduled, initialized, and ready phases,
each with a time stamp that tells
us relatively when it occurred.
So for this example, you can see that it took about one second
to go from initialized to ready.
So we can at least get some information about how long
things take to boot.
Now to get a sense of the average startup performance
and what it looks like, we need to create a loop, where
we kill the current pods, let new ones boot up,
read their time-stamp data, and then repeat.
Do that about 100 times or so, and we end up
with something that looks like this, which shows us
that for a blank pod, it takes about three milliseconds
to boot it up, which by the way, is a lot faster than booting
a compute engine instance.
And the reason for that is that when a Kubernetes pod boots up,
it's not starting a brand new VM each time.
It's just booting containers.
See Kubernetes pods are made up of a series of containers,
which provide endpoints and services for the instance
itself.
A pod is considered ready once all the containers have
been initialized and their health checks pass.
Now to help facilitate proper boot up,
the initialization process breaks up containers
into two main groups.
The first is the Init Group, which initializes specific
containers in a linear .
Order
The second is the Parallel Group,
which initializes the remaining containers all at once.
The setup has some interesting quirks and chances
to be problematic.
For instance, if a single container during the init phase
takes a long time, it will block loading of everything else.
So to understand where a pod boot up performance is going,
we need to be able to time the specific startup
duration of each container in each group.
So let's start with the init containers.
After a bit of digging, we found that the Kubernetes
get pod command will return the status of the target pod
during boot up time.
And when you have the init containers defined,
the Status field will list the status of the init containers.
For example, a status of init 16 indicates
that one of six init containers has completed successfully,
which means that we can pull the pod every 10 milliseconds or so
and get a sense of how long each stage is taking to boot up.
When we graph that information, it
becomes really easy to see that for this example, the database
initialization and stats frameworks are taking
the longest time to boot.
Now timing the parallel containers
is much trickier, since there's no time stamp or extra data
emitted when a container makes a transition
from initialized to being ready to handle requests.
So in order for this information to be surfaced,
we have to probe our endpoints, much like health checks
or readiness probes would.
These systems sit in a tight loop and touch the end points
until a specific response is received,
which lets the load balancer know that this container is
ready to handle traffic.
So for our set up, we modify each container
to expose a new entry point, which
returns the string "success."
Repeatedly, we're pulling this end point
from an external location, yields three different states.
First is error, which means that the service is
not up or running yet.
Second, is code of 400 or 500 series, which
means that the service is booting,
but the end point is not available yet.
Or finally, "success."
If we pull the end points every 10 milliseconds or so,
we can figure out how long the container sits
in each of these states and chart it as a pretty graph.
So armed with this information, we
could finally start looking at what was causing Bacon Live's
performance problems.
But for this episode, we're out of time.
If you'd like to know more, check out
the rest of the Cloud Performance Atlas content.
And remember, when it comes to performance,
every millisecond counts.
[MUSIC PLAYING]