JEFF JOHNSON: Hi developers.
My name is Jeff, and I'm a software engineer working
on Open Source at Google.
ALPHA CHEN: I'm Alpha, and I'm a lab developer at Pivotal.
JEFF JOHNSON: I hope your Cloud Foundry
Series is going great so far.
You and I are going to spend the next few episodes talking
about Cloud Foundry and Stackdriver.
Stackdriver is Google's suite of logging, monitoring,
and developer tools.
I'm going to be joined by experts
who know way more than I do about Cloud Foundry
and Stackdriver.
So Alpha, can you give us a little background about how
Cloud Foundry is deployed?
ALPHA CHEN: Sure thing, Jeff.
Cloud Foundry is made up of many services that
run on an on-prem data center, or the VM
infrastructure of a cloud host, such as Google Cloud Platform.
These services are deployed as jobs by BOSH to several VMS,
and together they form a working Cloud Foundry.
JEFF JOHNSON: So we've got all these applications that
must be doing tons of logging, in addition to Linux services,
logging to sys log.
How can an operator make sense of all that?
ALPHA CHEN: That's a great question.
Every BOSH job writes logs to a known location on the instance.
So an operator can use the BOSH CLI
to grab a recent set of logs or stream them as they come in.
This is fantastic for initial setup,
but operators really need to set up a long term
solution for monitoring and storing logs.
JEFF JOHNSON: That's why we work together with Pivotal
to develop these Stackdriver tools BOSH release.
This drop-in solution gives operators
long term access to this information in Stackdriver
logging and monitoring.
ALPHA CHEN: We work together in Seattle
using a Pivotal process.
We co-located once a week in our office,
and once a week in yours.
And together we developed the Stackdriver [INAUDIBLE] tooling
that enables Cloud Foundry to talk to Stackdriver.
JEFF JOHNSON: We did this all out in the open
and it was a lot more fun than I thought it was going to be.
Now hear me out.
We had a lot of folks who knew about BOSH,
a couple of folks who knew about Cloud Foundry and Go,
a little bit of GCP of course, but not a lot of overlap.
By pair programming we were able to spread that knowledge
across, and we all came out with a much better understanding
of each other's products.
So Alpha, can you pair with me and show me
how to set up Stackdriver tools on Cloud Foundry?
ALPHA CHEN: Let's get to.
It First, let's upload this Stackdriver tools release
to our BOSH director from our open source
project hosted on GitHub.
Now we'll have to modify our deployment manifest to include
the release.
The two jobs, Google [? Phone D ?]
and Stackdriver agent will need to be
collocated on the instances that you want monitored.
After that we'll have to run BOSH deploy as usual.
In recent versions of BOSH we should
be able to deploy this as an add-on.
JEFF JOHNSON: Wow.
That's the fastest BOSH deploy that's ever happened.
Editing is a real miracle.
It might take a few minutes on your machine.
So now that we've got jobs running across our Cloud
Foundry, which means our host logs and metrics should appear
in Stackdriver, let's make our data
little more interesting by waiting a week or so.
ALPHA CHEN: Wow, that went by fast.
JEFF JOHNSON: Great.
Let's dive in.
ALPHA CHEN: As the logs stream in,
we can view and drill down the specific events.
These logs include all distance log output
from the instances in our deployment,
as well as the logs from any other BOSH jobs
on the instance.
Each entry contains contacts, including the job name,
which allows operators to easily filter and search through logs.
All of these logs come from the Google Fluent E job
that we deployed earlier.
JEFF JOHNSON: As for the Stackdriver agent,
that's responsible for gathering host metrics with Collect D
and sending them back to Stackdriver monitoring.
I'm going to hop into Stackdriver monitoring,
and we can now see detailed information about our Cloud
Foundry infrastructure.
Let's take a look at the dashboard I created earlier.
The dashboard shows metrics gathered by the agent.
Now we can look for abnormalities
and set up a learning threshold, such as high memory usage.
ALPHA CHEN: This is a great start
to having a single pane of glass for a CF operator.
These deployments are made up of tons of virtual machines.
So having a heartbeat like this is a great thing
to keep an eye on.
JEFF JOHNSON: Let's look at the log metrics we've created.
This metric is based off of an event appearing
in the Stackdriver logs.
We can drill into the actual log events where this occurred.
This gives us a query in the Stackdriver logging
console specific to our event and time range.
I'm going to filter to our specific virtual machine
and remove the event filter to gain some context.
This is an incredibly useful tool
for keeping an eye on your entire system
and drilling down when something pops up.
That's host monitoring with Stackdriver.
ALPHA CHEN: And check out the description
for more resources on integrating
Cloud Foundry with Stackdriver.
[MUSIC PLAYING]
JEFF JOHNSON: Thank you all for watching.
Check out some more videos we've done about Stackdriver
and Cloud Foundry, and check out our play list
for even more information.
Make sure to subscribe to the Google Cloud channel
right here.