[MUSIC PLAYING]
JEFF JOHNSON: Welcome back, developers.
Jeff here.
We spent a lot of time talking about Cloud Foundry,
but today I'm joined by Sharat of the Stackdriver team,
who promises to show us how to debug our production
applications with Google Stackdriver.
SHARAT SHROFF: Howdy, Jeff.
JEFF JOHNSON: Howdy, Sharat.
Can you tell me a bit about Stackdriver?
SHARAT SHROFF: Of course.
So Stackdriver is a suite of logging, monitoring,
and other diagnostic tools, which is primarily
targeted towards developers who are building and running
their application in the cloud.
It is designed to alert developers of problems
occurring in production and easily diagnose
the root cause of the problems.
JEFF JOHNSON: All right, sounds great.
SHARAT SHROFF: So one good way to experience Stackdriver
is by actually using the tool.
So why don't I just give a quick whirlwind demo of Stackdriver
diagnostic tools?
JEFF JOHNSON: All right, let's dive in.
SHARAT SHROFF: All right.
So let me open up an application here
that I've got running on the Google Cloud Platform,
and let's start with logging.
So every developer, of course, spends time looking at logs.
And what you would see here is a fairly simple
straightforward log viewer.
And this log viewer has--
of course you can view logs as they get
generated by the application.
But we have added a few more things
like the ability to filter by label or do text search.
And we can do things like log base metrics.
We can export these logs to BigQuery.
And then you have the whole power of BigQuery-- the ability
to slice and dice and [? permute ?] the data
in any which you want, available.
And all of this-- the export to BigQuery--
can happen in just a couple of clicks.
But one thing that we observed was
while developers and engineers take comfort in the fact
that there is data in logs, they really
don't like wallowing in it.
JEFF JOHNSON: Right.
SHARAT SHROFF: They don't like spending time like--
you could easily spend days looking at the logs,
trying to find that one specification that's
occurring in production.
So with that in mind, what we did was
we said we can probably do a better job of pre-processing
these logs as they come in.
And so we created Error Reporting.
And the best way to think of Error Reporting
is it's a snapshot.
It's a single snapshot of everything
that's happening in your application represented
on one page.
We take all the logs that's generated,
scan through each of them in near real time,
and then we intelligently group these
into different error buckets.
And for each of these error buckets
we provide some information like how often it's occurring,
a quick histogram of how frequently it occurs,
and how severe the issue is.
When it was first seen, when it was last seen.
We have the ability to slice these error
groups by time or by versions and by the application itself.
And you can of course click on this auto-reload
that allows you to effectively keep
getting the stream of information
be updated in near real time.
JEFF JOHNSON: Wow.
SHARAT SHROFF: You can click on any of this--
let's go ahead and click on one of these error groups here.
And what you see here, as you would expect,
is every single instance of that error
group captured in this view.
So you can drill into any of those instances of errors
within this error group.
In addition to, of course, showing you the occurrences
as we described and when it was first and last seen,
we give you a more detailed histogram
of where this is occurring.
And we capture the stack trace.
The view the most engineers are used to
is to basically look at this mind numbing
view of this stack trace.
JEFF JOHNSON: Yeah.
SHARAT SHROFF: My eyes glaze over whenever I look at this.
So what we try to do is, we scan through every one
of these stack traces and we try to surface
what we believe is the most relevant portion
of the entire stack trace.
You can always view the whole thing
by clicking on the Show All, but the past view
is meant to represent what we believe
is the most relevant section where we believe the problem is
occurring, where we believe is the exact source
of the problem.
So with all of this, what you have is
developers can get a snapshot very quickly,
drill into the snapshot of the specific instance
where the error is occurring, and have some clues on where
the exact source of that error.
So if you left it at that, you can imagine a developer
coming to the error reporting page, going to the logs,
and trying to understand the root cause.
So the common next steps we observed
is that developers would then try
to reproduce that problem that's occurring
in production on a separate dev box, on a separate test box.
And then they would go and insert some breakpoints,
and then inspect the application state
to try to understand where the problem could
be exactly occurring, trying to find
the root cause of the problem.
When they believed that they found
the root cause, what developers would tend to do
is to write some more code and insert some logs and then try
to make sure they validated that same hypothesis in production.
And so that would require the developer to write some code,
redeploy the entire application, wait for the problem to occur,
and then bingo, you would see the logs and say,
now I'm sure that I've found the root
cause, let me go write the lines to go fix the problem.
JEFF JOHNSON: Yeah, a few hours later, right?
SHARAT SHROFF: It's a few hours later.
Some places take a few days later--
JEFF JOHNSON: Sure.
SHARAT SHROFF: --because the problems might
occur at different places.
So especially you think of scenarios
where developers have rolled out only one code to only one
section of the instances and not all
because it might have some impact to the application
running in production itself.
So what we did was we started asking ourselves
some basic questions.
Could we capture the application state in production
when the application was running without actually impeding
the application itself, without pausing the application,
without impacting the customers or the users
of the application?
We asked ourselves the question whether we
can insert lock statements on the fly
when the application is running without, again, having
to write any lines of code.
So keeping that in mind, we built the debugger,
the Stackdriver debugger experience.
The best way to think of the Stackdriver debugger experience
is, let's take one of the specific instance
of applications here.
Let's look at this.
We've got what we have highlighted here,
and the relevant and the past few of stack trace,
is a specific file name and a line number.
And that's highlighted to you as a link, as a handy link.
If you click on that, we pop open the debugger.
And what is happening right now, the debugger
is trying to take a snapshot of that application
as it's running in real time.
And I think, as we were speaking,
it already took this snapshot.
And specifically what happens here
is no matter where the next time this code is executed--
I think this application has multiple instances.
It's running on multiple instances.
It doesn't matter when this occurs next,
it's going to basically capture the snapshot
and bring it all back into this debug view.
JEFF JOHNSON: Wow.
So you don't even have to pick, this
is my special instance I want to debug--
SHARAT SHROFF: That's right.
JEFF JOHNSON: It's my entire application.
SHARAT SHROFF: It's your entire application.
It can run on one instance.
It can run on 100,000 different instances.
There's no problem at all.
JEFF JOHNSON: That's incredible.
SHARAT SHROFF: We will scale completely without any problem.
And you can also add things like regular expressions,
which have to evaluate to true.
So you can imagine taking a snapshot
when a specific condition occurs,
or when a specific expression evaluates
to true, because you don't want to be taking snapshots
for every time that the code occurs.
It's when the code occurs and the user equals Jeff,
you can totally do that.
JEFF JOHNSON: Wow.
SHARAT SHROFF: And when you do that,
you instantly are able to capture the application state.
JEFF JOHNSON: Wow.
SHARAT SHROFF: We can also go insert logs
into the application logs by a simple command.
And you can do that through the Cloud Shell.
So you basically click on the Cloud Shell as you're used to,
and then we have very simple debug commands
handy that you can use.
And you can just go ahead and insert logs there.
One thing that we also noticed was that developers really
like to look at their logs in conjunction
with their source code.
So not only can you navigate your source code,
which is automatically selected for you--
the right version number and file names
are automatically selected for you--
but we also show you the relevant logs in this.
And you can use the same things like filtering that you're
used to in the previous few.
You can also look at the various snapshots
that I've taken before, or you can
see the snapshots that have been taken by other users trying
to debug this application.
With everything that we have done in Stackdriver,
you can share the same view.
We enabled deep linking into our experience.
It's as simple as copying and pasting this URL.
I could just copy this URL and send it over to you.
And when you open it, you would see the exact same view,
with the exact same highlights, with the same snapshot summary.
So that when we decide to debug an application together,
it's as simple as just copy pasting this URL.
JEFF JOHNSON: And that's something
that I traditionally have to say, OK,
come on over to my box.
I've got it just right.
Hope I don't hit refresh or something.
SHARAT SHROFF: That's right.
JEFF JOHNSON: That's incredible.
SHARAT SHROFF: It's pretty cool.
We like it.
JEFF JOHNSON: Well, seeing all these amazing features,
I would love to get these integrated with Cloud Foundry.
SHARAT SHROFF: Let's do it.
JEFF JOHNSON: Let's jump right into it
and see how we can get this thing set up.
So I've got a GO application, this
is running in Cloud Foundry as I said.
Can you show me how I can get it set up with the debugger
and error reporting?
SHARAT SHROFF: So one of the first things to do
is we first bring the--
your application is written in GO, correct?
JEFF JOHONSON: Mhm.
SHARAT SHROFF: So what we'll want to do
is bring in the debugger agent for GO.
For that you'll need to modify how
your app starts, to run this agent instead
of running your app directly.
So at this point maybe we should do it.
JEFF JOHNSON: All right.
All right.
So I'm going to pull in the debugger
agent to the same folder my projects in,
and I'll modify my PROC file, which specifies
how my application starts.
Instead of running the application directly,
I will just run that debugger agent instead.
SHARAT SHROFF: Yeah.
And next what you'll need to do is
provide some kind of source context for the debugger,
so that it knows which specific application version it's
working with.
And this can be generated with a simple GClock command.
JEFF JOHNSON: OK.
That sounds like an excellent job
for a continuous integration, continuous delivery
sort of environment.
SHARAT SHROFF: Exactly.
JEFF JOHNSON: I'll go ahead and just put it
in my deploy script for now.
So how does permissioning work here?
SHARAT SHROFF: So if you're running your application
in GCP, the debugger will pick up all the service
account credentials for the virtual machine
it's running on.
But if you're not running on GCP,
you can provide credentials to the service account
through a simple environment variable.
JEFF JOHNSON: Well, my Cloud Foundry is running on GCP
so I guess I'll get all those service account credentials
automatically.
But for folks who are running on other cloud providers,
we'll put information in the example application
on how to set up credentials.
SHARAT SHROFF: Great.
So now for the integration part, we'll pull in error reporting.
For the GO version you'll want to place error.catch
in a defer statement whenever you want to catch and report
panics.
JEFF JOHNSON: OK.
So I'm going to pull in the SDK with GO vendor.
And then hop into my program and import the SDK.
I'm just going to set up a variable
to hang onto the error's client.
OK, and just initialize that.
So my application is running with an HTTP
stack that supports middleware.
So I'm just going to paste in an empty piece of middleware.
And in there is where I'm going to put that
defer errors.catch statement.
So if an error occurs higher up in my HTTP stack,
in the Handler, it will get caught at this point
and reported out.
So with that in place, all I need to do
is install my middleware.
And that's it?
SHARAT SHROFF: That's it.
So now we can view the errors that are generated
and you can start debugging in production.
JEFF JOHNSON: Wow.
Well, let's pop open my web app and see if there are any bugs.
SHARAT SHROFF: Excellent.
JEFF JOHNSON: So hop into my application and say, "Hello."
And nothing happens.
So I happen to be trying this out in production,
but how would I know if a user hit this,
and I hadn't done that?
SHARAT SHROFF: So there are two ways you could be notified,
or you know the user hits it.
One, you can be notified.
Stackdriver has the ability for you
to subscribe to new errors occurring
and you can be prompted through an email.
Or if you're using one of our mobile apps,
then you could get a mobile notification as well.
You could, of course, always go to the Dashboard page--
the error reporting page-- and view it.
If you said that the error has occurred right now,
we should probably go look at your inbox
and see if there's a notification waiting there.
JEFF JOHNSON: OK.
So I'll hop into my email here, and it
looks like I do have a new email from Stackdriver.
OK, let's check it out.
So it looks like the error has occurred,
and I'll just check out the details.
So on this page it looks like I can see it's a new error,
it's occurred a few times here-- with different names
because I've been playing around a little bit--
and I can expand this and see my request right there.
That was that "Hello web."
I said I could see it in the request.
So I better hop into the debugger
to see what is going on because I thought
I had some great code here, apparently we've
got some issues.
Click on that stack trace.
And I'm going to select my application,
just navigate to that specific request.
OK.
So I know that my key-in value must be empty because I'm
throwing a panic there.
So I'll just set a break point above that, so I can
see what data they're getting.
OK.
So I'm getting that from VARS.
You can expand VARS, and VARS has values in there.
It's got 2, it's got a key, and it's got a value,
but why is it empty?
Ah, OK.
So it looks like when I'm looking at VARS,
I'm pulling out KNV, but my actual request--
the form side of it-- says KEY and VALUE.
Well, that's embarrassing.
Crazy it made it through code review.
But now that I've seen a request with real user data,
I should be able to fix it pretty easily.
Thank you, Sharat.
SHARAT SHROFF: No problem.
JEFF JOHNSON: And thank you all for watching.
Check out my application in the description.
This is a Cloud Foundry application written in Go
with the Stackdriver debugger and error
reporting ready to go.
See if you can find any bugs.
[MUSIC PLAYING]
Thanks for watching.
It's been a great series.
If you've missed a few of our videos,
make sure to check out our playlist
for more information about Cloud Foundry and Google Cloud
Platform.
I especially recommend this video on Stackdriver.
If you haven't already, make sure to subscribe
to the Google Cloud channel.
Thank you.