[MUSIC].
>> Good day, and thanks for
joining me in this discussion of
end-to-end observability in Azure and in Applications.
I'm Dave McAllister, I'm
Senior Technical Evangelist in
the Observability Team for Splunk.
Observability is where I want to start.
Observability really is a data problem.
It requires data, open standard data.
In general, we find three classes of data that are
prevalent in observability: Metrics
or do I have a problem?
Traces; where is the problem?
And also how long the problem took.
Logs, finally, what the applications
or infrastructure think is happening,
that's causing a problem,
or detect, troubleshoot and root cause.
But observability has changed.
We've now moving from
an observability 1.0 where each of these three classes
were sufficient onto themselves and
to an observability 2.0 model.
Observability now needs to go from the end point,
the real user space,
all the way through the application,
the back end, and it's environments in here.
In general, there are lots
of moving technology pieces here.
We have pieces that make sense in here.
We can start by looking at the front end.
Synthetics, how we can do testing,
baselining and optimizing,
which brings this data together.
User monitoring, the real user monitoring.
We understand what our users are
experiencing while we are at the same point in time,
keeping track of the aggregate of how
the users are impacting our current systems.
We move into the endpoint monitoring
because our users are
no longer making use of just a desktop,
but they could be on their phones,
or we could even be looking at IoT type devices,
such as in automobiles.
Finally, our environments, the entire process behind it,
the services, the digital
experience that they are experiencing here,
the third party applications,
our new Cloud Native Microservice
container-based applications,
and in their own compute environments,
be they virtual machines,
be they in the Cloud making use
of networks or service meshes,
or even now Serverless,
where we don't even own or understand what's going on.
We get the service when we need it.
However, data is only as useful as we can aggregate it,
analyze and visualize it,
and in the long-term responded to it.
From there we go from a frontend
through the application and
infrastructure backends all the way to
the backend of our monitoring capabilities.
Observability really needs to have
this analytics powered integrated environment.
Our 2.0 Observability is around to having
a unified experience where data is
correlated automatically across this,
so the cohesive workflow,
regardless of where the exploration starts
or what features are data are available here.
We need to have a model that brings
all this together so that we don't have dead ends.
We don't have places that suddenly our information flow,
or forensic stop inside of here.
Then we need to be able to make sure
that all of these pieces, collaboration,
visibility, and incident management help us
reduce that time to resolution.
We live in an instant gratification society,
3.7 seconds is enough to
cause someone to abandon the shopping cart.
We need to be able to respond as quickly as possible.
One component doesn't mean that you are observable.
It's all of the data together, Metrics, Traces,
and Logs that deliver observability.
What about Azure?
Azure is one of the leading Cloud providers
that are dealt with these days,
and host a huge amount of enterprise ready,
high capability systems here.
It also requires that
same capabilities of being able to look at the Metrics,
the Traces and the Logs,
and those metrics can be from
the infrastructure environment, here,
the App Services, the functions themselves that are
prior all the way into the application space.
Monitoring Azure, monitoring the Host,
monitoring the Kubernetes, the AKS,
the Azure Kubernetes Service,
or even monitoring the Applications
with the APM are simple,
by collecting the appropriate data,
we make use of open Telemetry so that we can bring in
the data in an open standard method to
allow this data to pull in and be correlated
across all of the features and all of the data streams.
But with that, let's
take this moment and take a look at a demo.
Let me start by giving you
a little background on what you're going to see here.
Our fictional demo company runs
an e-commerce site and it runs on a platform they've
recently refactored using a Cloud-native approach with
a Microservices Architecture and
Kubernetes for the infrastructure behind it.
Now, like many other companies,
this has been critical for them
to do during the pandemic,
to scale out for the massive demand for
Hipster products like vintage typewriters,
home barista kits, and vintage record players.
Now, luckily for the SRE team that manages the site,
they now have access to Splunk Observability Cloud,
which not only includes updated versions of
the APM and infrastructure monitoring capabilities,
they've come to depend on,
It also includes Splunk RUM,
real user monitoring to provide
end-to-end visibility on end user experience and log
observer to provide easy to use in-context
troubleshooting for the Cloud
forward and Cloud-native environments.
Now, imagine I'm an SRE and I've been made
aware of some issues with
our frontend to investigate further.
The first thing I want to know is,
what are these issues and how bad are they?
In RUM I can go see what's going on with the frontend,
and I noticed two issues: a
spike in the number of frontend errors,
L, and a high backend latency.
It's not clear at this stage which of these
two is more of a problem or whether they're related.
I'm going to go ahead and take a guess
and check the backend latency.
Well, it's clear that the endpoint/cart/checkout
has a high latency,
and it's correlated for issues with
high page load times as well as high latency times.
Now, that indicates that there is
a slow backend point
causing performance that's poor across the board.
I click through the /cart/checkout link
on the endpoint latency panel,
and it takes me to Tag Spotlight where I could
see a number of errors over the last 15 minutes.
Now, they don't seem to be
isolated to any particular tag.
To dig in further,
I click on "Sessions" to view exemplar sessions.
Now when I look at these,
I can see there are some they're taking
what seems to be a little bit longer.
I click on the link here,
so I can see the full trace.
You can see here that we're showing
that the entire trace through the frontend,
and then also when it gets to
the backend where it seems to complete successfully,
but it's taking a little while.
Now, I could follow through with
this specific trace, right.
But rather than do that,
I'd like to be able to see what's going
on with the aggregate behavior.
I can do that by looking at the workflow.
When I do that, what it
does is takes me to the full set of
services that are backing
the checkout action
in application performance monitoring,
and you can see here that
the payment service seems to be having some issues.
There's a bunch of errors in the system,
has identified as root cause errors.
In other words, the services with
the most downstream errors that are
contributing to a degraded experience
>> Now, one of the things that's new here
is that when I select that particular service,
in addition to all of the other information I
have for troubleshooting that specific service,
I also have the ability to see other information in
the Observability Cloud that's related to
it that might help me in my investigation.
We'll go ahead and
automatically surface that for the user.
I can go ahead and take a look at the infrastructure
that underpins this particular service,
or we can take a look at the logs.
I'm going to start by looking at the infrastructure,
since this is something that's
pretty useful to be able to do,
to rule out that as a cause of problems that I'm seeing.
You'll note, when I click on this,
it takes me to a context view
of the Kubernetes Navigator,
where it's been narrowed down to the Payment Service.
When I go ahead and take a look at the pod that
the service is running
on and then look at it in greater detail,
you can see that there's probably
not a lot going on here.
The CPU percentage is really low,
less than one percent.
The memory is pretty stable.
There's some network throughput,
but no errors and no events that indicate any issues.
So probably best to continue my investigation,
to go back by taking a look at that Payment Service.
To follow that further,
I'm going to go ahead and do something similar to what I
did on earlier on RUM,
Real User Monitoring, and look at
the text spotlight for APM.
Here, I can see if there are
any correlations in the tag values
with the errors I'm seeing.
As you can see,
there doesn't appear to be anything that's
particularly correlated to it.
If I look at the tenant level, for example,
they're all evenly distributed with the errors,
but when I look at the version for the code,
one thing you'll notice is that
the newer version, 350.10,
has all the errors associated with it,
where 350.9 has a lot of requests and no errors.
That seems like a suspicious thing
I should track down further.
The best way for me to do that is
to go ahead and take a look at the logs.
I do have the option of looking at the traces further,
but I'm going to go ahead and dig into the logs.
Just like with the infrastructure is already in contexts,
so when I get there it narrows things down to
the logs that are coming in for the Payment Service.
As I can see here in this high density view,
a log of server provides to me that there are
some error logs that are coming
through that are associated with the Payment Service.
If I go ahead and click on one of those,
I get a little bit more detail on that and have
a much more nicely structured view.
As you can see here,
there's a message that says there's a failed payment,
there's an invalid API token.
A dead giveaway here is that
the name of the token has test in it.
In all likelihood, the Developer, probably me,
accidentally left in a test token
and now that's moved into production,
so it's not working anymore.
I just need to confirm that that is the actual issue.
I'm going to do that by adding
that message to the filter.
You will notice that all the logs
that are visible now have errors
because they all have that failed
payment processing message in them.
I want to see if it's correlated
as I suspect with a version of the code.
Instead of grouping by severity,
I'm going to go ahead and group by version of
code and apply that visual analytic.
What I see is going to do is refresh the view,
and show me all the things that have
particularly message on Payment Service,
and they're all associated with the
350.10 as I suspected.
If I get rid or hide actually the message that are coming
from the failed payment processing errors,
I can see there are some logs that
are now being captured here.
We filtered in that have the version 350.9.
That seems to be proved that I
was looking for the errors that are
associated with version 350.10.
The answer is pretty simple.
Let's go ahead and revert the code to a known good state,
so that we could restore a good service,
and then try and figure out what happened.
That's not the end of the story because as an SRE,
I also want to do preventative work here.
Now I know that this could be a problem,
I might want to go ahead and have
dashboards or alerts set up on this particular issue.
That's one of the things that's
really interested in this,
is that I can simply save this as
a metric here from log observer.
What's going to do is take the query I'd captured in
the Visual Analytics line
along with the filters that we applied,
and allow me to go ahead
define a rule that takes these logs,
converts them to a metric,
so that I can then use them in the rest of
the system to do dash-boarding and alerting.
Pretty exciting stuff.
Just to close off, what have we seen?
Well, we've barely scratched
the surface for the Splunk Observability Cloud.
In addition to all of
the new additions to the product line
and on top of the beautiful, elegant user experience,
we're able to take in all the signals
associated with our infrastructure,
and our applications, and
correlate the logs, the metrics,
and the traces to enable our users
to have no dead ends and
experiences while they're exploring
the data or investigating an issue during an incident.
As you can see, this is
the industry's most comprehensive Observability Suite.
Infrastructure monitoring,
application performance monitoring,
digital experience monitoring, logs,
as well as being able to do instance responses.
It's based on the fact that you need all the data,
you shouldn't throw away data.
When we're trying to answer questions that
we don't even think of now,
we need that data to be able to have
the responses to answer those new questions.
We need this data in real-time.
Remember that 3.7 seconds here,
it needs to be massively scalable.
Our systems are no longer one server type approaches,
they can be hundreds or even thousands
of products inside of a Cloud-based.
Finally, because of the scale
and because of the complexity,
we need to be able to have AI
and ML Analytics that help us determine
what's going on and call things to
our attention even when we didn't know they could happen.
In closing, the most effective debugging tool
is still careful thought,
coupled with judiciously placed print statements.
Brian Kernighan said this in Unix for Beginners in 1979.
''Observability is the new print statement.
The Observability Cloud provides
that insight for careful thoughts.''
Some suggested resources are here.
We can make these available to you as well,
but you can always also just try it out
for free at splunk.com/o11y.
If you have questions, feel free to contact me or follow
me on LinkedIn or reach out to Splunk directly.
Thanks, and I appreciate your listening.