>> Hey. We're back to talk about hybrid and
multi-Cloud architectures with Eric and Uli.
This time we're going to talk about something that's
crucial to that, which is observability.
Join me as we get back into this conversation.
>> [MUSIC]
>> Welcome to the Azure Enablement Show,
where we'll be discussing the challenges
you and our other text-heavy customers have encountered.
Together, we'll be talking with experts to find
out how they think about these problems,
recommended tools and best practices,
and tips that I've learned from
years of experience that you can use.
Today on the Azure Enablement Show,
we do something we've never done before.
We're going to take our script and we're going to simply rip it up
because we want to talk today about
architecture and architecture is discussion.
We're back for another discussion here.
I thought that we are done last time,
but we're still going because
last time we were talking a little bit about
what you have to do for building stuff
in a multi-Cloud or hybrid environment.
Then we got to the question of like, okay,
it's cool that you built it, but now it's time to run it.
I would really like to get into this question
because the tools that we were talking about building.
To take this full circle,
do they help with reliability?
What are we going to do that actually
run/operate this from an architectural perspective?
Eric, I know you have some thoughts on this.
Tell us these thoughts, Eric.
>> Well, first of all, I don't think we'll ever
going to be done, David, do you?
I don't think so. We'll always find other stuff to talk about.
>> That's true.
>> But this one is topical.
For me right now because I'm helping a few
of our clients and customers go through this,
which is in the quest to build
resilient applications and self-healing applications.
One of the things we need to do is to get telemetry and
signal from these apps as to whether or
not they're operating correctly.
The biggest challenge associated with
this is how do I receive that telemetry?
Can I monitor an app from inside the app?
Is that okay or is that an anti-pattern?
But hint it is. It's a huge anti-pattern.
Now I need to plan for and design
a complete monitoring application architecture
that sits alongside the important stuff,
which is My Apps functionality
and its ability to contact multiple services.
Oftentimes what I've seen is that this is grossly underestimated.
Until there's a breaking problem with the application,
only then do we say, well, where are the logs?
Will the log sit inside the container,
which actually sits inside of a hosted infrastructure
where we can actually see and this as a mysterious text file.
Well, now we need to hurry up and write
some code to cradle this application.
All of this to say,
I think about this as an observability pattern,
something that we need to have discipline about
and have patterns associated with
how we implement the observability function.
>> The worst thing is when we get called into
a situation and the client tells you logs,
what logs? What is a log?
I think you start with what's
instrumentation and how do you bring information
out of your application that is
useful for troubleshooting scenarios.
Then your observation, Eric, comes in,
which is now you have observation logs.
How do you use them and where do they transport them?
The first thing is a,
acknowledge that you need data
to really understand what your application is doing,
and b, instrument it.
Great news is that they are truly packages,
software solutions out there that
help you with those kind of things.
But it's really important that you
do put login and instrumentation into your system.
>> I want to get some clarity on something
you said about the anti-pattern, Eric.
One pattern that I've seen before,
and I know it comes from our friends.
I saw first from our friends at Google,
was to have an endpoint
built into your application that you can query.
They call it Health-z, or stat-z,
or stuff like that where you can hit blah,
blah, blah, blah slash health-z.
The application itself is responsible for saying,
Oh, I'm feeling fine,
or here's what my capacity is,
or I can get that the database is running great,
I can or hand back information.
That sounds like you're asking the app to monitor itself,
which you said was an anti-pattern.
Are you talking about a slightly different version?
Or is that like a terrible idea because they seem to like it?
>> No, I think it's a good idea,
but I think depending only on that is really
where a version of this anti-pattern lies.
That to me is an example where the application is actually
responsible for telling others how it's doing, which is important.
But there might be a circumstance in
which the application is unable to
ascertain how it's doing and it goes silent, cold, or dead.
Now we need something that looks in from
the outside that's harvesting logs,
metrics, and alerts to say, hey,
the application is not responding,
and as an outside observer,
I'm actually telling you this because the application is
unable to articulate the fact that it is failing.
>> The pattern that you were referring to
both of you actually is a watchdog pattern.
Watchdogs are being used for all sorts of things.
For me, the Google recommendation is great.
I personally would recommend that anybody should do that.
But the implementation is where Eric is going,
where the order management as an application,
for example, shouldn't also have to function to report its health.
That should be a watchdog process that effectively
interacts with the logs or other elements of
this thing at the watchdog then offers
the endpoint that the Google friends have been recommending.
For me, you're more arguing about
an implementation detail than about the specific thing.
>> Well, I'm going to ask another question about
this and I am realizing that there
had been more hotel bar fights
over the question I'm going to ask you,
than many topics that I know about an observability.
Push or pull when it comes to observability.
Is it up to something
outside your app to be saying how you feel and how you feeling?
Or is it up to your app to be saying doing good,
doing good or here's my stats,
here's my stats, here's my stats?
If the dog doesn't bark,
that's when you know there's a problem.
Are either one of those preferable over the other?
I have seen many a conversation.
I would love to know what the two of you think about this.
>> I know personally,
I'll go out on a limb and say,
I'm a big fan of
exhausting logs and metrics from
an application to an alternate pipeline.
That pipeline is part of
the observability fabric that sits outside of
the application and monitors for
anomalies and events and raises alerts.
Now there are certain circumstances in which you might have
heuristic behavior inside of that observability fabric.
It says, well, hey, if I've detected
an anomaly or I haven't
heard from this application in a long time,
let me go ping it.
Let me see if it's up. Let me do a whole host of
tests and diagnostic tests
to determine if this thing is running or not.
Again, that to me is a watchdog implementation.
By the way, I will tell you that I had a manager when I was first
getting started in this technology stuff that said, well, hey,
why would I implement a watch dog without a watch cat to watch
the watch dog and then what if the watch cat
goes down because then I meant you to watch mouse?
I'd never really had a good answer
for that but I thought I'd bring it up.
>> Well, there are patterns for that as well if you want to
use consensus protocols for example.
If you really care that much about it you can have
three watch dogs and if two agree,
then things are still good.
If only one is disagreeing or dead you're still okay.
If you're looking at transaction
management systems that's what they do.
Consensus protocols is the answer to your question
Eric and that's the way to do it.
But to answer David's question,
I think Eric's suggestion of creating
a separate pipeline that ultimately pushes the information,
from an application perspective you
just keep writing and then there is
an alternative piece that effectively takes
the telemetry and pushes it somewhere else.
But I think there is also scenarios and
certainly in IoT we see it a lot where you do
send a life cycle where once in a while
I'm sending a heartbeat saying yeah I'm still here, all is good.
You can do that into a log or you can also do it as
an event and effectively make it available
as a push thing from your application.
It really depends on your scenario.
In the Cloud, mostly we do a little log writing and putting it
into the system but in IoT scenarios I've see a lot of heartbeats.
>> Well, I've seen people who treat logs
as heartbeats without being an explicit heartbeat.
Like if I stop seeing a certain volume of logs,
I call it a problem.
I don't know what I think about
that quite frankly when I see that.
It's also like, I know people are like if I don't get
paged by my system once an hour,
I'm really concerned that it's down and I
always thought that was like a serious anti-pattern.
>> It depends on how paranoid you're.
>> Sorry, say it again.
>> It depends on how paranoid you are.
>> Well, and if you like to sleep I think
is another concern there.
>> There's also systems of intelligence that
you can implement on the observability fabric side.
I'm using the term observability fabric to indicate
this separate app process that monitors other apps.
This observability fabric logs and metrics are great ways
for algorithmic analysis of the applications performance,
can actually be brought to bear quite easily.
If I'm seeing logs at a certain volume,
I can actually ask this system of intelligence,
this AI model to say,
is this weird that I'm seeing it?
If it is weird, let me know about it.
Or is there anything else associated?
Is this unusual behavior
associated with an analyzing these metrics and
logs associated with a particular element of functionality apps?
We can train these systems of
intelligence and the more telemetry that's
received in the pipeline is smarter
that these systems intelligence will get to say.
It might even go from saying, hey,
this is weird to, hey,
you should actually provision
another Kubernetes cluster because
it looks like load is increasing.
>> Yeah. That's actually a really great one.
I have a really good example for this.
The Azure Infrastructure actually uses
this technology using machine learning models
to predict if a specific host is going
to fail and we proactively migrate VMs off that host and take
the host out of the cluster in order
to avoid downtime and failures and stuff like that.
Great pattern Eric. But I think David the other piece
you wanted to talk about was a little bit of the operational side.
We have focused on
the observability and logging and so forth which is
clearly an important element but what else was on your mind there?
>> Well, when I think about operational stuff,
there's a number of components
and I'm happy to stick around and serve
the observability realm because I do have one more follow up
on that and then I'm happy to take it further up.
I think maybe we can only get to
the observability piece now and then
we can do the broader peace if we want to
talk later about more operation stuff.
Let me just ask the observability one as a way of taking us
out of this particular episode to be clear about that.
My question is this,
I think of logs as often being single system,
single-point pieces of information like this lag there.
But our apps now are not single pieces,
are not uniform things,
are not single things.
In fact, many of the cases if you've got
an app that has a bunch of
components or microservice stuff like that,
you have now many lags or many observability things there.
I'm curious from an architectural perspective what
we can do to make it easier to deal with
these not closely couple systems
in the same way so that you can have
a sense of what is going on in the app,
not like what's going on over there,
there, there, there there.
I don't know what's going on with the whole thing but
certainly I can look at that one over there.
>> Yeah. My thought and we're
happy to hear what you have to say here as well obviously.
But I think tactically
implementing the concept of a correlation ID across
multiple microservice boundaries and then collecting
those logs in a uniform or centralized place help you figure out,
well, this is the event and its let's view it and let me track
this throughout the call stack
of where the application is actually interacting with.
As a result, I can now marshal and bring
together logs and events across
multiple service boundaries into
a single analyzable set of
events that I can actually see and track.
I think that correlation ID might
be a direct answer to the question you raised there David.
>> Yeah, I think you need to do two things.
One, you need to first define what
your application is especially once you
get into microservices and other architecture patterns,
it gets confusing what is an app?
We need a first definition of what an application is and
then Eric's suggestion of a correlation ID
across the transactions or the flow of
information that you have is a great one and it's certainly
a technique that lots of people have been using
over the years to effectively manage
composite applications is really what we're talking about here.
>> Well, I know that we're going to want to be
talking about how architecture changes.
I'm going to leave as a teaser,
like cool correlation IDs, that's great.
But what if I have an app that's made up of a bunch of functions,
like if I'm doing functions as
a service or serverless sort of stuff.
I know we're going to hit that later.
When I drop that as a teaser and when we end this here,
and maybe we can come back and talk about some other stuff.
>> Cool.
>> That was a really cool set of conversations.
I hope you get a chance to watch those episodes.
We're going to move on to another topic here on
our Interior Architects as part of the Azure Enablement Show.
[MUSIC]