>> Hey friends. Prometheus is
an open source systems monitoring and alerting toolkit.
But did you know that you can get
all the Prometheus metrics without Prometheus Server Management?
Learn how Azure Monitor for
Containers makes it super easy to collect
all those Prometheus metrics and create
dashboards without any management overhead.
Keiko is here to show me how today on Azure Friday.
Hey friends, I'm Scott Hanselman and it is Azure Friday.
I'm here to learn about Prometheus metrics
and Grafana dashboards with Keiko.
Can you talk to me about how to get
Azure Monitor working with containers.
I'm seeing more and more that people are coming
up with really clean ways to take
open source software and the best of it and
combine it with the best of
Azure and it sounds like that's what you're trying to do here.
>> That's exactly what it is.
So we have a lot of customers and again,
we had a lot of feedback saying, "Well,
we love the Prometheus metrics but is there a way that we
could actually take that into Azure Monitor?"
So that's again the reason why we ended up coming
up with all this mechanism in terms of,
how do we actually make sure that customer doesn't have to
monitor a monitoring tool or even manage it?
>> Monitor a monitoring tool.
>> Exactly.
>> I can see where that would be a hassle.
>> Yeah. That's where we wanted to make it more easier.
Especially with enterprise customers, it's going to be a lot.
Hundred AKS clusters it's just too much for them.
>> Oh, my goodness, that would be overwhelming.
>> Exactly.
>> Okay, and I noticed also that when it comes to
dashboards like Azure has
its own sense of dashboards and there's
application insights and all kinds of things but people
get a Grafana dashboard
tuned just the way they want it to be and they make it perfect.
They really like their Grafana and they want to stay with it.
>> Yeah. That's exactly what we're trying to
actually do here to actually open up to the community.
>> Okay, cool. So what am I
looking at in this diagram on our screen here?
>> So this is actually a good diagram.
It actually talks about how
customers currently today utilize Prometheus.
So on the left-hand side, you'll see
three different endpoints which is the node,
pod annotation, and then the kube-services as well too.
So what it actually does is that Prometheus provides you a way
to instrument your SDK into
the application or into the infrastructure, what not.
Then from there, you have three ways to
expose your metrics through the endpoint which is the node,
pod annotation, and kube services.
Let me explain a little bit quickly on that one too.
So for the nodes, let's say you have your metrics or
performance on each of the nodes and you want to actually
collect them or it could be logs that are on that nodes.
Sorry, it's a log like
event that you want to attach to see as well too.
So these are some of the stuff that you wanted to actually see.
So number one, you actually see
URL IP address but nothing and I put
it in like node exporter and in
case you're not familiar with exporters,
they are actually like a plug-in for Prometheus
that you could actually pull in the data automatically.
It is predefined metrics because
normally when you actually go through this end points,
you have to define a metrics and
these are metrics that I want to expose.
>> It's interesting also that I'm
hearing you say that the way the Kubernetes is
designed it's flexible enough that none of
the things that you're doing here are hacks.
Like it wants to be extended and you're doing
it in the most natural and normal ways.
>> Exactly.
>> So nothing here is unusual or inappropriate.
It's correctly how you
integrate Kubernetes into a Cloud like Azure.
>> Well, yeah. Especially with like application or
in-house like services that they're running,
it's something that we wouldn't know.
We don't know their metrics.
We don't know what they wanted to collect,
and this is where Prometheus actually helps us out really well.
>> Okay. Then what's
the log analytics agent and where is that running?
>> So that's actually a DaemonSet on
deployment replica that we run on AKS.
>> Okay. So a DaemonSet is a kind of
node that sits along the other nodes that I have.
>> So think of it as a specialized pod that's for monitoring.
Of course, you could actually use it without monitoring
and you can actually run it on each of the nodes as well too.
>> Okay. Then everything
goes into Azure Monitor and then, of course,
once something's in Azure Monitor that's where
the really interesting stuff happens because I've
got alerts that can fan out and do
all kinds of interesting stuff especially Log Analytics.
So it sounds like once I've plugged this connector in,
a whole world of potentially monitoring stuff opens up to me.
>> Oh yeah, and then especially with we have an out of
the box metrics that we also collect other than just Prometheus.
So this is where you actually could do
the correlation within Log Analytics.
>> Interesting. So if I already have my system running locally
or in another Cloud and I've got my metrics
and I've got my Prometheus metrics handled,
I've put together my dashboards and then I
have decided to move up to AKS,
I plug it into Azure Monitor for Containers and then suddenly
all the work that I've done before
becomes even more worthwhile
because Azure lights up with all this.
>> Yeah, that's correct.
>> Cool. You have a demo?
>> Oh, sure. Let me actually show you.
So I have an application that already
has the Prometheus SDK embedded in there.
Then here, as you can see,
it is a YAML file that has a pod annotation already embedded.
So you actually seen that I'm actually saying,
"Hey, you know what?
Scrape that information for me as well
as where the path is and then what port I should be utilizing."
>> The scrape is a good thing.
We think of screen scraping and it's a bad thing.
>> Oh, no.
>> In this way, scraping in metrics is a positive thing.
>> Yes.
>> Okay.
>> We're actually making sure that,
okay you know what, you
might actually want to scrape what you want.
We've actually within the agent what we did was we
created a config map which is a configuration
for that agent so that you could actually
scrape the metrics you only want to select.
>> Okay. That way you're not getting
everything and a bunch of junk you don't want.
>> Exactly. Especially with the exporters
like for example the node exporters.
You got a 100 metrics
and you're going to push it into log analytics.
You don't want to actually have the chaos and the mess, "Hey,
why don't I actually just
pick and choose what I really want to do?"
So we're given the flexibility to our customers.
So now that we've actually
have the pod annotation
actually embedded and then we actually deploy it,
here's actually how it looks like.
So let me actually show you the deployment already done.
Here you'll see the Prometheus demo that I already have deployed.
So now let me actually show you what it
looks like inside Log Analytics now.
So here, I'm actually showing the Log Analytics view.
Within the datatype called insights metrics,
let me just actually show you what it looks like in Log Analytics.
>> So in this case, you're running a query in the language of-
>> Kusto query language.
>> Kusto query language in the world of Azure
on data that came from within Prometheus.
>> Oh, yeah. This is an example
of an application that's showing bad request and a good request.
So you see the good and bad.
So you'll see the blue.
>> It's pretty bad.
>> Yeah, this is really bad.
It's a bad application on purpose.
Then you have the good one of those too.
So now, I'm just going to add to this, of course,
you could actually set alerting from here as we just
talked about earlier and then, of course,
you could also put it into the Azure portal dashboard as well too.
You can actually pin to the dashboard.
>> Yeah, even though I love Grafana dashboards
being able to go and create a dashboard really custom,
get it all nice and set up and then share with
my colleagues is really cool feature about it.
>> Oh, yeah. So now,
let me actually show you how it looks like on Grafana now.
So here, I'm actually showing you two things.
One of them is that application that I just
actually put in into Prometheus and then now,
with the KQL query.
I'm able to actually showcase but at the same time,
we have out of the box metrics which I mentioned earlier.
>> Okay, hang on. I feel like there's an inception going on here.
Are you saying that you have information coming out of
Prometheus over to Azure Monitor and now you're having
a Grafana dashboard based on the Kusto Query?
>> Yeah.
>> Inception.
>> Oh, yeah. This is actually
a great collaboration as we just talked
about with Azure Monitor and
also with the open source which is Prometheus and Grafana.
>> That's really cool.
Because I might have existing Kusto abilities,
or people internally that just loved Kusto and know how to do KQL,
I can do any combination of
all these things and it just works seamlessly.
>> This is something that we'd like to actually give,
the flexibility to our customer because again,
with open source coming in,
observability coming in,
we don't know what the metrics are anymore.
This is where we want to make sure that
customers are using Azure as much as possible.
So here as you can see here,
the system out of the box
experience that I'm actually showing you,
we're showing you the cluster overall like
utilization and then there's actually
multiple things that you could actually do yourself as well too.
Cluster namespace, node CPU memory, node disk usage,
IO, networking, as well as Kubelet Docker operations too.
Interestingly, a lot of people don't know
that Kubelet, there's a lot of issues.
Let's say, I'm actually pulling
down the container image and that could actually happen there.
Why is my SLA down?
So again, these are some of the stuff we
have and then lastly as you can see,
hey, here's actually my application that I just talked about.
>> You put the exact same chart except expressed differently.
>> Yeah.
>> That's cool.
>> So let me actually show you
one more thing where let's say yeah,
these dashboards are actually great.
It actually compliments what we do as Azure Monitor.
But again, what if I do
want to actually see the period review that we do have.
>> I love that you have a link between.
>> Exactly.
>> That's nice.
>> By the way these are something that
anybody could actually create.
It's not something that we're actually providing anything special.
>> Right, you're plugging into
the system the way it meant to be plugged into.
>> Exactly, or providing a customer view
of a little bit something different such as noisy neighbor.
What am I actually showing?
What am I actually like running as a workload?
You want to see a workload in a live site maybe.
So here's actually all the controllers
that we are actually showing and
then also the pods and then containers as well too.
If they're having some issues,
you could actually see immediately through
live data or like in a view Log Analytics which will
take you to the Log Analytics page
and show you the container logs or even logs as well too.
>> Wow, okay.
>> So lastly, let me actually show you
because it seems like you liked the template that we just did.
>> No, I just think it's a really good looking template.
I'd like a nice dashboard.
I've always talked to people about
even if you have a small company,
if you have a small business maybe you're just
three or four engineers,
a really great way for a startup to feel
fancy is you buy an inexpensive TV from Costco,
you put a nice Grafana dashboard
on it and you hang it in your lobby
so that when people come into your startup and
they see your dashboard and they go, "Oh."
So we can do this with the power of Azure Monitor.
>> So now, it just happened to be we
just published our gallery template.
So if you go into Grafana,
you're going to actually see a gallery in
the dashboard that says Azure Monitor for Containers.
In here, all you have to do is put in the template.
>> Look at that downloads one.
That was us, we won.
>> Exactly.
>> We're first hopefully when people see this,
there's going to be a lot more, that's very exciting.
>> Exactly.
>> Fantastic. I'm learning all about
the amazing stuff you can do with Azure Monitor for Containers,
bringing your Prometheus metrics
and your Grafana dashboards together
with Azure doing amazing things today on Azure Friday.
[MUSIC]