Welcome to this screencast where we’ll get started
with the HPE Storage Array Exporter for Prometheus on Kubernetes.
My name is Michael Mattsson.
I’m a tech marketing engineer with Hewlett Packard Enterprise.
First, I want to give an introduction to Prometheus.
So Prometheus is a time series database.
And it basically retrieves metrics from infrastructure and applications.
It also provides an interface for users to query that database.
And the normal mechanism is that it pulls metrics
from something called an exporter or a job.
And it also have the ability to use the Push Gateway for short-lived jobs
that just runs for a short period of time
and outputs with metrics upon completion.
It also have the ability to discovery targets dynamically on Kubernetes
either by using labels or they’re also configurable through the file interface.
Normally, you will query the database through the Prometheus Web UI,
but you can also use API clients
and also use Grafana which we’re going to use in this example.
There’s also an alert manager.
We’re not going to touch on that in this particular demo
because I’m going to use Grafana for the alerting functions,
but it has multiple different outputs
and you can use very complex query languages to set up your alerts.
This demo is focused on the HPE Storage Array Exporter for Prometheus
which allows you to gather metrics from HPE Alletras,
Nimbles and Primeras in your infrastructure.
We’ll also use the HPE CSI Info Metrics Provider
that will allow you to gather metrics from your backend array
and your persistent volume claims running on your Kubernetes cluster.
The data gathered from the different backends,
it looks very similar but it has some nuances.
So the HPE Alletra 9000, Primera and 3PAR have one set of metrics
and the HPE Alletra 6000 and Nimble Storage gather a different set of metrics.
But at the end of today, we have all the storage accounting,
space accounting for both snapshots, free space and used space
to be able to build graphs and alerting based on that data.
There’s also a very detailed volume performance metrics,
but at the end of the day we want to be able to gather the read and write IOPS,
the read and write throughput
and the read and write latency from the respective backends.
In this demo, we’re going to use the HPE CSI Info Metrics Provider
for Prometheus as well, so assume that we have a Kubernetes cluster installed,
the HPE CSI Driver is installed.
And we also have a bunch of different HPE storage backends.
We have a couple of workloads running
that use this persistent volume claims so we can gather the metrics.
On the cluster, we will deploy the HPE CSI Info Metrics Provider
and we will install the HPE Storage Array Supporter
for each of our individual backends.
The HPE CSI Info Metrics Provider will give us information
about the backend volumes,
how they relate to the Kubernetes specific labels, right?
So you will see what the persistent volume name is, the persistent volume claim,
and also which name is based, you can find the persistent volume claim
and also which storage class a particular volume has been provisioned from.
And that will allow you to build an aggregated view of that correlates
with what is that running on the cluster and what you have on your backend.
In this demo, we’ll also deploy Prometheus and Grafana from scratch
to kind of give you a comprehensive walkthrough on how to get started
with all these different components.
I’m switching over to my terminal.
And I’m just going to show what we got running here.
So this is a fairly standard Kubernetes cluster.
I have a workload running that generates IO.
I also have the HPE CSI driver already installed,
so we’re not going to go through with those steps.
I’m also connected to a few helm repositories
and we are going to create a new namespace
where we’re going to install Grafana and Prometheus.
I have a set of values files for their kind of preconfigured
all the different components such as Prometheus has its values file,
Grafana has its values files.
And we also have, for each of the exporter,
for each of the backends, they also have a separate values file.
So we’re going to start by installing Promethues.
I also have a list of systems in a simple text file here.
And these kind of refer to the different backend host names that I have.
So I have three Nimbles and one Primera in my environment.
And they also need a secret.
That secret is very similar to the HPE CSI Drive for Kubernetes Secret.
It used to have the backend IP address to use a name and password and such, right?
And for deploying, each of these backends are really exporters.
I’ll just write a simple script that will go through the list of systems.
It will install the helm chart for each of the HPE Array Exporters.
And the particular version we’re using and what name and space, we want to drop it in
and use that exporter configuration file.
The configuration file contains a reference to which Secret we want to use
and other information that needs to be filled out.
Once the exporters are installed,
we also want to install the HPE CSI Info Metrics Provider
and you will install that in the HPE storage namespaces well
and it also has a specific configuration file associated with it.
At this point, we should be able to visit the Prometheus Web UI.
We can type HPE in the query box
and we should be able to see some metrics being gathered by Prometheus.
A switch into the graph view is a little bit easier to see what we’re getting here.
But at the end of the day, here we see
the read throughput of a bunch of Nimble volumes.
And we can also see that
each specific metric has a very rich list of labels that are associated with it.
We can see which array it’s running on.
You can also see which pool the volume is running in
and also the volume name obviously
and also the PVC name is the volume underscore name
that we can see in the label list there.
But we’re mainly going to use Grafana to visualize the data in this demo.
So let’s go back to the terminal and install Grafana.
Once Grafana is installed,
you’re going to have to use a command to scrape out the password
and I’ve done that from a different terminal window.
But once you’ve done that, you can switch over to the Grafana UI
and input the admin username and the password that we just retrieved.
The first step when we log into Grafana is that
we need to connect to the Prometheus database.
And we do that by connecting to a data source.
We select Prometheus.
We can give it a name if we want to, but I’m just going to keep the default.
And then we give the endpoints of helm chart we just deployed with Prometheus
which is my-prometheus-server.
We click Save & test and we can see that the data source is working.
It’s been updated.
And next I want to configure our notification channel
because I want to retrieve emails and all my alerts.
So let’s give the notification channel a name
and we also need to give an email address or multiple email addresses.
I’m also going to configure this as the default notification channel.
And now, skip the Test, I’m just going to Save it.
Next, I’m going to import a panel that I have preconfigured.
So I’m just going to paste that in the Import JSON window here
and I’m going to load that up.
This is basically a latency monitor of the backend volumes
of all the volumes running on the Kubernetes cluster, right?
So you will be able to see the read and write latency
from the backend volumes,
but it will also see the Kubernetes-related information
in the metric here as well, right?
So you see that’s a PVC.
And Kubernetes name of the volume
and which namespace it’s currently running in.
If we go in and edit panel, we will be able to see the detail queries,
so there’s one series for the writes and there’s one series for the reads.
We’re also going to inspect the alert.
So the alert is going to kick off once the latency goes over a hundred milliseconds.
And then I’m going to go back and configure
to make sure that the alert becomes active.
And once the alert become active, I’m going to ramp up my workload
and put some pressure on the system
and see if I can induce a latency spike and receive a notification.
Switching back to my terminal,
I’m going to scale my Primera workload,
add more replicas,
and this workload is running iozone repeatedly on the backend volume.
So we will probably see as this workload will scale up,
we’ll see additional latency on the backend.
I’m going to speed up this part of the demo.
And we can see that once the heart turns green,
we’ve got an event that the alert is working,
the queries are being run against the database.
And as more latency gets induced,
we get a alert pending by the heart turning yellow
and we can see the event in the graph there.
It does go back to normal again, so we don’t get an alert.
We will wait for another spike.
There we go, it’s now pending.
And if it stays in that state for the allotted time,
we will be able to see that the alert hits and we get a broken heart
and the event is a red marker in the graph.
We can also see what particular metrics are being hit in the graph here,
what particular volume, what namespace it’s in,
which backend system the volume is running on.
We should at this point receive a notification by email
and you will see what metrics that crossed the threshold.
You’ll also get a image that has been rendered
from the dashboard attached to the email.
And you will be able to view your alert rules
or go to the alerts page to manage the alert itself.
And this concludes this demo.
This has been a very simple example on a very simple graph on how to get started
with the HPE Storage Array Exporter and HPE CSI Info Metrics Provider
for Kubernetes, and how to monitor your workloads and set up alerts with Grafana.
To learn more about these technologies,
visit hpe.com/storage/containers,
and I hope to see you soon.
Take care.