In this video, I'll introduce you to alerts rules and notification channels in Oracle
Private Cloud Appliance.
I'm logged on to the Grafana dashboard for my Oracle Private Cloud Appliance.
My system name is blurred out for security purposes.
Grafana provides an "Alerting" button on the left side which includes 2 options, Alert
Rules and Notification channels.
Alert rules are defined conditions that are regularly evaluated by Grafana.
When an alert occurs, the Notification channels define how the alert is delivered.
If you wish to configure custom alerts using your own external notification channel, you
must first configure the proxy for Grafana using the Sauron API endpoint.
To do so, log in to the management node that owns the management virtual IP and run the
following command: curl -u admin -XPUT 'https://api./v1/grafana/proxy/config?http-proxy=http://:&https-proxy=http://:'
Enter the host password for the admin user.
You'll see the message, "Grafana proxy config successfully updated!"
When you do this, the grafana pod is restarted so run the command:
kubectl get pods -n sauron until you see the grafana pod is running.
I'll first click on Alert Rules.
I'll scroll down and you can see that a number of alert rules are provided by the factory
build.
You can filter to see rules that are OK, rules that are Not OK, rules that are Alerting and
so forth.
You can also search for rules by keywords.
I'll type endpoint and see only those rules that contain the keyword endpoint are displayed.
Before looking at these more closely, I'll click on How to add an alert.
As it says, "Alerts are added and configured in the Alert Tab of any dashboard graph panel,
letting you build and visualize an alert using existing queries."
So depending upon where you want to add an alert rule, you need to go to that particular
dashboard.
For example, I'll click Dashboards, then Manage.
Under PCA 3.0 Service Advisor, I'll select the Kube Endpoint dashboard.
Scroll down you'll see that we have some Endpoint Alerts already added.
Total Endpoints, Total Endpoints Addresses, Max Addresses Per Endpoint, Endpoints Per
Namespace, and Endpoints Address Not Ready.
These are the same 5 alert rules that we see when selecting Alerting, Alert Rules, and
filtering for Endpoints.
There are a couple different ways to edit or view an alert rule.
First, I can click the gear button for the respective alert rule from this window.
For this demonstration, I'll click the gear button for the Total Endpoints alert.
This displays the 'Edit Panel' view for the associated dashboard, which is the Kube Endpoint
dashboard in this example.
Notice that the Alert tab is selected for this panel.
Another way to edit an alert is to select the alert panel from the associated dashboard.
So I'll click the back arrow to return to the Kube Endpoint dashboard.
Now I can scroll down and select Edit from the drop-down menu for the panel and I'm back
to the Edit Panel view.
I can click the Alert tab to edit or view the alert.
Now I'll demonstrate how to create an alert rule.
As mentioned earlier, "Alerts are added and configured in the Alert Tab of any dashboard
graph panel."
So I'll click the back arrow to return to the Kube Endpoint dashboard.
And from the dashboard, I'll click the button to Add Panel.
Next I'll click Add new panel.
I'll give my panel a title of 'test alert' Now with the Query tab selected, for my query,
I'll enter count(kube_endpoint_created).
I'll click anywhere in an open spot on the window and my query result appears on the
graph.
Hovering my mouse over the line, I see that the value is 39.
Next I'll create an alert for this graph dashboard panel.
I'll click the Alert tab and then click Create Alert.
For the conditions, I'll select the last() function from the list.
And I want an alert to occur immediately, so for the value, I'll select something below
39, or 30 for this example.
My alert appears on the graph almost immediately.
I want to send this to a notification channel, so I'll scroll down and click the plus sign
and select a pre-configured slack channel named linux-virt-pca.
I'll cover how to configure these notification channels later in this video.
I could include a message but I'll leave this empty for this demonstration.
I need to click the Apply button in the top-right corner of the window.
And now I'm back to the Kube Endpoint dashboard with my new panel named 'test alert' appearing
on this dashboard.
I'll click the Save dashboard button and provide a description, then click Save.
My dashboard is saved.
Now if I go back to view the alert rules, I'll remove the endpoint filter and scroll
down.
And at the bottom of the list, I see my new "test alert" alert rule which initially has
an UNKNOWN state.
I'll refresh the screen a few times and the state changes to PENDING.
Waiting a little longer and the state changes to ALERTING.
I'll click on the gear icon to view the Edit Panel and point out a few more things.
The alert rule is evaluated every minute for 5 minutes by default but you can change this.
On the graph, we see where the alerts occurred by the vertical lines.
And as mentioned earlier in this video, "Alerts are added and configured in the Alert Tab
of any dashboard graph panel."
So if I were to select a different visualization panel, Stat for example, the Alert tab is
not available.
You can only created alert rules for dashboards that use a graph visualization panel.
Finally, let's talk about notification channels.
I configured this alert rule to notify a pre-configured slack channel named linux-virt-pca.
When I look at this slack channel, I see that I was notified by the 'test alert' alert.
To configure notification channels, select Notification channels under the Alerting menu.
Here you see a list of the currently defined notification channels for my Oracle Private
Cloud Appliance, including the one I used called linux-virt-pca.
To create a new notification channel, click the New channel button.
Give your new channel a name and select the type.
There are several types available.
For the Slack type, this sends notifications to Slack via Slack Webhooks.
This is beyond the scope of this video but you need to create Slack App which provides
the Webhook.
I've created a Slack App named PCA - Grafana Alerts.
Once I've installed my app, I can get the Webhook by clicking 'Add features and functionality'
and then clicking Incoming Webhooks.
I've actually created Webhooks for 2 Slack channels but the one I'm using in this demonstration
is named linux-virt-pca.
I can copy the Webhook URL from here and paste it into my Grafana Url field for the Slack
incoming webhook url.
You also need to provide a bot token.
This token can be obtained by clicking the OAuth & Permissions menu option for my Slack
app.
From here I can copy the token and paste it into my Grafana Token field for my new notification
channel.
I'll click Back here and then select my linux-virt-pca channel and show some additional information.
This shows that my Webhook Url is already configured.
I'll click Optional Slack settings and scroll down to see the Token is also already configured.
I'll click Notification settings and show that I've checked the boxes to include an
image and send reminders every hour.
I can test my notification channel by clicking Test and Grafana sends a test message to my
Slack channel.
I'll view the test message in Slack to verify that my notification channel is working properly.
This concludes this demonstration.
Thank you for watching.
Please check our additional information on Oracle Private Cloud Appliance in the Oracle
Private Cloud Appliance Documentation or in the Oracle Private Cloud Appliance Learning
Library.