I'm in the process of building a new alert,
which I have named "Critical States."
But I don't have any trigger condition logic yet,
so I'll add that here on the Trigger Condition tab.
The first thing I need to set is the type
of object I'm alerting on.
This determines which tables
and fields you'll have available
for building your condition logic.
In the I want to alert on dropdown
at the top of the list,
you'll see common alert object types
with a few dozen additional object types listed below.
At the very bottom of the list,
you have the option of building a custom SQL query.
Remember, the alert engine only looks at the database.
So, all the alerts are SQL queries.
But the alert editor almost completely eliminates
the need for you having to write those queries yourself.
But still, you do have the option.
The alert object's type defaults to alerting on a node,
which is what I want in this case.
The next section defines the alert scope.
But I'm going to skip that for the moment
and jump down to the actual trigger condition section.
And here's where I'll add my alert logic.
When I click on the plus at the bottom
to add a new condition,
you can see that there are
three different condition types to choose from:
Single Value Comparisons, Double Value Comparisons,
and And/Or Blocks.
Whenever you start a new alert,
you'll have an and/or block already added
at the top of your condition list.
And below that, you'll add your additional conditions.
Double value comparisons are used when you want
to compare one field to another field in the database.
But I don't need to do that right now,
so let me get rid of this condition
and start with my single value comparison condition.
The first field in the condition indicates
which database table the alert engine will query.
Based on the object's type field up above
where I said I want to alert on a node,
the Node table is selected by default.
But if you look at the dropdown list,
you'll see other tables that relate
to the object type you selected,
which opens up more of the database to you
for building more advanced alerts.
Next, I'll choose the specific field to query.
In the dropdown, you'll see a list of recommended fields
at the top, followed by custom properties
and then recommended events.
And then if you scroll all the way down,
you'll see Browse all Fields,
which, again, lets you dive deeper into the database.
Now, I've named my alert Critical States.
And a node being down would be the most critical of states,
so I'll start with that.
So, I'll choose 'Status' for the field to query.
And then I need to choose the expected value
for the status field.
Now, one of the nice features of the alert builder
is that once you choose a field to alert on,
it will actually query the database for you
and show you all of the values
for that field in the database.
So, you don't necessarily have to know
or type in the return value.
You can just select it from the list.
And I'm gonna choose the status of 'Down.'
Now, before I do, though,
notice that there are several other status values listed
in the database.
When you're building your alerts,
you'll need to make sure that you're accounting
for those other possible variables in your alert logic.
For example, rather than setting my condition to alert me
when a node is down,
it might be tempting to alert any time the node status
is anything other than up.
The problem is that that would most likely result
in a lot of false positives,
because you'd be alerting on statuses
that don't necessarily make a lot of sense
to include in a single alert.
So, when you're building your alert logic,
you need to be very specific
in what you want the alert to do.
If you want to alert on multiple statuses,
you should probably include a condition
for each status you want to alert on.
All right, so I have my first condition set.
I'm alerting on any nodes in the database
with a status equal to down.
But I've named my alert critical states,
and right now, I only have a single state.
So, I'm going to add another condition.
I'll add another single value comparison,
and this time, I want to alert
if my CPU utilization is over 90%.
So, I'm going to select 'CPU Load',
and set the return value to '90%'.
But this time, I need to change the condition qualifier
from is equal to to 'is greater than'.
And in fact, I need to change the condition qualifier
for my and/or block as well.
If you look at the alert as it is,
I'd only be alerting when the node is down
and has a CPU utilization above 90%.
That's not going to work.
So, let me change this from an "and" statement
to an "or" statement.
Now, if either of these conditions are met,
I'll have an alert event.
Make sure you pay attention to your condition qualifiers.
But I've decided I don't want to look
at CPU utilization all by itself.
I want to look at memory utilization as well.
I want to alert when both CPU and memory utilization
are above 90%.
So, I'll add another condition.
Now, the first challenge with this condition
is going to be making sure
that I have the right field selected.
Notice, there's a Memory Used field
and a Percent Memory Used field.
Watch out for fields with similar names.
I want Percent Memory Used,
and I'll set my return value to 'greater than 90%'.
But now I have a problem.
I want to alert when both my CPU
and memory utilizations are above 90%.
And right now, I'm alerting
when either of them are over 90%.
So, I need to add a condition.
I need to add an and/or block,
and I need to move my CPU
and memory conditions into that block.
You can move conditions around by drag-and-drop.
Now, you can see that my conditions
for CPU and memory are nested within the and/or block.
And I'm going to just delete this condition
that was added by default,
because I don't need it.
Now, notice that when I added the and/or block
that it was already set to an "and" statement.
So, how will this alert be processed now?
Well, starting with the first condition,
the alert engine is looking for any nodes
with a status equal to down.
But now, how does it process the and/or block?
And/or blocks are evaluated as a single condition.
So, the question is,
does this and/or block evaluate to true?
Are there any nodes with a CPU utilization over 90%
and memory utilization also over 90%?
If so, both conditions of and/or block would be met,
which means the entire and/or block would evaluate to true,
so I would have an alert event.
The three conditions that I've added so far
are all looking at current values in the database.
But you can also alert on database events.
I'm going to add one more condition,
but this time, I'm going to scroll down
a little further in the list before recommended fields
to the recommended events section.
And I'm going to choose 'Last Boot has changed.'
This will alert me if the node has been rebooted.
But notice that alerting on events looks a little different
than alerting on field values.
I'm not asked for an expected return value.
Based on the event type you've selected,
the alert editor sets the logic of the condition for you.
All right, so let's run through my alert one more time.
First of all, remember my initial and/or block qualifier
is set to "or."
So, if any of the conditions are met,
I'll have an alert event.
So, condition one, check to see if any nodes are down.
Condition two is another and/or block,
so this is evaluated as a single condition.
For this to evaluate to true based on the qualifier
for this condition group,
both conditions within the group have to be met.
So, this is asking if there are any nodes
with both CPU utilization
and memory utilization above 90%.
And finally, condition three, have any nodes been rebooted?
If any of these three conditions is met,
I'd have an alert event.
So, here are my three critical states I'm alerting on:
a node is down, CPU and utilization are both high,
or a node has been rebooted.
Now, I'm okay with my alert logic now.
But I'm not okay with the alert scope.
Remember the scoping section up above
that I skipped earlier?
Notice, right now I'm alerting on every node in the database.
You will almost certainly wanna scope your alerts
based on who needs to see that alert.
If I'm an application owner,
and I'm only responsible for a single server
or a small group of servers,
I don't want to see alerts
on every node in the organization.
So, I'm going to change my scope from all objects
to only the following sets of objects.
And I'm going to choose just one of my servers.
So, I'll select the 'IP_Address' field
and choose my server IP.
So now, I'm only alerting on a single IP address.
I can also select a range of nodes.
Maybe I have a specific DNS naming convention
for all my nodes.
For instance, all of my servers
are called CTLab something or other.
So, I can add another condition
and look for DNS names starting with CTLab.
But again, I need to pay attention now
to my and/or block qualifier.
All right, my alert logic is set.
I know what I'm alerting on,
and I know what conditions are going
to trigger an alert event.
So now, I'm ready to move on.