This video demonstrates the setup process for Site Reliability Operations, or SR Ops,
and Site Reliability Metrics, or SRM.
You can skip to these topics from the YouTube description for this video.
Let’s run through the SR Ops setup process.
We start by performing the one-time quick-start setup for teams.
For the first step, your administrator will need to install the Manage Team Flows update
set, and perform a one-time activation of the team catalog item.
Here in the ServiceNow store on the Site Reliability Operations page, we download the Manage Teams
Flows xml file.
Logged in as admin, we import the update set into our instance.
We start by elevating our privileges to the security admin role,…
…then navigate to Retrieved Update Sets to upload the file.
We preview the update set, …
…then commit it.
The next step is to activate the team catalog item.
We navigate to the catalog items list and search for Create new team.
We then select the Site Reliability Operations scope in the application picker,…
…and copy the catalog item.
In the copied catalog item, we click the Process Engine tab and select Global – Create New
SRO Team in the Flow field.
We rename the file, set it to active, and save it.
Now we need to copy the sys_id of the catalog item.
For the last step, we navigate to the SR Ops system properties,…
…and paste the sys_id into the Catalog Item sys_id field.
We also need to change the Update team subflow name property to this global value.
Now let’s set up our first SR Ops team.
Logged in as the SRO Administrator, we access the Site Reliability Ops Workspace.
In our example we have no teams set up yet.
To add a new team, we can either click the add icon under My Teams,…
…or access the Teams tab and then click Create New Team.
In the form we enter a name for the team, add team members, and submit the request.
We receive an email notification when our request has been approved, and back in the
SRO Workspace our new team displays.
Now we’re ready to set up on-call schedules and escalation policies.
Let’s create a new on-call schedule for the day shift.
We’ll use the existing workday eight to five schedule, …
… and then add another shift to cover nighttime hours.
Now we’re ready for the next step: adding team members to the shifts.
We add members to the primary and secondary day shifts,…
… and primary and secondary night shifts.
For the next step, we set up the escalation policies.
By default, this escalation type rotates through the rosters: primary, secondary, and catch-all.
For the catch-all, we can select who gets notified.
After defining escalation for both the night and day shifts,…
…we’re ready for the final step: review and publish.
If you need to edit a schedule, access it in the Shifts tab.
Back in the SRO Workspace, a reminder displays for our next on-call shift.
Now we’ll create and configure our services.
In the Services tab, we add a service.
We can map to an existing service, or create a new one.
Here we create a new service to support our new community clinic, and select our new team
as the support team.
To add a child service to the parent service, we click the icon on the service box,…
…and save the relationship.
Here are the new services, and now we can edit their properties.
We add a version, and select the environment.
We’re ready to integrate our service with an APM tool and create alert rules.
We’ll set up an integration with Azure.
We want this integration to create an incident.
We can also trigger a notification on either the alert or incident record.
And we can see the alert rules here.
Note that you can also create additional alert rules on the integration.
You select conditions from the available options in the reference picker.
To create a custom condition, first we define a criteria or script based condition.
For our example, we want the alert rule triggered when the severity is critical.
Now we can select the new condition from the Conditions list, then select an action to
take when the condition is met.
If you need alerts at the service level, you can add them here in the Alert Rules tab.
You have the option of creating an alert rule to specify what action to take when the alert
is created on the service directly, …
…or a CI under the service,… …or both.
Like the alert rules on an integration, you can select existing conditions or create custom
conditions and specify the action to take when the rule is triggered.
The Related Alerts tab displays alerts created for CIs that are associated with this service.
Now let’s go back to our Azure integration, and copy the generated Webhook URL so we can
paste it into a new alert that we’re going to create on the Azure portal.
Here’s the app service we need to add the alert to.
We start by adding the new alert, and adding a condition.
We select the signal type that will trigger the alert.
So whenever the virtual machine gets restarted, it will send an alert to the action groups
we specify.
We create a new action group, and select webhook as the action type.
Here’s where we paste the Webhook URL we copied earlier.
And we add authorization credentials to the URL.
You’ll need to set up a user in the instance with the authorization credentials and the
event management integration role.
Now we enable the common alert schema.
Always be sure to enable this option.
Finally, we give the alert a name and save it.
In our example we’ll get an alert any time the virtual machine in Azure restarts.
Now let’s run through the SRM setup process.
We need to establish a service level objective, or SLO, with an error budget.
Back in the SRO Workspace, we create a reliability indicator and define the service level indicator,
or SLI.
We enter a name that indicates what this SLI measures.
For the type, we select Errors because restarts indicate app service errors.
Next we enter the APM properties.
The name needs to match the name of the SLI in our APM tool.
In the Integration field, we select the integration we just created.
We want the maximum number of restarts to be 5 per month, so we specify greater than
5 count.
The last step is to enter the values for our SLO.
Now we can set up an error budget.
We start by adding thresholds to determine when actions against our SLO will occur.
You can set thresholds for the rate at which your error budget is being consumed, or how
much remains.
For our example, we want actions to occur when 20 percent of our error budget remains.
With the threshold selected, we can specify Error Budget Policy based actions, such as
creating an incident, … …sending an MS Teams message, …
…creating an incident and notifying on-call personnel,…
…sending an email, … …and sending a Slack message.
We want an incident created and an email sent, so we select both of these.
Now as alerts occur, we can monitor our SLO in the Details tab and we’ll be notified
when our threshold’s been breached based on our error budget policy.
Now let’s see how to respond to alerts.
Back in the Azure portal, let’s restart the service to test our new alert.
After a few minutes, our alert appears.
Back on the services page in the SRO Workspace, we see that we received the alert.
When we open the alert, we can see when it was created,…
…and the escalation in progress.
Note that you can also set up notifications to post to Microsoft Teams, or an existing
Slack channel.
As a team member receiving an alert, we click Acknowledged to stop the escalation, and Assign
to me.
In the Services page, we can see the active alert and open incident.
Back on the home page, the incident appears under My Work.
To resolve the incident, we can open the alert and set up remediation tasks.
But in our example we already have an incident created, so we can open and work it.
We can also create a Change Request, or choose other actions to resolve the issue.
For more information, see our product documentation, knowledge base, or podcast.
Or ask a question in the ServiceNow Community.