(mellow music)
- The teams within the Triad:
Change, Incident, and Problem,
all work closely together on a daily basis.
And, certainly, within the moment
for any particular thing that might be happening.
- That Triad's a huge part of our operational success
here at Rackspace.
Certainly, we never wanna have an incident,
but when you're dealing with 24/7/365,
infrastructure issues are going to occur.
And when they do,
we certainly wanna track that, accordingly.
- One of the things that we pride ourselves on
in this organization is
our operational discipline.
We believe that our customers rely on it,
and depend on that discipline.
That discipline is what creates the consistent experience
that our customers get from us.
- Operational discipline is looking at
the policies, the processes and the procedures
that we are involved with,
and constantly monitoring and improving them
to make sure that we are aligning with the business needs.
There are several different types of Change Managements.
What we deal with is really
more on the technical aspect of changes.
Anytime a change is happening
to a production environment
that has a potential to impact
two or more customers,
it has to follow the Change Management process
as defined here within Rackspace.
The Change Request starts off with an individual making
a request for a change.
Once that request has been made,
it is then transferred over to a Change Sponsor
to make the correct assessment on that change.
It then moves on, if necessary, to the Change Board
to then review and provide additional guidance
based on their experience on previous maintenances
that have come before the board.
There's various reasons as to why
something would go to the board.
Now, typically we align based off of RPN.
The RPN score, it hits 65 or above,
it's an automatic, "We're gonna take this to the Board."
- An Incident's any event that affects multiple Rackers,
multiple customers, or both.
It could be a tool that breaks,
a service or product that fails to deliver
as customers would expect.
The Incident Management Team,
their role in an incident is to coordinate and communicate.
So, coordinate response, you know,
for the various technical teams
and the support teams that may be involved
in a particular situation,
and also communicate.
Not only externally to customers as it may be appropriate,
but also internally to Rackers to make sure
there is proper awareness of the situation that's going on.
We use ServiceNow to document incidents internally
within the IM team,
and we use that tool to communicate incidents
out to Rackers via what's popularly known as the IM blog.
- So in Problem Management,
we talk a lot about ownership,
on many levels.
We want the teams that have either been involved
in the incident or have created the incident
to really own their part in it.
So, Problem Sponsors which are directors and above,
will typically own the RCA,
make sure that it gets conducted,
and make sure that action item owners
are assigned to get these things taken care of.
- When a problem occurs, whether that's an incident,
maintenance that was not prepped properly,
or an issue with technical equipment,
Rackers have this root cause analysis methodology
to take them through a deep dive investigation
in which technical issues are uncovered,
escalation and communication issues are uncovered,
monitoring issues are uncovered,
and there are a lot of brainstorming tools
as a part of that process,
to come up with these ideas,
and come up with good actions.
I hate to be in an RCA and hear an action
that says have a meeting about a thing.
I want my RCA action items, as much as possible,
to be things that are achievable
and that move the needle forward.
- From a customer perspective,
many, many times, right,
customers, when they come in and see how we do it,
the rigorous processes that we use,
our Triad program in general,
we're able to close that account,
each and every time.
They really appreciate the operational rigor
that we bring to everything that we do.
(techno music)