[upbeat music]
>>THWACKcamp
>>When it really comes to any software package
there's normally two ways that you get it.
It's either something that you go and do your research for,
and then you go out and request and purchase,
or it's something that you inherit.
Monitoring systems are no different, and in my history
I inherited the monitoring system from my boss
who basically just one day said here,
this is yours, enjoy.
And we have a special guest, Ben who also inherited.
Ben, how did you end up
getting handed your monitoring solution?
>>So I really got handed it
because the person that was running it, it was a sub task.
It wasn't their main focus day to day.
And the company saw the need to have a dedicated resource
to help massage the monitoring to make sure that
everything was tuned in properly.
>>Okay.
So I came from Big Law.
So multiple servers across multiple geos,
lots and lots of small offices.
What's your infrastructure like?
>>Kind of the same.
We've got some big offices.
And then we got a bunch of little networks
around the country and around the world.
Comes down to a few thousand compute devices.
>>So we basically inherited similar kind of footprints,
I guess.
>>Fairly.
>>All right so we both,
both gone through this inheritance task.
What do you think is the very first thing you need to do?
>>So I think the first thing that you need to do
when you're inheriting the monitoring tool
is to gain that control.
One thing that I ran into was
before, before I stepped in, there was 12
or so people that had admin access to make alerts
make dashboards, edit custom properties, things like that.
That really are the key to a solid monitoring system.
And it's not that they did it incorrectly
but again, having too many hands
in the soup doesn't make, always make for a good soup.
So you want to gain that control
and define where does the buck stop?
>>Yeah.
I ran into a similar thing where it was handed over to me
and I had, I think we had six or seven people as admins
which you think is fine,
but then when people start customizing views
or building new custom properties
where we already had existing ones
or, you know, God forbid duplicating
and triplicating alerts that go to everybody.
You need to have someone, you need
to basically become the CEO of monitoring
at your company. >>And we have the same
and we sort of had the same issue,
you know, people were editing dashboards
not realizing that that was a global dashboard.
So we had network devices that were
trying to pool server information.
So the load time when you would click on the node
to load time for that screen to pop up in your computer,
could be an extra 30, 40, 50, 60 seconds.
And while that time may not seem that critical
I mean, what's 30 seconds, but in the midst of a crash,
in the midst of an issue where
it's all-hands-on-deck type of thing,
30 seconds let's face it can seem like forever.
>>Yeah. Especially when people are breathing down
your neck or worst case scenario
when your email just keeps chirping saying
hey, more is down, more is down, more is down.
Any delay like that is just,
it just slows down the overall process.
>>When I became the monitoring guy,
for lack of a better term, you know,
I really set the rules of engagement forward.
I said, look, this is how we're going to alert.
This is how we're going to do things.
This is how I see it best.
Now. It's not just me saying, and we're doing,
it's me asking for input from my end users, my customers
because quite frankly,
I'm not the one digesting all these alerts.
I'm setting them up and they're getting them.
But when the alert goes off at three o'clock in the morning,
I'm not the one waking up.
>>No >>So I found out
I found out I found it very prudent
to get the buy-in also from my customers,
as far as the rules of engagement were
when it comes to monitoring.
>>You and I had a talk,
it's probably been about six months now.
And we both ran into a problem in our organizations
where we had to pull the switch, flip the trigger.
I don't know how we want to phrase this.
We had to basically make the absolute worst decision ever.
And what did we call?
I think we referred to it as nuking monitoring from orbit.
>>Yep.
>>So, >>Nuking monitoring.
>>So what led you to that decision?
>>So it wasn't an easy decision.
Again, there was nothing wrong with
what the previous admins have done with it,
but with the advancements of the Orion Platform,
even over the last year, year and a half,
it became very obvious that we needed to kind of
back up and refit and go back at it.
So I talked with, I talked with my boss.
I talked to my director.
We talked with key stakeholders
and I said look, we have these Orion products.
And when we first purchased Orion, we bought NPM
and then we bought, then we bolted on SAM.
Then we bolted on, oh my gosh,
NPM,
SCM
all the different tools kind of,
kind of gets bolted on, right?
So what I want to do is kind of approach
in a holistic view and say, okay, here's all the products.
How do we get them really tuned in to make them effective?
And that's when the call came,
yeah we need to do that.
Also look at things like we had over 200
and some alert definitions
and those queries hitting the database, you know
every one to five to 10 minutes.
The performance of my database was struggling so hard
'cause if you mind
each alert, equals a SQL query to the database.
>>Yeah and depending on the number of results
that come back, it could kick off
one or more trigger actions
and then it just keeps escalating.
>>Snowball
>>Yeah.
And the problem is you add, that's a great example
because if you add an alert and it has two actions
and even if it polls once a minute,
it's not that big a deal
on a small to medium-size environment
but that's only one alert.
When you start talking about
you know, 20, 30, 50, a hundred alerts that are all active,
that stuff really kind of adds to the list
and it makes it makes monitoring the monitoring so much
of a priority that you kind of get lost in the shuffle.
So I think >>Yeah, especially--
I think planning in advance, like what alerts do we keep?
What reports do people actually read?
What dashboards do we have to rebuild?
Is really important when you actually
have to make the decision to be like,
that's it, we've got to start over.
We've got to burn it to the ground and
we got to build fresh and new.
>>And that's the one thing that I made
the decision on was I didn't want to carry over anything.
I-- >>Oh you did a full cut.
>>A full just new database
new servers, new everything, because
what I'm trying to gain from this is better utilization.
I can send out all the alerts all day long
but if the application team isn't responsive to them,
what's the point?
Why am I doing all this?
So I took the time to get away
from plain-text emails and make HTML code
because people like pretty,
you know, I cleaned up the database.
I tried to get rid of, I'm trying to get rid
of the nodes as they become decommissioned.
We're trying to get rid of them as fast as possible.
Cause again, one less node
and when you're talking about
having an environment of 14,000-plus nodes,
which isn't that bad,
but then you look at the
element counts of almost a hundred thousand elements,
the numbers start adding up real fast.
>>Okay. Ben, you talked a little bit about
how you had to like rebuild from scratch.
What led you to that decision and
how did you get people on board?
>>So it really, it came to that decision because
during a yearly health check I noticed
we are kind of in the middle.
It wasn't a terrible monitoring system.
It wasn't the best monitoring system.
I looked at the demands that
were being asked of the monitoring system
and I knew that where we were,
was going to carry us much forward.
So I really just walked into my boss's office
I said, look, this is where we are,
these are the demands you're making of me.
We need to get here.
I can get us here
but we need to kind of re-evaluate the monitoring.
And so it became easiest way is just
to pull the plug and start from scratch and
really build out the solution for tomorrow
not for today.
>>I think I had a similar discussion except mine was
I don't think you built this right.
[laughing]
And mine was a little easier sell.
>>Again ours was built fine.
Like it was, it was chugging.
It was doing, it was doing its job.
But again, there's a huge difference between okay and good.
>>Yep.
>>And personally being, being in responsible
for the application, I don't settle for okay
I don't think anyone really wants to settle for okay.
We want to settle for great.
We want to settle for the best.
So that's really where I'm trying to push
this monitoring solution at the company
say, hey, this is where we were.
This is where we're at
and look at the difference.
Where there's return on investment in, in dollars
in our pocket, because it's this
thing go down or whatever that tangible result is,
that's the key win.
>>When you take over monitoring,
whether it's by inheritance or
when you actually do it yourself
you kind of have to, and you mentioned a little bit
we kind of have to prove it a little bit to the environment.
You talked about how you put in
a dozen servers and then you expanded that out
a little bit and you went more
and you're adding more and more
and you have plans to include other people in as well.
But how much of that,
does it ever end?
or does it, it's a constant cycle.
>>You know, I think monitoring is really a constant cycle
'cause as new things get added to your network or
removed from your, from your network
you really need to adjust how you're monitoring that.
You know, whether you're introducing a different kind
of SD-WAN solution or a new type of server new program
you know, the businesses are always moving forward.
And that's the one beauty about IT
is our day to day is never the same right?
Like by the time we're done talking here
technology has gone five or six, four steps ahead of us.
And now the monitoring is what you really need to prove.
So I don't think there's ever a good kick
up and put your feet on the desk type of solution of
monitoring, because in monitoring
you gotta be thinking four or five steps ahead
because if you're not, that next down
that next outage you might've been able to prevent.
>>And a lot of that comes with
actually having touch point meetings.
It's the monitoring engineer actually getting up
from their desk, going into a room or
getting on a call like this and just having a conversation
being like, these are the alerts I want to turn on
and alerts are not monitoring.
monitoring is a completely separate thing than alerting,
but that's a different discussion
but this, this is what we're monitoring.
This is what we'll alert you on.
These are the reports that come out of the box.
What would you like to see?
And then having that kind of meeting on a certain cadence
whether it's monthly, weekly, quarterly whatever,
it's, you kind of have to have those followups
especially with the stakeholders, the one that gave you
if you do, you know, build from scratch again
especially the stakeholders
who gave you the thumbs up in the first part.
>>Yeah. And that's the one thing I'm trying to
get into place now.
And it's kind of, it's a little more
difficult given the current situation we're in
but I'm trying to get those
type of meetings in place right now.
I have them weekly with my manager, my director
but we're looking to expand that to be more key
to have more key stakeholders.
So we're actually looking at forming
what we're calling a monitoring committee.
So it involves our incident management, involves
our help desk, involves the main application owners to say
okay, this is what we can do.
What do you want?
How often do you want it?
And how do you want it delivered?
Because you know, there's a difference
between delivering an alert through an email or
some sort of paging system, or
do you just need a report ran?
Cause not everything has to be an alert.
You can always just do, hey, here's your weekly report
of how many systems rebooted this week
or whatever that might be.
>>We talk a lot, obviously we've we met on THWACK
we've met in person a couple of events and stuff like that.
But THWACK is one of those resources that
I think may be under known
for especially people that inherited.
Cause if you, if you were bought it
if you bought the solution and you were working
with SolarWinds and you spoke with our people
and you read up on us, then you know about it.
But if you were just handed it,
did you know about like for me,
I honestly didn't know if THWACK existed for six months
after I was handed the solution.
>>Same, same boat.
You know, I didn't know about THWACK the first few weeks
maybe a couple, maybe a couple months I was there.
But when I found it, man, let me tell you
and I'm not, this is no seamless plug for THWACK.
Well, let me tell you the, the,
the blinders came off because
now I'm not just the, the monitoring guy.
Yes I am the monitoring guy at my company,
but I have connections to people like you.
And so many other experts that have been doing this
you know, for the collective
you know, collect it all up and add it thousands of years
if you think about it,
and it's a place where you can come with your ideas
and really not not be embarrassed
to ask that question because we all
I mean, I can't think of how many times Kevin
I, I hit you up and said, Hey, I'm having a brain fart.
How do I do this?
And you're like, bro, right here.
I'm like, Oh yeah, tah dah.
'Cause we all have those days
especially when you're sitting in your
you know, you've been working on an alert for eight hours
and it's just not kicking off
the way you want to kick off.
You can look at THWACK and
maybe you look at the content exchange
maybe that alert, you're trying,
you're busting your butt over,
might already be on the content exchange.
>>Someone's already >>Download it,
>>built for you >>import it, and you're done.
>>Yeah >>Yeah
Then you just, you know, make it pretty.
>>Make it pretty.
>>I've always found that interesting, the number of people
and how quickly their skills move
and especially if you're something
like your background is systems and networking
my background systems and networking
and I'm an accidental DBA,
but I had zero problem posting
like really what I felt were elementary DBA questions
on there because I, that wasn't my day job.
But no one ever made me feel less
for asking what I thought was a beginner question.
And I got great resources and stuff like, you know
the Content Exchange and the Alert Lab and the Report Lab
and the SDK forums and just the, just
just the feature requests to be like, oh cool
they're already working on this, great
or no one said this, why has no one found this thing?
I totally want to make a recommendation.
So you build a feature request
and get all of your buddies to vote on it.
And then the product team looks at that one
when they're deciding the next release.
>>Yeah and I think the one good case story
I have personally not to blow my own horn here
but you know I had a problem where
systems were down
for six months.
And so I remember going to THWACK and
I found something on THWACK that was
sorta kind of what I was asking.
Like I wanted something that would just automatically say
okay, this has been down for X amount of days.
Let's take these actions
but I wanted to make it a little more advanced.
I want to actually make it into an alert that would say
hey, I'm killing this node off.
Let's do it.
So, and I think that's where really where
you and I connected really heavily was, you know
this idea came to me literally out of the amusement park.
>>I remember you telling me that
>>I emailed you and said hey Kevin, here's my idea.
By the time I left the amusement park
you already had it through your lab.
We had it rock and rolling
And within what a week?
Two weeks maybe.
>>Yeah I think you had it in production
in two weeks.
>>About yeah.
In a full-blown environment.
So if you think about THWACK
here's, you know, from idea to concept
to actionable item, that was three, maybe three weeks.
>>Yeah.
>>So I think that's where the power of THWACK comes.
That's what I tell people that are coming to monitoring.
The power of Orion is really nested
in the community of THWACK.
>>Yeah and for something like that
I've said it before, it's almost like emotional support.
It's you go there and maybe you're having
you described that one of those just horrible days
you've been working eight hours staring
at this alert and you just can't fathom
why it's not doing the thing you want.
And then maybe you just get that really small
on the grand scheme of things.
It's a very small win
by going to THWACK and finding something.
But that kind of thing
can save your day. >>Yeah
That's one thing you've totally accomplished.
And you're like, that's a check mark off my list.
One less thing I've got to do.
>>Well, one big victory I had recently
was when I was rebuilding my system
I didn't want to have to rebuild all these dashboards
one by one by one.
Went to THWACK, found a script
where I can export all the views
and from the old system and import them into the new system.
Now yeah, there was some legwork I had to do,
like I redefined some custom properties
and things like that.
But to make those minor changes
within the SWQL queries was a lot less time intensive
than having to sit there and rebuild dashboards.
So like, for the most part, most of my users,
when they logged in to the new system,
they really didn't see much of a difference
other than the speed, but the dashboards,
the familiarity they had was there.
And I think that made a transition a little bit smoother.
>>Okay, Ben.
So if you had to tell somebody where to get started today
cause they were just handed a monitoring solution.
What's their first step?
>>Your first step is getting
on THWACK, get part of the community, learn
your craft, become weaponized to answer those questions.
Sign up first the Academies,
start working on your certification.
Start asking those questions, again
get on THWACK and ask whatever question that is.
Don't get caught up in the whole noob status.
We're all noobs at one point in time.
So ask your questions because I've been there.
I know Kevin you've been there as well.
>>Yep.
>>So if you're feeling that, know that you're not alone
ask those questions, and really get up to the community.
You can find me on THWACK @the_ben_keen.
If you have any questions about possibly
going nuclear in your environment
or you just need to vent about something, I just ask
you know, give me 24 hours to get back to you
because you know, day jobs.
>>Yeah. We can't all have the privilege
of having a day job on THWACK, like some people.
>>Right?
>>So if, obviously if you need to reach me, I'm KMSigma
we will take any and all questions.
And honestly we'll probably end up redirecting you
to something that someone else already wrote
because sometimes the original storytellers
are just the best.
Ben, thank you so very much for making time.
It's always a pleasure talking >>Absolutely
with you and I look forward to see you
in person hopefully soon.
>>Hopefully soon.