[upbeat music]
>>THWACKcamp.
>>Today, I would like to talk about,
bigger SolarWinds deployments.
You know, like the enterprises to global players,
companies who use various departments around the globe
and who use many, many applications in the background,
who need to be consolidated, right?
And I was like, yeah,
I could probably talk about that just on my own for an hour,
but that would be a bit boring.
So I was like, yeah, I'm in Germany
and I have an expert sitting here in Germany
who is really, really experienced with SolarWinds
and he brought two or three stories with him today.
Now, here he is, Holger Mundt. Holger,
who are you exactly?
>>Hey Sascha, as you might already know me,
because we both go along like for six years now,
but the audience might not know me.
I'm the owner of I.Tresor, here in Germany,
SolarWinds implementation partner,
I'm doing SolarWinds for nine years now.
So the next year will be my 10th SolarWinds anniversary.
So the story was all the way back to,
where I even started as a SolarWinds consultant.
So, the SolarWinds guys came to us and said,
"Hey, we got this big project here in Germany."
There's a customer who wants to switch from like say,
Kiwi CatTools, that was the original outcome,
and they want to have more bigger environment.
So that's where they came to us
and they said, "Hey, can you help us with that?"
And they need some scripting
and they need some stuff.
So, yeah, like I said,
they started out with Kiwi CatTools
and they had approximately 15,000 devices
that were backed up there in their Kiwi CatTools
and the need for it was like,
they needed some granular authentication
and authorization and auditing methods,
and Kiwi wasn't able to do that though.
So they had to look at the NCM then,
and yeah, that was the solution,
the more mature SolarWinds tool.
They wanted to stick with SolarWinds
and that's where they came up with,
hey, let's have a look at NCM.
>>So if I'm not mistaken,
I think at some point I was involved
a little bit with that project
that was like a financial sector,
but that's all we can say about this.
They had a mix of different network devices, right?
So not just one particular window in depth.
>>That's great yes, yes. >>Okay.
>>They were like heterogeneous networks with,
I think they had back in the days,
they had some Brocades, they had some Cisco,
some Junipers, and some other vendors that,
well, didn't play a big role, but they still had them.
>>Do you remember in the early days
how the network looked like?
So it was heavily distributed obviously
because of all of the branches,
but one data center or how did it look like?
>>They had multiple data centers,
but the NCM instance, and back in the days,
also the Kiwi instance was in one data center
and it was mirrored to a second one
with a fail-over then they had the virtualization fail-over
for their systems there.
>>Okay, okay.
So it was a big Kiwi instance already, yeah?
>>It was a big Kiwi already, yeah.
>>Okay, all right,
so then you arrived on the scene with NCM.
>>That is correct, yes we arrived on the scene
with NCM and the customer also set a new goal.
So they had those 15,000 devices
in the Kiwi, and they said well,
we're growing and we want to manage more.
So we need, now, 30,000 devices,
within the next three years, the situation
would be, yeah, that they would have
that many devices in their systems.
So we had to look at the SolarWinds guidelines
and they said, well,
you should put 10,000 devices per polling engine
and you should have three polling engines per instance.
What a coincidence that were exactly those 30,000 devices
that we wanted to manage.
So we had a single NCM instance that was,
then capable of handling all those nodes and devices.
>>So you basically maxed out the system.
>>We maxed out the system.
We even had a couple of times
where we went over the suggested 30,000 devices.
We, top times, I think we had like 45,000 devices on those,
three polling engines.
And we were already thinking well,
hmm so let's see what will happen.
But it went okay.
So the guidelines that SolarWinds gave us were pretty,
well-sized that you don't overdo it.
You can overdo it a little bit,
but still don't look too deep into overdoing that.
>>Well, between 30,000 and 45,000
there's a minor difference.
>>There is a minor difference,
but yeah, we were stretching the goals,
we were stretching, yeah.
>>Okay, how did the deployment look like?
>>The deployment was a main polling engine
and two additional polling engines
and we had a separate SQL database,
even though the guidelines said,
you should look for a hardware SQL instance.
We went with a virtualized environment,
but gave it a lot of resources
and a dedicated virtualization infrastructure for the SQL.
So I think that was one of the key functions
where the system could do more than suggested.
>>And I remember that there was a little bit discussions
about they check the market right,
to see what else is out there.
>>What we had at first running,
so it was the whole system was kind of automated.
So we looked into a CMDB
and we checked the nodes from the CMDB
and imported them into the SolarWinds instance.
And from that point,
so we had approximate daily changes
of 400 to 500 devices per day
that needed to be automated.
The downside where we could do it with native SolarWinds,
GUI stuff was we were not allowed
to scan the whole network.
So we had a fixed database in the CMDB,
and we had to get that into the SolarWinds database.
So that was something where we developed a script for that
in order to dock onto the CMDB,
process all the information, because the CMDB,
usually it should be fine,
but there were minor glitches in the data quality.
So the script did a lot of checking and so on,
and then it imported to NCM.
>>So, wait, you said there was like 400 changes a day.
>>Yes, so they were putting up systems,
taking systems down, moving systems,
so those 400 changes are like new nodes,
deleted nodes, and changed nodes.
So that was putting it all together.
That was the daily work that the script had to do.
>>So then they tried to unify their hardware,
if I'm not mistaken right?
>>Correct, yes there was the key
to have a general vendor change
and consolidate everything into a single vendor.
So with that they also were looking into the vendors' tools
for managing the configuration of those devices.
So they thought that well, if we change everything,
we have a single vendor and we can use the vendor's tool,
they probably give it to us with the hardware for free
or for a very low amount of money.
So that's where they were aiming at.
But still they had that other vendors
as legacy hardware.
So your switches, your routers,
you don't change them on a yearly basis.
When they run, they run.
You make updates, you check the configuration,
but you don't change the hardware.
>>Okay.
So you, more or less, convinced them to stay with NCM.
[chuckling]
>>Yeah, so the other hardware vendor
they came in and said, well, we're able to do,
or we're confident that we will be able to do what NCM does.
And they had two years
of time in order to prove that,
but they weren't able to do everything.
So now some stuff is done with the vendor's tool.
For example, there's stuff that NCM
is just not able to do.
But there's a lot of stuff that NCM can do,
and there's more stuff that NCM can do
than the hardware vendor's tools.
So they use both tools for it.
So we work nicely together.
>>Okay, scalability.
You changed quite a few things now
and you expect growth, right?
>>That's correct, we don't expect growth,
we already see growth.
So, we already scaled out the system.
So the customer was gaining traction
and with the new vendor coming in,
new machines coming in, new nodes coming in,
we found that, well, our single NCM instance
isn't gonna cope with it anymore
because from the 45,000 that we stretched earlier,
they were like aiming for 90,000.
So, well, that's a lot more than that.
Yeah, that's correct.
Yeah, you're already counting and you're perfectly right.
We have three NCM instances now,
because each one supposedly,
should handle 30,000 even though we know it does more.
But we wanted to keep in the SolarWinds
suggestions and guidelines
in case we need some help, and no one can blame us for,
hey, you're overdoing it, so-
>>That's right.
That was actually one question,
which I have in my mind now.
At the peak, you had like 45,000 with one instance,
which is way over what we would usually say is working fine.
Was there an instance where you had to reach out to support?
And if yes, what did they tell you?
>>Yes, there was an instance and they said,
"Hey, you're doing a lot with that system,
but it doesn't look like that
it's the number of nodes that's affecting it."
So it was, the support was still helpful.
And they said, "Hey, that's okay,
you're doing cool stuff.
We wanna see you, how we can fix your issue."
And they did.
>>And Holger, I remember you mentioned,
like companies on a merger spree,
I just know shopping spree, so what was a merger spree?
[laughing]
>>The merger spree was a long-time SolarWinds customer
who acquired multiple other companies.
And those companies also had
their SolarWinds instances there.
So they were in the same field
and they were using same tools.
So we were consulting the customer and said,
"Hey, let's make the most out of it."
There were licenses on the one subsidiary.
They had like, SAM, NCM, NPM.
The others had VNQM, UDT,
so we had a whole mix of licenses.
So we said, let's go ahead
and lets consolidate those together.
We were working with SolarWinds sales there,
and they made a compelling offer to the customers
so they could merge everything together.
And then, we had still the three instances
that we wanted, or the customer wanted to have,
into a single, centralized platform
and a single, centralized monitoring solution.
So we went on and we said like yeah,
let's have one single instance to rule them all.
And that's where we started to consolidate them.
So at that time I was a bit afraid,
so, oh my God, how are we gonna do that?
And I'm not only contributing to THWACK,
I also pull a lot of information out of THWACK.
So that's where I was like,
let's have a look on THWACK, did anyone else do it?
And I found this little gem right here.
And yeah, there is one migrate nodes,
volumes, interfaces to the other database,
or to another database.
And there was a question
and here Steve answered and said,
"Hey, have a look at GitHub"
and I said what's GitHub and what's the OrionSDK?
So, well, here we go.
There's the OrionSDK,
and we can download it here,
and here's some examples.
So that's where I basically started my consolidation work
and I utilize the SDK, wrote a lot of scripts.
So here's something that was a bit challenging for me,
the WMI nodes.
That's something that really took
a while for me to understand
how to move WMI data from one instance
to the centralized instance then.
So what we came up with is,
the multiple instances also had
multiple custom properties, multiple stuff.
So we had to move everything together.
So that was, well, that was a challenge.
So it took a while and it was like
four or five days of scripting
and we were able to do it,
and everything was in one instance then.
>>Okay, so obviously you can't migrate everything,
but most stuff probably would work with the scripts.
Write nodes, get a new idea anyway,
that's pretty much random.
So that should work out fine.
And the user accounts of the platform,
what did you do there?
Because I would imagine that as different entities
and everyone like "This is mine, you're not touching it."
Right, so politics like that.
>>Correct, correct, as you can see here
on the account limitations, what we did.
So we basically set a custom property
for each instance that this node was coming from.
And we were putting those nodes into groups
and then set an account limitation
that only those users can look into that group.
So that was the beginning,
overall as they were merging closer together,
we were able to release that account limitations
so that everyone could see the whole big picture.
So that was it, but it was a long way into the project.
So it was running I think, two years,
and then they basically said,
"Okay, we now are one team,
and everyone, we can show everyone everything."
>>Okay, so everyone is happy.
>>Everyone's happy, yeah.
So the benefits for everyone were,
they now have the full AppStack picture,
as you can see here.
The Mean Time To Innocence, for example.
So that's a phrase that I really like
that SolarWinds has put up,
that the Mean Time To Innocence is...
Yeah, they were able to get the point
where the actual problem was,
much quicker than they did before
because they have the whole,
SolarWinds Orion Platform modules and the whole AppStack.
That was something that really helped them a lot.
>>Okay, topic number three, enterprise collaboration.
And I remember you were talking of a very messy place.
>>It really was messy.
So each team had their own siloed approach,
they were using their own tools, nothing was integrated,
and yeah, in the event of some failure,
they were all saying,
"Hey, we look at our stuff, it's looking good.
We don't know what's there,
so we just gonna move the ticket to somewhere else."
And they said the same, "Well, our stuff seems to look fine.
So we gonna toss it to the next one."
And there was a pre-deployment phase
where I went onsite to the customer
and I basically had each team
for half a day in a room together and said,
"Hey, why don't you monitor internally
with your tools?
Can we do that in a once non-siloed approach,
a global approach?"
So they were like,
it was a show-and-tell workshop with them.
And on the last day we had all teams,
one member of that team together in the room,
and we agreed, hey, we think SolarWinds
is a good approach to that, and yeah.
So we said, we gonna need some guys
at your place who know this stuff.
And so they nominated two admins,
who were to be the SolarWinds Champions in that company.
And so the other teams, they then voiced their demands
and their needs to those guys,
and over time, so to say, they became like masters.
So the Padawan becomes the master.
And one of the guys is already,
posting on THWACK and he's correcting
templates in THWACK and that's pretty cool.
So yeah, the Padawan has become a master
and overall, the whole experience
of the monitoring solution for the teams was,
again, Mean Time To Innocence, I had that phrase
in the last section.
It was reduced.
So the saying, "It's not my problem"
was then really quickly resolved.
Well yes, it is your problem because see here in AppStack,
yours is showing red.
And yeah, so it was more efficient
in spotting the actual issues
and also they have customers of their own.
So they're supporting other customers.
So they could, now, really easily meet their SLAs
for their customers because they were seeing the issue,
fixing the issue, and you're done.
>>Okay, well, that was three really cool stories
which we're at today, right?
Thanks Holger for bringing these nice stories
and up for a beer now.
>>Up for a beer, thank you for having me
and enjoy the rest of the THWACKcamp.