[bell rings]
>>Traxo is a data aggregation service
for the travel industry.
We work with suppliers, corporate travel managers,
corporations, and also a big broad ecosystem
in the travel industry.
Because it is so fragmented, the need for real time
data aggregation and itinerary management
is ever more evident.
All those players in the travel ecosystem
have a need for real time data
that's very accurate and timely.
We're a multi-region, multi-zone
AWS deployment, fairly typical.
We tend to run a number of similar hosts
managed by a cluster scheduler
called Nomad from HashiCorp.
We also run a number of different tools from HashiCorp
including The Vault for secrets management
and Consul for service discovery
and configuration.
In that cluster we run a number
of different containerized workloads,
primarily containerized workloads on Docker.
Most of our services are micro-service, multi-tier,
and what we're monitoring is everything from the host
to the container to the application
and also ingress and egress
through the various load balancers.
First it was a internal StatsD Damon and Graphite,
so really popular at the time, five or six years ago now.
That was great, it worked great.
A little hard to manage,
and then as more SAS based solutions
started to appear,
we naturally wanted to gravitate towards those.
So we quickly found Librato, which was really great.
It supported StatsD, it was hosted outside our data center,
so we could access it.
Even if we were down, we could at least
see what the last bits of data were.
The original StatsD tagging system
cannot break down the more you try to capture,
the more segments of a particular metric you try to capture.
And it also made it really hard to write good aggregates
and also see on a dashboard what is US doing right now,
what is Europe doing right now,
segment by host, that kind of thing.
The incorporation of Librato
into the bigger AppOptics ecosystem was really easy.
So we looked at a number of different tools,
both above and below price point
and also some self hosted solutions,
again just to double check that we weren't missing anything
that we wanted to run ourself.
The conversion was gonna be minimal,
so just point to a new host
and start taking advantage of the tag metrics.
We also got the added benefit of the APM ecosystem
and since rolling that out,
we feel like we've got quite a bit better visibility
into the application performance
as well as the lower level host performance
that's going on in the system.
There's a couple of different use cases.
So there is proactive monitoring,
so hosts are up, how much capacity's
being used in a cluster?
What are the applications doing?
Are individual aspects of the business,
KPI's, what are those?
Everybody in the team's aware.
Support for deployments and rollouts
so engineers can monitor a set of metrics
that they expect to stay the same, expect to change,
and as those deployments finish in various regions,
they can cross check that
with what they're expecting to see.
The alerting is incredibly powerful.
We have alerts pushed out through Slack,
through PagerDuty.
Click and configure and everybody was up and running.
Folks can see if the app is misbehaving,
is it a particular host,
is it a section of hosts maybe
in an availability zone that's having problems?
We do have a correlation ID
that traces all the way in from the edge service
into one or many back end services.
Tracing is incredibly important
and that's really a new capability for our team.
We've found a number of different hotspots that way
including third party services that we rely on
that we're able to tell them when they're slow
or having problems.
For any 24 hour, seven day a week business,
being able to know when things are going wrong
is really critical.
We like to know when things are going right
and that's great, but when things are going wrong,
we expect to be woken up or alerted in some way
and that was something we knew we weren't gonna build
so the solution we picked with AppOptics
definitely had that and again,
had all the integrations for pushing data
to the places we wanted to see it
and those were really well done.
We had all the visibility we needed,
we could get ahold of the folks any time of day
and resolve those issues as fast as possible.
I'm able to do a better job
of supporting our customers in real time.
[music]