PRIYANKA VERGADIA: Well today I'm
invited by xMatters in their kitchen,
and Travis DePuy is going to walk me
through how they've been through this journey of migrating
from monolithic application into microservices on Google Cloud
Platform.
[MUSIC PLAYING]
Thank you, Travis, for having me over.
TRAVIS DEPUY: Thanks for coming.
PRIYANKA VERGADIA: Awesome.
Well, I'm super excited to actually cook with you today
and to eat, because that's what I like to do.
But before we do that, I would love
to learn more about xMatters and what the tool does.
TRAVIS DEPUY: We help keep digital services up
and running.
We do that by tracking down the current on call resources,
delivering notifications to them,
and letting them initiate workflow.
PRIYANKA VERGADIA: So what the tool basically does is,
it takes the times when there's spikes in traffic or spikes
in error rates, and then it notifies the right teams.
But that stuff is really hard to do.
How do you understand all of that--
and also very complicated?
TRAVIS DEPUY: Yeah.
Yeah, it is hard, and important critical work
for the industries today.
And there's, once you get the notifications, you know,
that's just the beginning of the process.
There's a lot steps that have to happen after that.
And so that's called toil.
You know, important manual work that needs to be done,
but it takes away time from troubleshooting the issue,
so we can help automate a lot of that process.
PRIYANKA VERGADIA: This sounds, obviously, very exciting.
I'm interested in seeing the tool.
Can you show me how this works?
TRAVIS DEPUY: Yeah, sure.
I have a computer right here.
PRIYANKA VERGADIA: Awesome.
TRAVIS DEPUY: Here we see part of the xMatters flow designer
canvas showing a Stackdriver alarm workflow.
The first step is the inbound HDB trigger
that parses the payload and kicks things off.
Then we start enriching the alarm with some metadata
that Stackdriver might not have.
In this case, we're getting the last commit and deployment info
from GitHub and Jenkins.
And finally, this is where we actually
fire the xMatters event to notify people.
Here we see Dan receiving the Stackdriver notification
on his mobile device.
He reviews the stock info from Stackdriver,
but then he can review the enriched
from GitHub and Jenkins.
He has a couple of different response options,
but he decided this is severe enough
that he needs to roll back to the blue deployment
from the current green deployment.
So he selects a response option and adds an optional comment.
Back in the xMatters campus, we see the various flows
that kick off when a user selects the relevant response
option.
Each organization has major incidents and processes
around kicking those off.
In this case, we see that xMatters
will automate creating a juror issue, and then a status page
incident, and finally, a Slack channel so that teams
can start collaborating.
We also see a rollback to blue that Dan selected,
and we see that a new juror issue
is created before triggering the redeploy using a Kubernetes
command.
The juror issue helps keep track of who triggered the deployment
and when, and also serves as a record
for future investigation.
The last one is my favorite, though.
Using a playbook tool such as Ansible,
we can go trigger an existing playbook
to attempt to remediate the problem.
There are limitless possibilities,
and all this can be tailored in the specific processes
of the organization.
PRIYANKA VERGADIA: I can imagine how this application must
have looked originally.
So I am super excited to learn how you guys managed
to take all these services and separate them out and create
microservices out of them.
But before we do that, I want to know what we're cooking today.
TRAVIS DEPUY: We're cooking vegetable soup,
and I think it's going to be pretty tasty.
PRIYANKA VERGADIA: OK, Travis, so where do we start?
I am super excited looking at all these veggies.
TRAVIS DEPUY: Yeah.
We're going to chop them all up.
So we've got some celery, some parsley.
We've got some carrots and some cheese.
PRIYANKA VERGADIA: Corns and colorful tomatoes?
Oh, good start.
OK.
So Travis, what do you think about the difference
between monoliths and microservices?
How do you define it?
TRAVIS DEPUY: Well, if you think of a monolith like a turducken,
you know, that meat monstrosity that's
the chicken inside of the duck inside of the turkey?
A lot like a turducken, monoliths are very intertwined.
Everything's all put together, and rolling out new turduckens
is very hard.
You have to rebuild each one.
PRIYANKA VERGADIA: Just like when
you're trying to build something and you
have all the things connected to each other,
and they're dependent on each other,
and you don't know which one to take apart,
because the other things might mess up.
TRAVIS DEPUY: Exactly.
PRIYANKA VERGADIA: Got it.
OK.
TRAVIS DEPUY: Conversely, microservices
are a lot like cupcakes, right?
If you have multiple flavors of cupcakes
and you want to change one little thing in the recipe,
you can then change the recipe, bake a bunch of new cupcakes,
and your customers get to have all the greatest and latest
cupcakes.
PRIYANKA VERGADIA: Well, that was
the most delicious explanation of the difference
between monoliths and microservices
that I have heard.
So Travis, what were some of the main challenges
that you all were going through while you
had this monolith turducken setting of the architecture?
I'm pretty sure there were challenges
that made you think that moving to microservices
was a good choice.
What were some of those challenges?
TRAVIS DEPUY: Well, we had monitoring and debugging
challenges.
So each customer essentially ended up in their own instance.
And so trying to track down issues
and trying to tell what the health of the applications
were was really difficult. We also found
that it was taking days, if not weeks,
to get new customers onboarded, which certainly isn't scalable.
PRIYANKA VERGADIA: I'm actually curious about how
the infrastructure was initially when it was a monolith.
TRAVIS DEPUY: Yeah.
So originally, we had CDs that we would send out to customers.
We were, you know, it was an on-premise application.
So we just had to take that and just put it into the cloud,
naturally.
And unfortunately, what this meant
is that we ballooned into 5,000 different VMs, which,
as I mentioned, isn't scalable.
PRIYANKA VERGADIA: Yeah.
So as you start to have 5,000 VMs, that definitely is a lot.
I'm assuming you started to think
you needed to move to Cloud.
What was that thought process like?
TRAVIS DEPUY: We rented space in six different data
centers across the world.
So we thought about, why are we getting
into dealing with hardware.
You know?
Google does that very, very well.
So why don't we leave it to the experts?
PRIYANKA VERGADIA: What were some
of the architectural components at a high level
that you were looking at as a part
of microservices architecture?
TRAVIS DEPUY: Yeah.
So naturally, the application was composed
of several different pieces.
So we had the web user interface,
we had API interfaces, and we also
have notification services, so being able to reach out
to email, or SMS, or voice calls were all
their own kind of service.
Then there was also the on-call scheduling,
so trying to figure out who's currently on call.
And so these were kind of the early things
we were able to identify that we could pull apart and tease out
of the application to make them their own service.
PRIYANKA VERGADIA: When you all were
starting to move to microservices,
what was the end goal in mind?
TRAVIS DEPUY: Primarily scale.
Maintaining 5,000 VMs is just not scalable.
So we tried to identify different aspects
that we could pull out some of the redundancy.
So, for example, we have this application gateway
that's speeding traffic into the different services.
So we're able to identify different places that we can
start to move to microservices.
PRIYANKA VERGADIA: And then so how
did that start to look in Google Cloud
from the architectural perspective?
What are the components you were using?
TRAVIS DEPUY: Yeah.
So we had a web user interface.
We had application APIs.
We had notification services and on-call scheduling.
So all these were different microservices
that we were able to pull out and break off
into their own microservice.
PRIYANKA VERGADIA: So application gateway approach
really sounds interesting and the natural thing
to do when you're starting out with microservices, breaking up
your monolith.
What are the components that you ended up using on Google Cloud?
What did the architecture look like?
TRAVIS DEPUY: So we used Google with DNS for IP address lookup.
And then users are routed through the global load
balancer, which gets them into the application gateway.
So from there, the user devices, such as web and mobile devices,
are routed into the various microservices that are all
running on GKE, using images stored in Google Container
Registry.
The data backend is running on Google Compute Engine.
Then we have the monitoring running, also
using Prometheus, Splunk, and, of course, Stackdriver.
PRIYANKA VERGADIA: Now, when you decided to actually get the GKE
services on, what were some of the services
that you picked out or selected from the monolith that
were going to go on.
What was the decision making process around that,
and what were some of those services?
TRAVIS DEPUY: So the application is
composed of many different pieces,
so it made logical sense to pull those different pieces apart
into their own services.
So, for example, we have the web user interface.
It provides the web interface for the various users.
Then we have the API interfaces, as well as, I
like on-call scheduling.
So it's its own service to figure out, OK, who's currently
on call when an event comes in.
And then there's the data services.
You know, keeping track of users and their devices,
and things like that.
PRIYANKA VERGADIA: So that brings up a very interesting
and a very important point when it comes to migrating monoliths
to microservices, because data dependencies are key.
And every service has some sort of dependency
on another service.
So this question always comes up,
as to how do you actually define which services should
be migrated first, or how do you deal
with those data dependencies?
TRAVIS DEPUY: Yes, certainly.
Microservices, you know, are very rarely
just standalone services.
You know, they always depend on other pieces.
So we just really made sure to release the services in cadence
with each other as we started to build out the application.
PRIYANKA VERGADIA: Yeah, so mainly just making sure
that you list out those services that you
may want to move to microservices,
and then you define the dependencies,
and then depending on which one's dependent on each other,
or less dependent on each other, you
define which one goes first versus next.
TRAVIS DEPUY: Yep.
PRIYANKA VERGADIA: OK, well that makes a lot of sense.
So we have a bunch of chopped veggies here.
What do we do next?
TRAVIS DEPUY: We're going to throw them in the pan
and then we'll add in the broth.
PRIYANKA VERGADIA: Oh, awesome.
Let's do it.
[MUSIC PLAYING]
This definitely seems like a pretty difficult and daunting
task.
How long did it actually take to finish out the entire migration
process?
TRAVIS DEPUY: Well, once we made the decision
to commit to Cloud, it took us about 12 months,
because we had data center contracts that
were coming up for renewal.
So it was either get our stuff out or buy more data center.
PRIYANKA VERGADIA: Yeah.
Yeah, well, those are obviously very
good situations to kind of push you towards a good timeline.
Was there a time when you were running in GCP as well as
in your on-premise environment?
TRAVIS DEPUY: Yeah.
We were running them both for about six months.
The flexibility of the Google Cloud Platform
allowed us to do that.
PRIYANKA VERGADIA: Well, it always
seems that it's important to understand
that the beauty of migration is, you're
not going to just move stuff from here to here.
It's always going to be a gradual process where
you move some services over, and you have both platforms running
for at least a considerable amount of time
so you can go back and forth between platforms.
It seems like that's what xMatters ended up doing.
You had six months in Google Cloud, and also running
parallelly on premises in order to facilitate the migration.
Can you talk a little bit about the challenges, or rather,
the success metrics that actually led to that migration?
TRAVIS DEPUY: Yeah.
So metrics are critical.
You know, we started with monitoring and observability
right off the bat so that we knew
that, as we introduced changes to the entire environment,
we could see how they impacted the different aspects
of the system.
PRIYANKA VERGADIA: So Stackdriver, I'm pretty sure,
played a really big role in that monitoring journey.
But I'm pretty curious about Google Kubernetes Engine,
and how did that play a role in those incremental changes
as you were gradually moving over to Cloud.
TRAVIS DEPUY: So GKE is definitely
the central part of our microservices architecture.
We also use GKE VPC native IPs for direct addressing.
And then we also use a sidecar pattern
for incremental deployments.
PRIYANKA VERGADIA: So Travis, what
is the most challenging part of actually moving
into microservices pattern?
TRAVIS DEPUY: Sorting out the service ownership.
So mapping the teams, the different services,
as well as deploying those services into the environment.
PRIYANKA VERGADIA: We did touch a little bit
on how to find out the services that
needed to be migrated, and then prioritize them
in the sequence, and all of those things.
That is really hard to do.
Should we check on our soup?
Because I think--
TRAVIS DEPUY: I think it smells like it's getting there.
PRIYANKA VERGADIA: Yeah, OK.
So let's do that.
So Travis, this change must have been
really new for the development and the operations team,
because they are learning something new.
Moving into microservices, they're
designing things differently.
They're working on them differently.
The cadence might be different.
How did you go about making the development and operations
team feel comfortable about the change?
TRAVIS DEPUY: You know, Jez Humble
talks about this really well.
He says that bad behavior arises when you abstract teams
from the consequences of their actions.
So we really tried to align the teams with the services
that they had to build and maintain.
PRIYANKA VERGADIA: So basically, they
will be accountable for the SLOs if you give them
more ownership and more accountability for the work
that they are doing.
So now that you've gone through this migration,
what are the impacts that you're starting to see?
TRAVIS DEPUY: Well, happier customers mean happier teams.
Empirically, we found that we had a 43% performance
improvement as well as 60% less incidents.
PRIYANKA VERGADIA: Well, happy team and happy customers
definitely sounds like a win to me.
Apart from this, as you are now already into a Google Cloud
with the microservices, broken down from a monolith,
what are you seeing as the next steps?
Where do we want to go from the architectural perspective?
TRAVIS DEPUY: Well, total domination of the market,
of course.
PRIYANKA VERGADIA: Well, apart from that.
TRAVIS DEPUY: Well architecturally, we'll
look into moving into service mesh architecture,
so you're probably using Istio.
And that will allow us to get rid of this application gateway
and it'll let the microservices talk to each other directly.
PRIYANKA VERGADIA: Yeah.
Service mesh definitely seems like the next logical step
to take where you can connect, secure, control, and observe
your applications while in a microservices fashion.
Well, awesome.
That was very insightful, Travis.
Thank you so much for having me over.
TRAVIS DEPUY: Yeah, thanks for coming.
PRIYANKA VERGADIA: Thanks for joining us today.
I'm glad you were able to catch the recipe of migrating
a monolith to microservices with xMatters.
If you liked this video, then stay
tuned for more such episodes of Get Cooking in Cloud.
[MUSIC PLAYING]