RAVI LACHHMAN: Hi, everybody.
My name is Ravi Lachhman.
I'm an evangelist at Harness and very excited to be
talking at Google Cloud Next.
I had a lot of outages in my day,
so hopefully we can learn a little bit
how not to get burned.
But also with me is my esteemed colleague, Brett.
Brett?
BRETT ZANE-ULMAN: Hi, I'm Brett Zane-Ulman.
I'm an engineering manager here at Harness.
And I'm excited to be here today.
Go ahead, Ravi.
RAVI LACHHMAN: Perfect.
So what we're going to be talking about today--
so at Harness, we've actually went
through a pretty big migration on GCP.
But a little bit of level setting,
what on earth is Harness?
And we'd like to decide our time before GCP,
our time on the way to GCP, and our time on GCP.
It's very nicely segmented here.
So in the beginning, let's talk a little bit about Harness.
A divine light came down and shined
upon the continuous delivery world, and Harness was born.
So again, a bright light shined.
So what are we actually talking about here?
We're talking about Harness.
At Harness, we're a continuous delivery as a service.
So what does that mean?
A continuous delivery as a service
is something that helps orchestrate
a couple of parts for you.
So getting your software or your platforms into production
really requires orchestration between approvals,
all the confidence building steps
that you have, such as testing, and also just a way
to do release, right?
So are you leveraging a safe type of release like a canary?
So you see the canary doing a dab there.
Are you doing a blue-green?
Now if these terms are new to you, don't worry.
We'll be here after the presentation to kind of answer
some questions.
But Harness really packages all that up for you as a service.
Also, since we have a lot of customers that depend on us,
we always have to be up.
And so what the Harness platform does is exactly that.
We're supporting over 100 enterprises, thousands
of community users, and also, hundreds of thousands
of deployments regularly.
Ironically, we actually help empower people coming
and talking on stages in different technology
conferences because if you're looking
at where their applications go, most likely,
they could be using Harness in the background
to help deploy that.
And so what do we deal with at Harness?
We actually have some unique challenges at Harness,
is that we have a SaaS version of our product.
So people expect uptime all the time.
But also, we're supporting two other type of cloud deployment
models.
I like to call it kind of SaaS, right?
So certain customers might not like being
completely in the public cloud.
And so there's limited connection.
They're running parts of the Harness platform
on their hardware.
And also, we have a completely Air Gap version--
I call it No SaaS'in Around--
of the products so customers can be completely disconnected,
again, in trying to leverage the Harness platform.
And so what are some of the challenges of the platform,
right?
Because again, we have to maintain
three different versions of our platform.
So the first thing before our time with GCP
was scaling of the platform is basically non-existent.
We had a very rigid architecture.
And Brett will walk through that in a couple of minutes.
Also, this was our first foray into Kubernetes.
Actually, none of our workload for ourselves
were hosted or deployed on the Kubernetes.
Now a lot of our customers are using Kubernetes.
But it was actually our first foray into Kubernetes.
We went to GCP.
And also, we had no separation of duty, right?
So what that means is that we didn't
have lower environments deploying
to other different lower environments.
Basically, our prod environments that we
were building the platform on deployed straight to prod.
So we had the lack of separation of duties.
And also, we had multiple versions
of the platform, which was becoming difficult to maintain.
And also, we were completely on SaaS.
So that means that any sort of downtime for us
would be very detrimental to the customers.
And so, I'm about to hand it over to Brett.
Let's talk about our time, Brett, before the mighty GCP.
Brett?
BRETT ZANE-ULMAN: Sure, Ravi, thanks.
So as Ravi mentioned, our production environment
prior to moving to GCP was pretty rigid.
This is basically what it looked like.
It's very simple.
We had just a couple of EC2 instances that
were hosting our UI, which is just some static content
and then three other VMs running our orchestration servers
and connecting to our database.
Each one was behind a load balancer.
And that was pretty much it.
What we had to do is update every time we
want to deploy a new version.
And Harness is a deployment tool.
So of course, we're using Harness to deploy Harness.
But at this time, we didn't have any other environments set up
for deploying to production.
So production deployed itself.
And how did this work?
So our servers, which are called managers,
orchestrate workflow executions during a deployment.
But the actual calls to deploy and update a service
are done by a worker, or delegate, in the customer
environment.
So in this case, being our own customer,
our delegate is in the same VPC as the servers.
And while the workflow runs, it goes
through one server at a time, updating the binary
and restarting the service.
This is all well and good, except that during the course
of the deployment, the orchestration engine
itself gets updated to a new version.
And since it relies on communicating back and forth
with the delegate, it's very sensitive to data model changes
that may have been introduced by the new version.
Of course, if this goes wrong and hangs,
then you get stuck with production
in an intermediate state.
RAVI LACHHMAN: And now it's time to introduce Kubernetes.
So given all the challenges that we had,
looking to really leverage Kubernetes to help
overcome that, a lot of our customers
are moving towards Kubernetes.
And also our platform actually deploys to Kubernetes.
So the next rational step for us was actually
to be going towards Kubernetes.
Brett, let's talk about some reasons
why we were looking to leverage our very own Kubernetes.
BRETT ZANE-ULMAN: Sure, Ravi.
So, as Ravi mentioned, over the first year that we even
had a production environment, we saw the rise of Kubernetes
in popularity.
It was becoming more and more popular with our customers.
And so we wanted to leverage that for ourselves.
Our primary goal was to dockerize our microservices
and start using Kubernetes.
And when we looked around at the time,
GKE was really the best and one of the only hosted Kubernetes
services.
We already had Kubernetes support in Harness on GKE,
so why not leverage that and use it for ourselves?
So we built out the infrastructure
and defined all of our Kubernetes configuration
in Harness to deploy all of our microservices,
exposing an NGINX Ingress Controller via a load balancer
and then routing to our various services.
So once this had been working in QA for some time,
we were ready to make the switch in production.
So making the migration is what we're here for.
So let's talk about that.
RAVI LACHHMAN: Yeah, this is what the people came for.
We have two pieces of infrastructure now.
How do we make the solution?
[INAUDIBLE] And it's so kind of gearing up
towards the switch, kind of the ethos that we have at Harness,
that we help enable people doing production
changes in normal hours, right?
If you kind of take a look back at years gone by,
usually, you're deploying during your green window
or a black window.
Pick your color, right?
It's 2:00 or 3:00 in the morning.
And everyone's tired, right?
And so since we help enable people
make this type of deployment or really critical deployments
in normal business hours for the geography,
we decided to drink our own champagne
or eat our own dog food, that, hey,
let's do a dinner time switch, because we
had two sets of team.
So around dinnertime in San Francisco
is actually breakfast time in India.
So for all the resources that we had to line up,
it made the best sense for us to do it around dinner time.
And so Brett, let's talk a little bit about the switch
that we had to do.
BRETT ZANE-ULMAN: Sure.
So we were migrating not only our own infrastructure,
but also our database, which is a hosted Atlas Mongo DB
cluster.
While the old production instances on the left
were still live, we deployed and brought up the new Kubernetes
stack, pointed at the database that was still
in AWS at the time, just to make sure it was running fine,
well connected to the production DB,
but without any traffic reaching it yet.
So once that works, we scaled it back down.
Then, for the database, we live migrated the data over
to another Atlas cluster, this time hosted in GCP.
The data there was keeping pace with the live production
database with a few seconds of lag.
Then, to make the switch, we took a few minutes of downtime,
turned down the old production instances,
so that they stop writing to the database,
then wait a few moments for the database live migration
to catch up completely.
Then we finalized the data migration.
So then we reconfigured the new Kubernetes manager to point
to the new database in GCP and redeployed with Harness.
So once that was up and verified on our end,
the only thing left was to switch the DNS entry
to the IP of the new load balancer.
RAVI LACHHMAN: Awesome.
And not wanting to do the big reveal,
but it's in the title of our presentation.
How long do you think it actually
took us to make the kind of [INAUDIBLE] switch?
And the big reveal-- this is actually
live Slack from when we made the switch.
It was a total of 76 minutes.
And it was very impressive that this change only took
76 minutes for us to execute.
But maybe we could talk a little bit
about what the final ecosystem looked like for us, given
the benefits and some of the lacking stuff
we had in our old architecture, Brett.
BRETT ZANE-ULMAN: Sure.
So I mentioned previously some of the problems
with the self deployment model and the data
model changes that can cause problems
when we upgrade ourselves.
So what we've done now, we've added
a number of new microservices.
And we've introduced a new internal Harness environment
that deploys production, so we're not self
deploying anymore.
And we've also introduced versioning to address the data
model changes that I mentioned.
So the payload is synchronized between the servers
and the workers at all times via the versioning.
So now we're able to quickly scale using GKE
and meet new demand or deal with production issues.
This is an example of what our pipelines look
like while they're running.
And with the products using these types of pipelines,
we're able to wrangle all the configuration that you
need to stand up your services.
And that definitely helped us when
we were changing both the cloud provider and the deployment
model at the same time.
RAVI LACHHMAN: Yeah, absolutely.
And I think just kind of adding to some of Brett's points
there, we are a continuous delivery company.
And it really shows the importance
for making such a big infrastructure change that
actually leverage a pipeline.
As we were going through the development and also
the testing of this, it's kind of interesting,
as the kind of the shift of infrastructure to the cloud
allows for a lot more iteration, and so running the pipelines,
kind of testing out what would happen in success scenarios,
what would happen in failure scenarios, was quite important.
So it wasn't all, this is the first time
we're doing something.
There was a lot of trial and error to get this right.
But with all of that, with all the trials and tribulations,
right, we made a fairly big change.
And it has really paid off for us,
partnering with Google Cloud, that we're able to, A, we're
increasing the number of customers that we have,
the infrastructure scaling appropriately,
and just really allowing us to grow and scale our business.
And so don't forget after all this is done
and to celebrate success, and really looking
at a very simple equation for what is success, right?
So there was certainly a lot of prep work, right,
so gathering and planning, taking inventory
of what you have, making sure that these particular pieces,
especially as we started to move towards more dockerization,
right?
It's a very, very [INAUDIBLE] approach that we took.
Also, for execution, make sure you
have the right staff in place.
If it's 2:00 or 3:00 in the morning,
it doesn't do anybody a favor, right, if it's
a time that everyone's awake.
You're certainly more alert.
And also, the journey is never over, right?
A common thing that we like to ask our prospects and customers
is, hey, when is your deployment over?
It's kind of an intrinsic question.
Because, again, you always want to make sure
that you're always refining and continually approving
what you do have.
That kind of lasting impact that we have for Google Cloud, even
our leveraging technologies like GKE,
when we have new engineers even from field sales to customer
support to even engineering where Brett's at, teaching them
on Kubernetes and GKE makes it easier for us, right?
So they have an easy way to kind of see and play
with Kubernetes.
And thanks once again for catching our presentation.
Brett and I really appreciated you coming to catch our talk.
And hopefully, we can catch up with everybody
in person one of these days.
Cheers.