>> Hey, good news.
We're here to talk to Robby Millsap today about
optimizing your release processes through Operational Excellence.
Welcome to the Azure Enablement Show,
where we'll be discussing the challenges you
and our other tech-savvy customers have encountered.
Together we're going to be talking with experts to
find out how they think about these problems,
recommended tools and best practices,
and tips they've learned from
years of experience that you can use.
Today we're going to be talking to Robby Millsap
about release practices and I'm hoping you're going to join me.
I think it would be a better discussion if I
did this while I was in customer mode, so let me switch.
Okay, that's better. Now I'm in customer mode.
I think it's time to go talk to Robby. Hey, Robby.
>> Hey David, welcome to the lodge.
>> Thank you very much.
I was talking with Canal
about a problem we've been having it work.
Basically, our release processes are terrible.
Let me explain what I mean by terrible.
They're terrible in that we do release once a
quarter sometimes and it takes the entire weekend,
I don't get to see my family in the entire time.
Then after that, we spend the week after that once we can stable,
recovering and fixing things.
Sometimes it works and sometimes it
doesn't, and it's just terrible.
I'm wondering if you
could help me out and talk me through some things.
>> Yeah, David, it sounds like you have quite a bit of
friction in your release process.
>> Yes. What do you mean by friction?
>> Well, as you know, I'm part of the Well-Architected team
here at Microsoft, and removing
friction from the release engineering process
is one of the principles of operational excellence.
>> Okay. Well, I
guess I can understand that and I understand that in terms of,
the word friction makes sense to me
because I'm always being told that we need to
go faster but I don't know how we would do that.
We put the release out there as fast as we can.
Our test all upgrade,
and the thing still break.
There's a whole bunch of stuff going on here.
It takes forever for the pipelines to run,
our tests even though we have these tests,
they don't tell us that things are going to break
a week later after we're done with the release.
It's just a lot of work.
>> David, I have to say that sounds
like a lot of what we call toil.
>> Yes, toil as in, what do you mean?
There's another word that I know,
but I want to make sure I understand what you're meaning by it.
>> Toil consists of those mundane, repetitive,
low-value manual tasks that you need to
run a release when the release is exceptionally painful.
I'm going to ask you, is there a reason that
these releases are so spaced out?
Is there a business requirement?
>> No, it's just the way we've always done it.
We figured quarterly releases, everybody does it.
>> Part of an optimal release process is to seek out toil,
the kind of things that you described,
and aggressively eliminate them.
The key to that really is through automation.
>> I did automation.
I think I know where you're going with this, automate everything.
I've heard it a lot of times and it sounds great.
But for us it feels like so pie in the sky.
For example, it doesn't even automate everything.
For example, we need these admin accounts with
special elevated permissions to do deployment to our VMs.
We don't want to put those in some pipeline process.
Automating that sort of thing seems like impossible to us.
>> I think you'd be surprised if you
really seek to eliminate toil and those manual tasks,
you'll find there's a solution
for a lot of these things that you're doing.
For example, you presented the example of these admin accounts,
maybe you don't want to put admin permissions
outside of Azure, which I totally get.
But you know that you can use just-in-time access accounts,
which will give security privileges
for a very limited window of time,
and that's still backed by Azure role-based access control.
So you're not putting your password into some external system.
Again, seeking toil and aggressively eliminating it,
you'll come up with some of these solutions like that.
>> I think that we could consider doing that,
but I'm still not clear why automate everything is the goal.
>> Well, toil slows progress.
It's disruptive, it's bad for
team morale which you've experienced.
It's also very prone to human error and
these automated processes will shift
efforts to delivering software,
which is what developers want to do anyway,
and what the business wants you to do,
delivering features,
not babysitting release processes at the end of every quarter.
>> That makes sense to me. I get that we
need to think about toil and I'll definitely deal with that.
But can we go back to our deployments that are just like,
I like when you're talking about friction before.
I would just want to understand what do you mean by that?
I'll give you an example. Our deployments
always seem to have these errors.
Sometimes, obviously, we've seen the logs,
but sometimes we don't hear about it for a few hours or days,
or maybe it's the week later.
We'll go look through two weeks with
the logs or code changes to figure
out who forgot to check a certain thing.
What do I do about these things?
>> I would say, first, if there's no government or compliance
reason why the code has to be developed so infrequently,
I would look at doing smaller,
more frequent incremental changes and leveraging
things like feature toggles to turn functionality on and off.
It's a lot easier to turn something on and off
versus a redeployment to roll back a feature.
Also, it sounds like you might benefit from
splitting up some of that work in parallel.
For example, taking that code,
building out different components which you
can put into a package manager like NuGet,
and then you don't have to recompile the world every time.
A smaller area of change means it's probably
going to be easier to discover when something goes wrong.
>> That makes sense to me.
But when I think about things like,
you were talking about feature flags and stuff like that,
that's like a configuration thing.
We have issues with
configuration especially because we
have multiple environments like everybody does.
We have prod, and dev,
and staging and stuff like that,
and managing the configuration between them is really tricky.
I'm finding I'm always going between
them trying to compare config files.
>> What you're describing we call
configuration drift and it can be another source of friction.
A great way to eliminate configuration drift is
to deploy your infrastructure alongside your app code.
I'm sure you've heard of Infrastructure as Code.
That's just that whole idea is provisioning
resources by defining it in some declarative form
like an Azure Resource Manager template
or something that's actually checked in alongside your code.
>> That makes sense. These amendment changes
do cause us lots of grief here.
Let's move on to
the next question which is
around the tests, I was mentioning to you.
I've got a whole bunch of unit tests,
like we're all in on the unit test,
that makes sense to us at all.
They turn green and we don't deploy until they turn green.
We try to do the right thing,
but things still seem to break in production.
At this point the business people
are coming to me like why are you spending
all your time on these unit tests if it's not finding
these things. What do you do?
>> That's a common question.
Unit tests are great at identifying coding mistakes,
but they're only going to tell you so much.
I would consider adding tests at
every stage of your development process.
For example, smoke tests to validate deployments,
integration tests to make sure
your distributing components can communicate with each other,
as well as stress tests to benchmark performance and
quickly identify when a performance problem has been introduced.
>> I get that. But sometimes code seems,
the word would be brittle,
when it's deployed to production it breaks easily.
What do we do about that?
>> One cause of that brittleness you described might
be the way your code handles unexpected faults.
I would look at injecting fault into
your testing plan to ensure that your site keeps
running even in a limited capacity
when some of those dependencies go up and down.
>> All of this sounds super,
and I'm excited to try these things.
But I still imagine myself sitting
there one weekend out of my month or
more while I try to just babysit the release process and I can't
leave the site down for maintenance, under construction.
I can't put up a big come back later until
morning because my boss would just
kill me if we were down that long.
What do I do with that?
>> I think I can help you with this one immediately.
Ideally, deployments should be
frequent and transparent to most of your users.
Have you tried staging your workloads?
I think that's something that you could do immediately.
Let me give you an example.
You've probably heard of blue-green deployments
or canary releases.
But the general principle is it's a lot faster to flip
a switch and send traffic from one source to another source,
versus a deployment of new code that has to be
applied on top of running applications.
>> I have heard of these things
and the reason why we haven't done something like
blue-green deployment where you
spin up two different versions of something and you're just like,
now we're blue, okay,
then we deploy the green side,
and then we switched to green is,
I don't know how we're going to pay for it.
It seems like I'm going to need like an entire replica of
our production environment if I
was going to do stuff like that. How do I handle that?
>> That's a good concern. But you know,
if you're using something like Azure Functions,
which most people are these days,
you can use deployment slots to achieve the same thing.
You can have a green and a blue deployment slot
and you can flip between those.
>> Okay, that's cool.
It's pretty clear that I've got a lot to think about here.
If I wanted to go look into
this more where would I go, what would I do?
Where would you recommend I look at?
>> Everything you need to know about
the Well-Architected framework can be found on the website,
which is down below.
We also have an assessment where you can actually evaluate
your existing or any future developments
against the Well-Architected framework principles.
>> Cool. Well, that's super-helpful.
Robby, thank you so much for this.
I think I'm going to switch back to host mode now.
Back to host mode. Well, I just want to thank
Robby for joining me
today and I'd like to thank all of you for watching.
I hope you'll join us again on
another episode of the Azure Enablement Show.