I'm Michael Krigsman, industry analyst and host of CXOTalk.
And, we are at Future Stack '16, which is New Relic's conference here in San Francisco.
And I have the privilege of talking with Mikey Butler, who is the Senior Vice President of
Engineering at New Relic.
Hey Mikey, how are you?
Good, Michael.
How are you doing?
I'm honored to be here.
Thank you so much!
So, you run engineering at New Relic, and very briefly, what does that encompass?
Well, it basically encompasses two large responsibilities.
One is actually putting product out the door.
And approximately two thirds of the 200 people that report to me focus on putting product
out the door, and one third focuses on actually keeping the site up.
As you know, we're a SaaS offering, so we've got 24/7, 365 responsibility; so one third
of the team is focused on that.
And that's what we do day-in and day-out.
You have spend a lot of time thinking about what we might call, "high-performing engineering
team," ... Yes.
...and the culture around that.
So please, share with us your thoughts.
Okay.
I would classify my thinking into two buckets, and I use the phrase "reactive and proactive."
"Reactive" tends to be the typical blocking and tackling that we're all familiar with:
getting engineers to produce code more quickly; having it get through the test cycle more
quickly; having it aggregate into a release more quickly, with lower friction; that's
the reactive side of things.
And that typically, is where it tends to end in most organizations.
I was blessed in the case of New Relic to have a very forward-thinking boss.
My boss, Jim Gochee, who's the Chief Product Officer.
And, he decided to do something very bold, and I was more than happy to sign up for it.
And that's something we call "Project Upscale," and it's a complete re-do of how we do engineering.
And, it is very state of the art.
And so, what we're trying to do with Upscale is to rethink completely what is the business
of an engineering team.
It's more than putting product out the door.
It's actually properly targeting the efforts of a team, which roughly is six people across
all disciplines: so managers, product managers, team leads, and individual engineers; having
them focus on the right things; and at the same time having them align with their passions.
If I hand out a remit to a team and say, "I want you to work on X," some percent of that
team may be interested in that; some percentage won't be interested in that.
But, what we created with Upscale is this sequence, or a set of desired deliverables
─ typically twelve to fifteen of them, and said, "Alright.
Of the forty teams you have, select the ones you want to work on...
And why do you want to work on it, and tell us what's in in for the business."
We then go back, we look at their proposals, we bless some, we say, "No, try again with
others," and when the dust settles, we actually have a good match between what the business
needs and what people have as passion.
And, therefore, as you well know, if someone's passionate about something, you get a lot
more out of it than if they're just in it to turn the crank and do their job.
So it's a very systematic way of creating a culture, where people feel highly invested
in the work that they do.
Absolutely.
For example, one of the nuances there, to give you just how far we took it, we actually
gave all 250 or so of our engineers and product managers the opportunity to select which team
they were on.
It caused, actually, a lot of anxiety inside of the company when we told our CEO, and luckily,
he basically signed up for it.
But in the beginning we said, "Yeah, we're going to basically let everybody decide what
job they want to do."
It's like, "Oh my God!
How long will that take to settle and what will happen on the other side of that , what
will happen ot product going out the door," all of that turned out to, let's say work
out perfectly.
It couldn't have been more of a textbook perfect outcome.
But, if we actually let people decide what team they're going to be on, and every six
months or so they get a chance to select another team, the advantage to us that we know the
passion is aligned, the advantage to the individual is that: as they follow their passions, they
don't have to leave the company; that there's a guaranteed opportunity every six months
or so to align yourself with your passions.
And as you know, in the technology space that happens pretty often and very fast.
And so, people tend to stay much longer.
Our attrition rates are far below average as a result of making this move.
So, many companies find that making this kind of change is difficult, because change is
hard.
So, would you share with us some of the lessons and the things that you learn that make it
... Yeah.
Well, part of it is just good old-fashioned gumption.
The management team is going to sit in the middle of what will seem like a firestorm
once you start this up.
You're going to go down the path, and even the engineering team is going to have doubts
about what you did; certainly the teams around you that depend on the engineering team will
see this as a moment of chaos, and you kind of have to just gut yourself for that.
Find courage and intestinal fortitude when you need it, and wrap your heart around the
vision, and keep that hold to yourself and to the team, and that actually carried us
through.
The interesting thing is, we hired a consultant to come on board.
He's done this with about a dozen other companies, and they all have about the same journey,
which is ... It is so easy to innovate the technology and the tools, and far more difficult
to innovate how teams do their work; and so that tends to be an exception-case in our
industry, and therefore tends to push a lot of the, what I'll call the "organizational
calculus" to its limit.
But, if you can go there and get through that, the upside is tremendous.
How did you ensure that trust was part of that organizational calculus, because without
trust, this would not have worked.
Yes, and so, trust was greatly enhanced by us looking at all of the forty teams, and
asking them to elect representatives to actually drive the process.
So management did not proscribe the outcome, we actually gave the team a set of business
problems.
We need the business priorities observed, we want teams to be passionate about what
they do, we want them to have specific charters about what they do, and we said, "You go off
and you design the system that responds to these constraints in the most effective way."
So, not only was it a better outcome than we would have defined, had we been sitting
in our ivory tower of management, but it helped the buy-in of the team, because their representatives
were working with them to fine-tune as they went along.
So, that built a lot of trust.
And finally, New Relic is a very data-oriented company.
Yes.
And so, what's the data that you look at?
How do you know that you're successful at this?
Well, that's actually a big talent.
So, we tend to drink our own champagne and so, all along the process of building code,
deploying code, we actually are using the Insight product and the Analytics for Everyone
platform, to continuously gather data.
For me, this was actually one of the reasons I came to New Relic; you know, it was one
of these things I spent a lot of time studying information when I was in school, and there's
a correlation between how often things change: how fast they change, how frequently they
change, or that the size of change, and the sampling rate that you need.
And, what we were finding is that if you actually use the old-school method of what I call "metrics-based
analyis," that you have to know ahead of time what you're interested in measuring.
With event-based and insights-based product, you don't know what you want to measure half
the time.
You just measure everything, and then you have essentially the full data cube that you
can spin wherever you want, and slice the data where you want after the fact, when you
know what the precedent really is.
So that's what we depend on.
... You heard that we're getting something in the order of 25 trillion events recorded
into the New Relic NRDB backend system.
We're probably responsible for a good percentage of that inside of our engineering organization.
I don't know the percentage, but we use it a lot, and what we do is essentially gather
data all about the entire assembly line and how teams are operating; and then, at periodic
reviews, we extract that data, rotate the data cube, slice it the way we want, and basically
get some very, very interesting insights that we couldn't possibly have gotten if we hadn't
used our own tool.
Fantastic!
Well, thank you so much!
It's my pleasure!
Thank you!
We have been talking with Mikey Butler, who is the Senior Vice President of Engineering
at New Relic.