RICHARD SEROTER: Hi, Charity.
Thank you for joining me today.
Can you tell us a little bit about who you are, what you do?
CHARITY MAJORS: Absolutely.
My name is Charity Majors.
I'm an operations engineer.
I've worked on systems--
Parse, Facebook, et cetera.
And now I'm the co-founder and CTO
of Honeycomb.io, where we work on observability tools.
RICHARD SEROTER: Awesome.
Well, I really appreciate you joining me today
to chat a little bit about cloud, and observability,
and things like that.
So first, I just wanted to ask you,
why do you see companies in the first place
even use cloud infrastructure from more than one provider?
Why do that?
CHARITY MAJORS: Why do that?
Well, because not every provider is great at the same thing.
Some providers are better at different things.
And we live in this glorious age of infrastructure
where you get to pick and choose.
So there's very little penalty to going to the best provider
to do the best thing.
RICHARD SEROTER: As you've looked
at what you've done with Honeycomb,
what have you learned from building software that
doesn't just run in one place?
Of course, you run all over the place.
It's great.
So have you learned something that maybe somebody who's
looking at building stuff that has to run in different places
can take a lesson from you?
CHARITY MAJORS: Yeah, absolutely.
If you don't have to, don't do it.
[LAUGHTER]
If you can build your systems, if you
can deliver to your customers what
you want to deliver to them using a LAMP stack, by god,
please do it.
If you do it on a single cloud service, by god, please do it.
The problem is that for many of us,
we can't actually build what we want
to build by following those very simple and tried and true
paradigms.
So keep it as simple as you possibly can,
is usually my best advice to people.
RICHARD SEROTER: Has there been anything
that's been particularly hard about it that,
when you looked at having Honeycomb work
across different public clouds, on-prem, whatever,
all these different environments, that you're like,
oof.
The networking or the storage or some other aspect
was always harder than you thought.
CHARITY MAJORS: Yeah, it's always hard,
which is why we keep kicking that football, baseball--
I don't know.
We keep kicking some ball down the road as far as we can.
We look for every excuse not to go multicloud, multiregion,
multi-everything.
We came up with a security product.
Everybody wants to have their data live
in their own on-premises.
And yet, they want the power and delight of SaaS.
So we came up with a really cool hybrid product
that you run using the SaaS solution,
but streaming all of your data through a proxy
that you run what's in your network, whatever that is,
and just generates the hash of each event and then
forwards on the hash or forwards on the encrypted version
of the event, but keeps the keys there on-premises.
We never get the keys.
We could never decrypt it.
So it's a nice little hybrid way of trying to fulfill
all those constraints.
That said, it only works for so long.
And we are looking at having to spin up an EU
site in the next year or so.
So don't do it until you have to.
Don't do it until lots of people with lots of money are saying,
yes, we want this.
RICHARD SEROTER: Wow.
That's good advice.
To me, one of the worst ticket calls
you can get to the help desk is, something is slow.
It's like, oh gosh.
Where do I even go with this stuff?
So when you start looking at systems,
either where they span on-prem and cloud or even
different clouds, especially give us
a little bit of a lesson on observability, I guess.
But where do you start?
I'm assuming from what I read from you,
it's not just staring at static dashboards.
CHARITY MAJORS: [LAUGHS] No.
No.
That's not going to get you very far.
Yeah, and like Liz Von Jones likes to say,
slow is the new down.
We've gotten so good at creating these resilient systems,
they rarely go down for customers.
But they do get slow, and that's just as frustrating.
Well, first of all, I think that observability is,
in large part--
it's not about debugging your code.
It's not about fixing the code itself.
But it's, in a large part, about finding, where in the system
is the code that you need to debug?
Which means that you need to be--
part of observability, I think, is
that you collect your telemetry in a way such
that you get one event per request per service.
And then you can populate those events with as many details
as you want.
But having one event per request per service
allows you to correlate all kinds of things
from request to request while keeping that unique request
ID, unique trace ID, disparities, et cetera.
It's interesting.
Something that a lot of newbies to observability do
is they will only put observability on their edge,
on their web tier, basically, using HAProxy, or Nginx,
or whatever.
But as long as they add these headers to every service
that's behind the web tier that keep track of how long this it
took to call this other service, and as long
as they print those values out and then surface them
in the log on the edge, you can actually
get an entire map of your entire infrastructure.
And you can pinpoint, oh, that's where the slow thing is.
Oh, that's where the slow node is.
And that's just so cool.
You're not going to find these problems of slowness
unless you're really keeping track
of the edges, the borders between services
every time you're calling out HTTP, or to SQL,
or anything else.
But that boundary is super important.
RICHARD SEROTER: That's really insightful.
Does that get harder?
Is it just different when those boundaries represent cloud API
gateways and they represent, gosh,
I'm [INAUDIBLE] a VPC in this cloud that cloud?
Is it just, if I generate the data,
then this shouldn't be rocket science, regardless of where
my stuff sits?
CHARITY MAJORS: Basically true, as long
as you could instrument the code.
There is kind of a bright line between code you can instrument
and code that you can't.
If it's code you can't instrument,
then you're reliant on whatever they're sending back to you.
So for like databases, if you're not Google,
you don't get to go in and instrument your database
to add a span there, add something there.
You've got to treat it like a black box.
So some ways that we can approximate having instrumented
it are we could do a tcpdump and stream
all of the network traffic and reconstitute those events.
We can tail -f the MySQL log to get the errors out.
And we can connect via the console
to dump out all the internal statistics every couple
seconds.
That gives you a pretty good view
of what's going on inside your database,
even though you're not able to instrument it.
Or if you've got serverless stuff--
serverless is actually-- it's kind
of how I tell people, think about serverless when you're
instrumenting for your software because you never
assume that you have access to hardware.
But you can still get really interesting stuff out of it,
as long as you're instrumenting it well.
Of course, when it's code that you write yourself,
you're in luck.
You could do whatever the hell you want.
When it's remote APIs, do have the ability to instrument?
Do you not?
If not, you have to treat them like a black box
and just track the amount of time
that they spend running and then what they give you back.
That's all you can do.
So there are really just those three categories.
You can't do anything, so just track it in and out.
You could do something to approximate instrumentation.
And then you can instrument the hell out of it.
RICHARD SEROTER: I like all of those.
So are there cases where you've seen where using--
to your point, I like, hey.
If you just pick one cloud, please pick us.
But other people pick other places.
But do you see spots where using multiple clouds
does actually improve resilience in some way?
Or is that DR myth a myth?
Do you see that multiple clouds actually fix my resilience
in some settings?
CHARITY MAJORS: I can pretty much guarantee you
that if resilience is what you're going after,
you will have more outages by having more clouds than you
will by just sticking to one.
[LAUGHS] There are legit reasons, but most of them
are business reasons, as far as I can tell.
But DR-- how often does Google Cloud go down?
Most people can afford a couple of minutes a year,
which is basically what you're going to get from the top three
clouds.
And if you want to double your engineering
time to try and set that up, well, now you
have three problems.
You have AWS, you have Google, and you
have all of that connective tissue between them.
So I don't think it's a good play for resiliency,
although any time in software you say never
or always, you're going to be wrong for some outlier case.
[LAUGHTER]
RICHARD SEROTER: That's fair.
I will ask you, though--
so when I'm thinking about an operational approach,
you're giving someone advice and they are figuring out,
all right, I'm using infrastructure
in lots of places.
Again, it could be on-prem and cloud, different places.
Do you recommend that I kind of treat each environment
as a silo, from a logs and metrics perspective,
and just data?
Do I collect it all together for centralization?
Should I use one tool chain for my work?
How do you look at, when I've got increasingly distributed
infrastructure, how should I tackle ops?
CHARITY MAJORS: God, that's a complicated question.
And it's not the kind of question
I could answer without asking a lot more questions.
But philosophically, you always want
there to be one place that you can look for everything.
Philosophically, you don't want to have
to be going, this paging thing to that paging
thing to that paging thing, because infrastructure--
they have ripple effects.
You never want to be getting paged
by a bunch of different systems about one problem.
Philosophically, you always want them
to trickle up to one place.
And one of the best ways to do that is using SLOs.
If you're using SLOs, then you can usually
ditch a lot of the paging on symptoms, paging on patterns
or error spikes here and there, which are going to be constant
when you're at that level.
Your system is never up.
It's just got pathologies all the way down,
but we manage to run anyway.
It's not about making sure it doesn't go down.
It's about making sure that customers
aren't affected when we do.
There's always a bit of a tug of war, though,
between optimizing locally and optimizing globally,
because sometimes the global answer for a local solution
sucks.
And the engineers who are working on it know it sucks.
And so you kind of got to bridge that gap
by surfacing whatever you can to the global solution,
while trying to make it better for the engineers who
are working on the local solution
to actually do things when things break.
And I think that there are some interesting ways
you can kind of partition this up,
because you don't want people who work on service over here
and service over here, that aren't really linked
in any way, to get paged at the same time,
even if they share a backend or something.
So the dream is always that you can start it with a SLO.
Start at the top.
Slice and dice.
Figure out what the dependencies are.
And then maybe branch off into, OK, here
are the local systems for-- because for example, if you're
just having the one logs to rule them all,
you can never store all of the output
of a TCP for your low-level systems
or all of the output for print everything or debug out.
It's too much.
But there are maybe lower level tiers where
you can gather more fidelity and more detail so
that the engineers who get alerted
and it's routed to them, they can dig in deeper.
RICHARD SEROTER: Yeah, that makes a ton of sense.
Well, this has been awesome.
I got smarter on observability and just cloud operations
thanks to these few minutes together.
I really appreciate you taking a few minutes with us.
CHARITY MAJORS: This was super fun.
Thank you.
[MUSIC PLAYING]