>> Hello, everyone, and thank you for being here, and I hope you are enjoying as much
as I am this conference.
So my name, as Meri says, is Maria.
I work for ThoughtWorks, we are a global consultancy and I'm going to talk to you about observable
microservices.
But before I do that, let me show you a picture, this is a picture of a place in New Zealand,
I went here last December with my boyfriend, and we like a lot of hiking and stuff and
New Zealand is an amazing country and I think everyone should go there at least once and
one of the amazing things about the country is that it's where the Lord of the rings with
as filmed.
We had to spend the night in Mordor, that is really cool.
And if you look at this really dark place where our friends Frodo and Sam go to destroy
the ring.
There are Orks all over the place, who try to kill them.
They have Golum who wants to steal the ring from them.
They have the really big spider and they wanted the ring back, to like, you dominate the world
and whatever.
So it's a pretty, pretty hostile environment.
There is an environment that us developers face almost every day, too, and that's production.
Right?
So production is a really hostile environment, you got like bugs, you've got security breaches,
you've got security overload and just weird behavior that you haven't seen anywhere else,
so yeah, that's production.
And in the good old days of the monolith, you have, you know, a handful of applications,
maybe that you need to maintain in production, and you kind of went by all of your little
tricks to monitor the systems and make sure that they were OK, but now we are in the land
of you know, distributed computing, so our problems are multiplied by 10 or by 100, so
it's a much more complex situation.
So a lot of the techniques that we use for the monolith are just either not applicable
for microservices or they need to be rethinked or and we need to get better at them and that's
what this talk is about.
Is it about those techniques and practices and I'm going to cover around three topics.
They are discovery, monitoring and debugging.
So let's start with the first one.
Discovery.
So you know, we have our clients, it ab an application, it can be another service that
is a client and it's requesting some information from a service, right?
And this is an area, a service has a well-known IP.
And it's a static IP.
So my client know wheres to find it, but that'sot always the case.
Sometimes you might be in the cloud or you might have some dynamic IPs and so that service
might not always have the same one.
And you might want to take advantage of one of the advantages which is scalability and
you might want to scale up that service and so now you have three or whatever instances
of the service and your client needs to decide which one needs to go on call to get that
information that he needs.
So this is a fairly common problem that has a fairly common solution, and it's the discovery
-- the service discovery pattern and that can be applied in two ways.
One of them you basically have a router which normally is a load balancer, and that load
balancer will call something called a service registry, which is basically where your services
will go and say, hey, this is where I live and these are my instances, and so the router
will call that service, get all the IPs, and it will get the balancing staff to figure
out which service to call.
It's called the server-side discovery.
There's another one called the client-side discovery.
The client will basically call the server registry to get that information and find
out what is the instance of the service that he needs to call, which is the client-side
discovery.
Some of the tools that we use are like supreme boot.
Supreme boot has a module or a library or whatever you want to call it that will allow
you to release it in an easy way.
So it's up to you which of the two you want to prefer.
One is the concept of the service registry and this is as I said before, a place where
you can store every service, where it's located what are the instances we have.
And it can be really a static file or it can be something complex like a distributed data
store which allows you to do service discovery and service registry but it also gives you
some kind of configuration management tool.
So the next thing is monitoring, and monitoring is important, because we need to know what's
going on in production.
We need to be aware of how our systems and our applications and our services are behaving.
And again, when you have just one application, that kind of was a manageable, you could call
to the host, you could like distract the logs and work from there, but like when you have
multiple services or multiple systems, then there are also different technologies, different
stacks, things start to get a bit fussy and difficult.
So there are a lot of things that we can do and I'm going to show you where to feel them.
A really good place is to stop having all of these logs all over the place and have
them in one single place, like aggregate them in one thing.
There's a really simple step, and you can use something like like elastic which is really
popular and will allow you to do this.
One thing is before you do that make sure that you have a conversation with everyone
when you actually kind of agree on what is the format of that log, and to have some consistency
across the board.
Because then you know, and you can have them all aggregated in one place, but if everyone
decides to do logs in a different way, it's going to be really difficult for you to go
through all of these logs and set information that is useful.
So that's one thing.
Which I say is fairly easy.
There's another one, these metrics you might want to notice, how the servers are acting
and how our services are functioning and what some of the metrics that will tell you whether
they're functioning or not.
Again, the micro-frameworks or frameworks of microservices test is, I don't know, whatever
is the framework that you're using to do the microservices will also give you something
really simple where you can either set up a metrics set point or you can have it for
free or like out of the box.
And the agreement is the same thing, aggregate all of that stuff and then send to a place
where you can then review, view, and then you know, look for patterns whatever that
is, that can be -- this one is the dashboard that Prometheus gives you, but you can use
whatever works for you basically.
And then you can go a bit further and also add to that monitoring all of your health
checks and how healthy your services are.
I'm pretty sure everyone has heard of health checks but basically they assess the health
of your system or your service.
They are really simple ones which just go and say, hey, is this application running
and they might just do a ping to that IP and that port.
They might be enough in some cases but in some cases you might not be.
In some cases you might have an you know, an FTP folder that the service needs to access,
so it is important that your health checks also look for the healthiness of all the dependencies
that your service has.
And you aggregate them, also.
And the last thing that I think we need to monitor is something called -- we want to
use something called syntactic or semantic monitoring.
So all of the things that we were talking about to now were individual services and
the data that they get or the data that the services will give us, that's how we make
them more observable, right, we can observe them and they give us information that we
need, but you know in your system as a whole, will be, will be performed by different services
talking to each other, so a business flow will span multiple services, right?
So we want to also make sure that that is working as a spectator, and it is working
within, you know, the times that we have stip lated that they're right, and all that step,
so synthetic or semantic monitoring kind of look into that tell you that you already have
usage flows or whatever you want to call them that you run against your static environment
and you can doing continuous delivery, you will run those tests in every commit or at
least every time you go to do a release or whatever, every so often.
So why would you take a subset of them that are representative of the functionality that
your application has and run them against production, and then save those details in
whatever monitoring tool that you're use, so that way you will have highlight any possible
errors that otherwise might have gone unnoticed, and the other thing is it will also allow
you to see patterns of like, hey, the response time for this flow was, you know, that met
a month ago now has gone up by X percent so maybe there's something that I need to know
and see what's going on in there.
And now that you have all of this information in a centralized place or at least in some
kind of tool that will allow you to see and look add spot patterns, then the next thing
that you want to do is actually just, you know, stack alerts that can tell you when
something is failed or when something is not right and alerts are really good, but there's
also a thing with them and is that they can get really noisy.
You can get a lot of alerts, and it's really difficult sometimes to differentiate when
an alert is important or that you can pay less attention or is less important.
So it's important to create a margin alert.
It will be kind of an evolution and that's OK, it will be perfectly fine, because you
will be smarter the next week than you are today, so that's cool, so dual alerts, you
might have some different type of alert if one of your services is giving you a 500 once,
that you know, this service has given you a 500 in every single request in the last
half hour, right?
So they might be different types of alerts.
So that is also really important.
Kind of the last thing that I want to talk to you about is debugging.
So we've been, you know, making sure that we can keep track of our microservices with
service discovery, you and you know, some scalability, also, we make sure that we have
all the information in the world for monitoring and we also have alerts that tell us when
something is wrong, and all of this has been in preparation for when the moment will happen
that you know you will find a bug in production, there will be an issue that needs to be fixed
and you will need to debug your production systems, and you know, in a monolith or in
a nondistributed application, that's kind of easier, you can have a stack trace and
you can kind of go through the stack trace and see where the thing happened, but in microservices,
you might have a request that has spanned multiple services, so it might be pretty tricky
to understand where that error was originated, so there's a simple thing that we can do that
can help us a lot with this and add correlation ID.
And an correlation ID is just an ID that you tie your request when you follow your system
and it will follow the request to wherever it goes and it can be used for a lot of things
but for debugging it's extremely useful because as I said before, it will allow you to understand
where that request got in and hopefully you will be able to find the root of the problem.
Whew, so, we have reached the end of the talk, and before I go I would like to just sum up
what I just said.
We talked about discovery, monitoring and debugging, as things that we as developers
and engineers, we need to either get better at or start applying if we want to succeed
to maintaining distributed or micro-service architecture in production.
And that was me.
Thank you very much.
[applause] I'll be around and if you have any questions,
so yeah, please come and talk to me.
Also you want to know what a lead developer at ThoughtWorks does.
So yeah, thank you.
MERI: Thank you, very much, Maria.
Maria submitted a 30-minute talk, and I went back and said, I'd really like to have this
talk, but could you do that in half the time?
She did an amazing job.