hello everyone, welcome to reach goals today I am going to talk about logging
in distributed systems there are a lot of questions surrounding it and let's
see one by one right so the first question is we already talked about how
to login us in the distributed system and I am going to talk about two
different ways the other one is how to count the specific error and generate
the reports for example if you want to know how many are four zero one errors
or how many errors are happen due to the order calculation or how many of the
errors are due to the tax calculation etcetera right also finally I am going
to talk about three real production issues and the solutions so let us talk
about logging in the distributed system so normally if you look under the
distributed system you have a number of clients across the globe and they will
connect to the load balancers and the load balancer in turn connects to the
multiple nodes which are situated across the globe right so something something
what you see on the screen so if you look over here all the logs will be
accumulated into that respective machines right so now if I want to get
certain information from the log it's very difficult for me to go into each
and every node and figure out what actually the log or what actually the
log error messages has so now the right way for me is I have to take all these
log files or the log messages and send it across this centralized location so
how did I do that you will be having something called as a agent at each and
every node and they in turn it takes all these log messages and it sends to the
centralized location so the satellite location is nothing but it is a server
or a cluster which is made of log stash right so finally in the log stash I can
click connect to the elastic elastic server or relax stick search right from
there what I can do is I can use the visualization tools like Kim Bona and
they can start searching the files which which is which is already there in the
log stash and I can use it to index that right so once the files are indexed I
can get the right information from the logs that is the type one or the way
which which is commonly used across the industry right now if I want to look
into the type 2 or the different way to do that what I can do is instead of
having the file generated and sending across to the centralized location
directly I can have a kafka queue or rabid queue
connecting from the logs and I can subscribe to that and I can send all the
messages to the log stash right or I can subscribe directly to get some relevant
log messages for example as I talk if I want to get the 401 error or if I want
to count number of it is happening in the order calculation then what I can do
is I can specifically subscribe with those messages and and I can get those
information and create a report out of that right
so there are totally two ways one is directly connecting from the machines
and having to the centralized location that's type one and the second one is
you can have a queue and from that queue you can pass it to the log stash and it
can do the similar similar mechanism to have a elasticsearch and you can use the
combiner tool to visualize that right so if you look at what is the best option
whether to go in the type 1 or the type 2 I would say the type 2 is the right
one because if you have multiple servers and it's a one server goes down what
happens is you will not be able to see the files or the logs generated from the
that server which is transferred to the centralized lot let's say we have a
queue it's very easy for you to scale it and scale up based on that information
or based on that mechanism you can get the logs into the queue and it can
finally transfer it into that log stash right so if you have a very wave if you
have a very traffic consuming system where if you want to scale it up or
scale it down I would prefer to go with the mechanism where you involve a queue
in between to get the log informations right so the next thing I want to talk
about is how to how to have a write log or how to have enough information in the
log for you to get searched right I'm going to talk about five different
points over here let us do one by one right so let's say if you want to send a
log message and the log message should have the correlation ID correlation ID
is nothing but it's a unique ID which goes from the client location and it
traverses across the in location where if you want to get any information
related to the log you can use the correlation ID to search that and get it
right the second one is response a response should have a unique ID unique
ID is nothing but it is the same as a correlation ID let's say if you want to
get some information related
and if you have the correlation ID and you can clearly track where the request
came from and how it got generated and where the error was coming up and
finally how it shows up in your response right and the third one is standard log
format it's have a standardized service so let's say you have multiple services
in a distributed system right each and every will every service will be
generating its own log formats now if you have a different form it's that if
it's very difficult to index and search it so that's why I prefer to have a
standard service in the standard service what you can do is you can convert or
you can have a wrapper kind of thing to transfer the or the log information and
can have a unique mechanism to create a standard standard log structure and it
can be sent into the centralized log server right so the next one is the
request context the request context is nothing but it should have all the
information related to the date time stack errors and service name and also
you should have an IP address and HTTP code etc if you have this context it is
easy to figure out what message you're looking for and what information you are
trying to extract from the logs right and it is also easy to do a indexing of
the log files and the final one is log should be searchable as I said in the
previous one if you have all this informations and if you have a key value
pair of kind it is very easy for you to search and get the relevant information
right so let us talk about the real production issues and how to solve that
the first one is the PA issues right so if you look at to do what PA is it's a
personally identifiable information right so in the logs if you have
informations like the the email ID or your name or your address then those are
all just called as a PA information so if you if you're creating a log
information you have to make sure you should not have those informations right
that is a security constraint the second one is storing the sensitive information
the logs I would say should not even store any sensitive information in the
logs for example and the sensitive information could be the your password
or the social security number or any other information which are related to
the user can be sensitive information right so if you have those information
it is it is a huge security risk and the other one is this is a performer
thing right excess logs impacts performance so you had a very clear what
kind of messages you are storing in the logs right so as I said you know if you
have a lot of log messages generated from your application or from your
services it's going to create a extra call to the backend where it gets to a
lot of log informations or log messages so that is rely there's really a
performance impact for your overall application right so you have to be very
clear whenever you do any log messages or when you add something extra to the
log you have to undergo the performance testing to make sure we are meeting all
the standards and is the least right thanks for watching