Hello everyone, welcome to reach goals. Today, I am going to talk about how to
distribute the cash and about consistent hashing with real world example .We
have an initial requirement in that we have a file which is a of very large size
which has 1 billion lines and in that we know we have a content like key and
value the key is a integer and the value is an IP address. right? so now our
requirement is, we have to take this file and distribute across
multiple caches so that we can read it very easily. right?
Since it's a large file, we cannot put in one machine, we are going
to put in four different machines. so the machine 0 will have the records
from 1 to 250 millionth line and machine 5 will have their records from
250 million to 500 million and machine 7 will have 500 1 to 750 million and
machine 10 will have 750 1 to 1 million records. right? so now if you have the
content distributed like this. Load balancer or the client can access the
different machines and it can get the content very easily. right? so the now the
load is equally distributed. right? so we can also have an algorithm in the load
balancer to connect different machines. space to another need correct that is a
configurable thing or you can have an algorithm in a load balancer based on
your need right in this we have configured such a way it connects to the
4 different machines right. After some point of time of a requirement slightly
changes, let's say what the change so we notice to the load on the machine 7 is
too high right so the number of requests coming to the machine 7 is too high so
we want to add one more machine which is machine 6 you can see in the screen
machine 6 the new machine now what happens here is our load balancer is
already configured to connect only to four different machines and now if he
wants to connect to 5th machine the data in the machines has to be equally
distributed or it has to be equally shared. right? so now when we have a new
machine the load balance that even though it connects to the new machine it
will not have a content so we have to redistribute the key values or the
content which are in the cache service right so how it happens over here is if
you look at the screen one the change happens in the machine 6 and
machine 7 it is not happening across all the machines right so that is what we
call it as a consistent hashing right so in the consistent hashing what actually
happens is we don't want to change all the keys or all the values in different
cache servers and it is only changed to specific service where there is a
minimum impact there is a little bit of formula over that and you can see that
in the next screen correct so here is a definition of consistent hashing if you
look into the red color what it says is when a hash table is resized and the
consistent hashing is used only K by n Keys needs to be remapped an average
where K is the number of keys and n is the number of slots that's what we did
in the previous screen like you know when the machine seven or machine six
has been added the change has happened only in the machine six and machine
seven not across all the machines right so that is what of how we apply the
consistent hashing for a distributed cache or when you want to have a
distributed load correct and the concept of consistent hashing is used in across
multiple companies like you know Netflix it is used in Facebook it is used in
Akamai etc right let's talk about a real-world example considering the
Netflix right so you have n number of machines connecting into the load
balancers and load balancers in turn connect to the different machines right
so each machine has its own movies and they would be having the movie ID as one
to 250 in machines 0 and 250 1 to 500 in machine 5 and 501 to 750 in machine 7
and 751 2000 in machine 10 right so then now all the load is distributed across
the globe and people are watching the movie very happily without any buffer
right all of the sudden you came to know that the machine 7 is getting heavily
loaded like you know machie 7 might be having a popular movie right at one
point of time lot of people want to see the same movie and all the hits will go
into the machine 7 which has the movies which are very popular maybe the number
ranging from 500 to sorry 501 to 750 right so now what we have to do is we
have to add one more machine? so now I will add one more machine
look at the green color which is connecting to the other machine which is
machine six right so if you have a new machine the content in the machine seven
has to be equally distributed or it has to be distributed into the other machine
which is machine six right so now if you see the movie ID ranging from five zero
one to six hundred goes into the machine six and the Machine seven will have the
movie ID from six zero one to seven fifty right so here there is a challenge
right so earlier we have can't figured out a load balancer to connect to four
different machines right now we have five different machines and the load
balancer has to be reconfigured to connect into five different machines
right so when the load balancer is connecting into the new machine it will
not have a content right before it gets connected the content has to be equally
distributed right so in this kind of a model what we are doing is we don't want
to add or we don't want to redistribute the load across all the machines instead
we are redistributing the load only between the machine six and seven right
this is the concept where they are calling it as a consistent hashing right
so we can treat this as a ring correct machine 1 to 10 can be connected as
Irving and you can see that is a circle visually right that is what they call it
as a consisted hash string right so this is a very popular concept which is used
in all the load balancers or when you wanted to have a distributed cache or
when you want to distribute the content across the globe with a minimum impact
to the existing machines then we call it as a mechanism we use it to have a
consistent load balancing. Right that's what I have. Thanks for watching