PRIYANKA VERGADIA: Are you looking
to optimize application capacity?
STEPHANIE WONG: Amongst all the other things,
global load balancing is definitely one to check out.
PRIYANKA VERGADIA: Well, stay tuned
to learn more in this episode of "Get Cooking In Cloud."
[MUSIC PLAYING]
Last time, we talked about load balancing
on GCP, including HTTPS and network load balancing.
There are a lot of ways to setup a load balancer
and back end instances to support something
like Beyond Treat's growing e-commerce website.
STEPHANIE WONG: They've become a national hit
and need to start scaling their website back
ends to meet demand.
One thing to highlight is GCP's compute engine
manage instance groups to take advantage of auto scaling
capabilities.
This lets you automatically add or delete
instances from a managed instance group
based on increases or decreases in load.
PRIYANKA VERGADIA: Auto scaling helps your applications
gracefully handle increase in traffic
and reduce cost when the need for resources is lower.
You define the auto scaling policy
and the auto scaler performs automatic scaling
based on the measured load.
STEPHANIE WONG: Check out episodes 4 and 5
from season one of this series for more on auto scaling.
PRIYANKA VERGADIA: But scaling global applications
can be challenging, especially if you
have limited IT budgets and unpredictable and burst-y
workloads.
You have holidays in each country
that spikes demand for treat orders at different times.
On Google Cloud, the flexibility provided
by features like auto scaling and load balancing can help.
However, auto scalers have some limitations.
STEPHANIE WONG: There are five major issues
that come up with auto scalers.
The most common issue is that your auto
scaled new back end running your website
isn't ready to serve your traffic fast enough.
You might have startup scripts that
need to be loaded before VM instances are ready,
meaning traffic ends up at existing instances
that are over capacity.
PRIYANKA VERGADIA: Second issue is that some applications
cannot be auto scaled at all.
For example, databases often have limited back end capacity.
Only a specific number of front ends
can access a database that doesn't scale horizontally.
STEPHANIE WONG: The third issue is
that your VM back ends might have too little headroom
capacity.
To account for a sudden burst of traffic,
an auto scaler should include ample headroom for example,
you can set the auto scaler to go into effect
when your back end reaches 70% CPU utilization.
To save costs, you might be tempted
to set this target higher, like 90%.
But this can result in scaling bottlenecks
when confronted with bursts of traffic,
like an advertising campaign that suddenly increases demand.
PRIYANKA VERGADIA: Fourth issue is regional coders.
If you have unexpected bursts in a region,
your resource coders in that region
might limit the number of instances you can scale to.
STEPHANIE WONG: And the fifth issue
is that you might have non-elastic licenses.
When you use licensed software, you
might have a preset maximum capacity.
So you might not be able to auto scale
and run that software on more than let's say 5 instances.
PRIYANKA VERGADIA: You can address these challenges
with GCP's global load balancing solutions,
like HTTPS load balancing, SSL, and TCP proxy load balancing.
First, let's walk through a recipe
of how our global load balancers handle bursts in traffic.
STEPHANIE WONG: As you may know, typical load balancers
use an algorithmic approach to distribute traffic
to back end servers.
Like round robin, where packets are equally distributed between
all back ends regardless of the packet source and destination.
PRIYANKA VERGADIA: Or hashing, where packet flows
are identified based on hashes of traffic information,
including source IP, destination IP, port, and protocol.
All traffic that produces the same hash value
flows to the same back end.
STEPHANIE WONG: Today, we're going
to focus on Google's global HTTPS load balancer
for layer 4 HTTP or HTTPS traffic.
Typical HTTPS load balancers use a DNS-based algorithm
serving to different load balancers in each region.
Failover to another region is possible,
but it can take minutes based on time
to live or TTL configurations.
Generally, a small amount of traffic
is still directed to old servers well past the TTL.
This makes it not ideal for burst-y loads.
PRIYANKA VERGADIA: GCP's global HTTPS
a load balancer uses a different approach.
Traffic is proxied through the Google front end or GFE servers
deployed throughout most of the global network edge locations.
Then the load balancer algorithm is applied at GFE servers.
STEPHANIE WONG: Here's how it works.
The HTTPS load balancer uses a special algorithm
called waterfall by region.
Traffic that goes through the load balancer IP address
is sent through a proxy to the back end instances.
The algorithm determines an optimal back end
by taking into account the proximity of the instances
to the users, the incoming load, and the available capacity
of the back ends in each region and zone.
The load balancer then distributes traffic
based on available instances.
To add new instances based on the load,
the algorithm works in conjunction
with auto scaling instance groups.
PRIYANKA VERGADIA: OK.
Let's walk through some scenarios
when Beyond Treat's traffic load becomes
too much for its managed instance
group to handle in one region.
First, let's talk about the traffic flow in one region.
Under normal circumstances, all traffic
is sent to the region closest to the user.
Then the following happens.
Traffic is distributed across instance groups,
which can be in multiple zones.
If capacity is unequal between zones,
zones are loaded in proportion to their available serving
capacity.
Within zones, requests are spread evenly
over the instances in each instance group.
STEPHANIE WONG: If an entire region reaches capacity
because the capacity limits you assigned to an instance group,
the waterfall by region algorithm is triggered
and traffic overflows to the next closest region that
has available capacity.
As each region reaches capacity, traffic
spills over to the next closest region, and so on.
PRIYANKA VERGADIA: Let's say you have unhealthy back ends
and you need cross regional overflow.
If the health check sees more than half
of the back ends in the region are unhealthy,
the GFE servers preemptively overflows some traffic
to the next closest region.
That way you don't have an entire region failing
as the region becomes unhealthy.
STEPHANIE WONG: OK.
Now let's say all the regions you have instance groups in
are above capacity.
Traffic is then balanced so every region
is at the same relative level of overflow
compared to its capacity.
For example, if global demand exceeds global capacity by 20%,
traffic is distributed so that all regions receive requests
at 20% over regional capacity while keeping traffic
as local as possible.
PRIYANKA VERGADIA: So how does using HTTPS load
balancing actually address the five capacity challenges
we talked about in the beginning.
STEPHANIE WONG: Let's see how.
Now that we've discussed how the GFE and waterfall by region
algorithm works, the HTTPS TCP proxy and the SSL proxy load
balancers can overflow capacity to other regions.
This means that it's a good idea to use a global load
balancer over the regional load balancers, especially when it
comes to global e-commerce sites like Beyond Treat.
Apps that use regional back ends have nominally lower latency,
but they can become overloaded very easily.
PRIYANKA VERGADIA: For latency in starting new instances,
if the auto scaler cannot add capacity fast enough during
bursts, the HTTPS load balancer will temporarily overflow
connections to the next closest region.
As soon as back end instances are scaled up
in the original region, new traffic
is routed to region closest to the users again.
STEPHANIE WONG: For apps like databases limited by back end
capacity, you can still overflow traffic
to the next closest region when demand in one region
is beyond capacity deployed for usual traffic needs.
PRIYANKA VERGADIA: As for too little VM headroom,
regional overflow still saves money
because you can either setup auto scaling with a high CPU
usage trigger, or make back end capacity well under a region's
capacity since overflow to other regions
ensure global capacity will always be sufficient.
STEPHANIE WONG: Now for regional quotas,
if compute engine resource quotas don't match demand,
load balancing overflow automatically redirects
part of the traffic to a region that can still scale
within its regional quota.
PRIYANKA VERGADIA: In the case of licenses,
the load balancer can move traffic to a region
where licenses are available.
You just set the maximum number of instances to the same
as the maximum number of licenses on the auto scaler.
STEPHANIE WONG: It's key to point out
that you do need a set up and manage instance
groups in the regions you feel are appropriate for your user
base.
PRIYANKA VERGADIA: You have to also take
into account things like app licenses and dependencies
like database scaling to meet the demand of auto
scaled website servers.
STEPHANIE WONG: For more information
about GCP's global load balancing,
check out the solution article by our friend, [INAUDIBLE]..
Link below.
PRIYANKA VERGADIA: That's all for today
on Get Cooking In Cloud.
Hopefully we were able to intrigue you
with ingredients and recipe to whip up
a resilient and performing dessert site for our global fan
base.
STEPHANIE WONG: If you liked this video
and would like to see more of this content,
don't forget to like and subscribe to our channel.
[MUSIC PLAYING]