Hello, everyone. Welcome to re:Invent
and to our special 'This is My Architecture' series.
We wanted to share a couple of 'This is My Architecture' sessions today
and provide some additional context on questions asked
and service updates in these videos.
We have two sessions lined up for you today.
Let's get started with Zoho.
When I talked to Raji on how they built Site24x7
a scalable, real-time user monitoring system on AWS.
Let's go watch it.
Hi, and welcome to another episode of 'This is My Architecture.'
I'm Aarthi from AWS,
and joining me in Mumbai today is Raji from Zoho.
Raji, thank you for joining us today.
Thank you, Aarthi.
So Raji, tell us what Zoho does, and what is your specific role?
Zoho has a suite of online applications
that are needed for end-to-end operation of any business.
And I'm a product manager in Zoho
for the product called Site24x7.
Site24x7 is the monitoring solution
that does monitoring for the full stack of your application
starting from your end user layer to your application layer
to platform layer to your infrastructure layer.
It gives the full, all-in-one monitoring solution.
And what are we going to talk about today?
You mentioned the monitoring solutions.
Is that what we're going to dive into today.
Monitoring solution, the Site24x7,
we run it on our service
And we wanted to do the real-user monitoring as a feature,
which-- we faced some problems when we had to run it on our own servers.
And that's when we wanted to use AWS.
And that's what we're going to talk about today.
Great, can you elaborate on some of the challenges
of running this real-user monitoring on-prem
before we dive into the architecture. -Yeah.
As I said, the real-user monitoring
is a real-time monitoring of end users' experience
as to how the application is behaving across geographies.
How the application is behaving--
How is it deferring with respect to ISPs?
How is it deferring with respect to browsers?
With respect to devices? There are so many things that we collect
from real-time users.
Having said that, this is a real-time environment.
We don't know the number of requests that we are going to get in.
So the volume, the elasticity, the scaling part
we can't handle it on our own servers,
and that's when we wanted to move to AWS.
Great, why don't we dive into the architecture
of how you do this real-user monitoring?
Yeah, so as I said, it starts with the user's browser.
So the user has to download the script into that browser.
So, for which the request is actually routed through Route 53
and it reaches the CDN.
So the request comes to CDN,
and then the actual script, the beacon script we call it,
is downloaded in the browser
and then that actually does the data collection part.
The various datas that I talked about.
So the beacon collects all the data
and as and when it collects, it, again, routes through Route 53
to the ALBs.
And from ALBs,
we just move it to the Collectors.
The Collectors are nothing but a group of EC2 instances,
which are auto scaled.
And from here, we actually push the data to SQS.
Okay, so once you collect all this data on the EC2 instances
and then push it to the NSQ/SQ, what happens after that?
And maybe before even I go to that,
I want to say why we are using this auto scaling environment
for the EC2 instances.
I said we'll get a huge amount of requests.
It's something like we get up to 23 million requests a day on normal days.
And, on peak days, peak load days, it actually doubles.
We get 50 million requests a day and 50 thousand requests per minute.
So such scale is possible only using the auto scale Collectors.
Now, after this collects the information and puts it into SQS
we have another set of EC2 instances, which are auto scaled
and that, we call it as Processors,
which, again, fetches the information from SQS
and does all the calculations that we want to do,
does all the processing.
And we use two components here.
One is for caching, other is for database storage.
And for caching, we use Redis
to do the-- have it in the cache and do the processing.
And finally store the data into Cassandra.
And this Cassandra is actually across multiple, availability zones,
so that we take care of high availability.
It's a cluster, so that we can take care of high availability of the data.
I think you bring a good point about the high availability architecture here
especially-- you talked about the scalability with auto scaling instances.
So is this entire architecture that we're seeing here,
is this running in a single region across availability zones?
Or how are your users directed to like different reg--
from different regions directed?
Yeah, so this is one architecture.
We have customers across the globe.
But then we don't use the same in multi-region.
We have the similar architecture
in different regions.
That's mainly because the data--
we have the data residing in these servers.
Those are, again, EC2 instances.
And there are a lot of regulations that come with each of the countries.
For example if you take the EU, (.eu)
GDPR has to be compliant
and you can't have your customer's data taken out of the EU borders.
So each and every country is coming up with their own regulations
and customers don't want their data to go out.
So that's the main reason we have the similar set of architectures
in different regions.
So when the user request comes in based on the domain name,
like .com, .eu, or .in, it gets routed
to the individual regions accordingly. -Perfect!
-That's how it is. -Okay.
So once the processor like completes all its function here
you then send data based on whether you're doing caching
or to the Redis or to Cassandra.
What happens in this side of the world?
Like I see a dashboard here--
Yeah, so all this data collected is being stored.
Then the user has to view it in the dashboard.
As I said, Site24x7 runs it on our own servers.
Now, the dashboard is also on the server.
We need to get the information, so those are APA requests,
that, again, comes through Route 53
to the ALBs
and from here to the processors.
And the processors fetches it from the Cassandra database
and gives the data to the user
for them to view the real-time experience of their applications.
Where exactly the problem, if there is any JavaScript errors, are there--
How the snapshot is. How the resources are loaded.
So all the information that we have collected
and stored it in the Cassandra
is viewed in the dashboard
using another-- That's the component there.
This is a neat architecture
especially having an operations person view real-time metrics
is super important to having a very reliable architecture.
What are some of the future enhancements
that you're thinking with this architecture?
So we have three things in mind.
One is this Route 53 component.
We can actually make use of the geo-based Route 53
even though we have this architecture in deployment in multiple regions.
We can make use of geo-based Route 53 to route it to that region,
so that's one thing that we have in plan.
The second thing is-- this is, again, a simple collector,
which just collects the information
and pushes the data to the SQS.
So I need not have these as an EC2 instance
or an application running in EC2 instance.
I can replace this with Lambda functions
that takes care of, again, pushing the data to SQS.
Again, Lambda function is elastically scalable as well.
And the third important thing is maintaining this deployment.
So we have to take care of updating the builds in each of these places,
and maintaining in multiple regions is, again, a headache for us.
So I'm thinking if we can use a CodeDeploy
to do all those automations for us
so it is easy for us.
-Automate and make it easy. -Yeah.
Raji, thank you for sharing how Site24x7 uses AWS
to do real-user monitoring.
Great architecture, and thank you for joining us today.
And thank you for watching 'This is My Architecture.'
See you soon!
And that's a wrap
to how Site24x7 built their monitoring systems on AWS.
Next, we move from monitoring to more security
with Ryan from William Hill,
talking about how they built DDoS Protection on AWS.
Let's go watch it.
Welcome to 'This is My Architecture.'
My name is Peter, and I'm here with Ryan from William Hill.
-Hey Ryan, good to have you here. -Thanks for having me.
Alright so what do you guys do?
So William Hill is one of the world's leading betting and gaming companies.
We provide gaming applications for customers all around the world
on many different platforms.
Alright, I know that you've been subject to a large DDoS attack some time ago.
And you've decided to move your security posture into the clouds.
What's the actual issue you've had to deal with?
So in 2016 we were the victims of a major DDoS attack,
which utilized one of the largest botnets of the time, the Mirai botnet,
and at its peak we were hit with over 177 cubits a second,
which impacted our applications.
Alright, that's something.
So you have chosen a couple of services. Let's talk through the architecture.
Sure, so in front of all of our applications,
we have CloudFront.
and we use CloudFront's partner applications
to mitigate against DDoS, so we use Shield Advanced
to be able to mitigate
both layer 3 and layer 7--
layer 3 and layer 4 DDoS attempts.
Things such as ACK floods or SYN floods.
Also low complexity layer 7 attacks,
such as HTTP floods.
For more complicated layer 7 attacks
we use WAF,
where we can create filters to block more complicated attacks.
This is a good time in the architecture to talk about Shield Advanced.
Ryan talks about how they used Shield Advanced
to protect against layer 3 and layer 4 attacks.
One recent feature we launched this year that may be of interest
is protection groups.
AWS Shield Advanced now allows you to bundle resources
like your CloudFront distribution, for example,
into these protection groups,
giving you a self-service way to customize the scope of detection
and mitigation for your application
by treating multiple resources as a single unit.
Let's go back to hearing
how they completed their security architecture framework.
So you used Shield Advanced to increase scale out
to swallow the incoming flood
and use WAF to make sure that you block.
Now there's more to this. Talk us through.
Sure, so from CloudFront and the protection that's therein,
we send the traffic across to Route 53,
where we have two separate, Route 53 hosted zones.
One for traffic going to an ELB.
For traffic going to an ELB,
we just send non-sensitive traffic,
But for more sensitive traffic,
we send the traffic across to an EC2 instance,
which hosts an F5 BIG-IP,
which is another form of WAF
and that adds an extra layer of protection
on top of the WAF we already have on CloudFront.
I see, so you do basically introduce another layer
to make sure that you are able to tackle even more
types and variations of attacks
with F5 being involved here.
I see this is happening within a VPC, so there's a segregation.
What happens to the rest of the services.
Yeah, so once the traffic goes through the VPC,
we send it across to a VPC peering,
which sends the traffic to one of our seven different channel accounts,
where our applications actually sit.
So from the VPC peering,
we see it goes across to the VPCs in those accounts,
which allows developers to create applications
with the DDoS protection already built in.
I see, so you do send filtered and non-filtered traffic
down here to your kind of cloud environment,
and there's on-prem involved as well.
I see Direct Connect. How does that collaborate?
Sure, yeah, so we're still in the process
of migrating to the cloud,
so we have a number of applications
that actually sit on our on-premise data centers.
So traffic from both the EC2 instances
and from the ALB
go through to Direct Connect,
which then allows our customers' traffic to go to our on-prem application servers.
Alright, okay. So we end with this part.
On-prem is involved.
But I see some more services. What about these?
Sure, yeah so for an added layer of protection
we send traffic-- we send logs of our traffic
from CloudFront over to an S3 bucket.
From there we then send the logs across to our on-premise analysis servers,
which use both proprietary and third-party analytics
and equations to be able to identify further malicious traffic.
Once that traffic is identified
we then send the IP addresses
back up to AWS
into this DynamoDB instance we see here.
And there's a-- Basically the IP addresses
are added to a table within DynamoDB,
both with an expiration date,
which due to the nature of IP addresses are changing constantly.
There is a Lambda function which then runs
that inspects that table
for any new entries or for any expired entries,
and then updates the WAF filters
with those IP addresses.
So it looks like a continuous improvement flow to your WAF filters.
Utilize the Direct Connect connection with the low latency and high bandwidth
to make sure that you have no losses in terms of connection.
And...yeah. Thank you, Ryan.
Thanks for sharing this very interesting solution.
And thanks for watching 'This is My Architecture.'
That's an impressive way to use Lambda functions
to inspect DynamoDB tables
and then update the WAF filters with the IP addresses.
One thing I want to add here
is with the latest version of AWS WAF
we support variable CIDR ranges for IP sets,
giving you the flexibility.
For IPv4, for instance, we support /1 to /32.
And for IPv6, we support /1 to /128.
The IP set match statement inspects the IP address of a web request
against a set of IP addresses and address ranges.
So you can use this to either allow or block web requests
based on the IP addresses.
One additional thing to note here.
By default, AWS WAF uses the IP address from the web request origin,
but you can definitely configure the rule
to use an HTTP header
like X-Forwarded-for instead.
And that's a wrap for 'This is My Architecture' session
at re:Invent.
We hope you enjoyed the two videos
and the additional information.
See you soon in another 'This is My Architecture' session.