- [Gonzalo] Hi, this is Gonzalo from the AWS Glue team
and today I wanna show you how to be more cost effective
by using the new cost saving features
just release on Glue Streaming ETL jobs.
Streaming ETL jobs in general are more challenging
than batch jobs from the cost point of view.
This is because they have a specific
and competing requirements that often require us
to make trade offs.
In most cases, the stream volume is variable
and where we might have some indication,
it will often constantly vary
and typically will have big periods combined
with others with little or even no traffic.
At the same time, it's common for streaming jobs
to have some required SLA, service level agreement,
that defines the maximum latency tolerated
from the moment the message is published
till the job has successfully processed it.
Even if this SLA is generous, it becomes a challenge
when we combined it with a highly variable volume of data.
On top of this, streaming jobs are expected
to run most of the time, we're not 24/7.
So a small difference in the cluster cost per hour
can make a big difference in the monthly bill.
Meeting all these requirements often means
you have to sacrifice cost effectiveness.
But we're gonna see now how we can reconciliate
all these things with the new features released.
The first feature is auto scaling.
Most streaming jobs don't have a constant volume
of data, therefore don't have
the same capacity requirements over time.
If you provision your capacity
to meet the SLA at peak times,
it means that during low volume periods
all that additional capacity will be wasted,
which is represented in the graph
by the orange areas that we see there.
By enabling Glue's Streaming auto scaling,
you can define a maximum capacity allowed
and let the job automatically adjust to the needs
and reduce the cluster size
where not needed to save on cost.
The second feature is a new type of a small working node,
a specific for streaming.
The new 0.25X is the quarter of a DPU in cost,
provides two cores and four gigabytes of RAM.
With this new node, the minimum cost of a cluster
is significantly reduced and this can be a good choice
for low demand use cases as well as jobs
with significant quiet periods.
Combined with auto scale, it means you can have a job
that is performing when is needed by adding more nodes,
but it's also highly cost effective
by not having excess of capacity when it's not required.
Okay, so let's create now a streaming job
to illustrate how we can use those new features.
So we are here in Glue Studio, create a new job,
now we are gonna change the source to a streaming source,
Kinesis, I have a stream already here defined,
now we are gonna put imagine these transformations
is wherever you need your job,
I'm gonna put this just as an example,
let's put the parent Kinesis
and then we are gonna save it into a bucket example.
So now we have our streaming job, we go to Job details
and automatically it has detected
that this is a streaming job due to the source.
And because it's a streaming job,
it has this new type of node are available,
so we can choose from the normal ones
plus this additional one special for streaming
to be more cost efficient
if the job doesn't need that capacity
and then we can enable auto scaling just by clicking here.
Now you will set the number of workers you need
the same as the regular job,
but when you enable auto scaling,
this changes to the maximum number of workers
instead of the request.
So it means that it's gonna use up to this number
of workers in total, but if we don't need that capacity,
it's gonna free nodes until it goes down
to the minimum of two, which is the driver and one worker
and that way you can save in capacity what is not needed.
Now, let's do some numbers for a sample streaming use case.
Imagine we have a job to process logs generated
by users on a website and we try to monitor
their experience in near real time.
This website is for small businesses in a region
and the vast majority of traffic happens
during standard business hours.
However, there are normally a few users working
at all times or even weekends
and exceptionally there can be something expected
so we want the service to be running at all times.
During performance testing,
we determined that during those business hours,
it needs four workers to be able to keep up
with the data streamed including another node
for the driver and using the default 1X nodes.
It means the computing cost
for this service running 24/7 is $1,584 per month.
Now, if we enable it to scaling with those same five nodes,
then we found that we only pay for two nodes all the time
and three nodes extra for working hours
and that brings down the cost
to a little over $950 per month.
Finally, during performance testing,
we see that at quiet times, the new 0.25X worker
can do the job as well, but during working hours,
we will need nine of those smaller nodes
to get the equivalent computer power of the old cluster.
Now, with this new configuration,
we found that the cost comes down
to a little bit over $343 a month.
That's a whopping 78% savings per month.
These teachers are already available
for you to try and start saving.
Thanks for listening.