Hello, everyone, I'm Seema.
Today we have Deanne, from Autodesk, with us.
Welcome, Deanne.
Hi, Seema, thank you, glad to be here.
Can you tell us a little bit more about Autodesk
and your use case?
Yeah, Autodesk provides design and makes software
for innovators, for them to achieve the new possible.
Okay.
For our use case, this is our data pipeline.
And we have data coming in to S3,
in the volume of 13 terabytes, daily.
Once the data comes in, we trigger
an event to simple notification service or SNS,
which then triggers invocations to the Lambdas that we have.
So our Lambdas do a lot of processing,
cleansing, the enrichment of the data,
and once those are done it actually writes metadata
into DynamoDB.
And that metadata is actually read by
Elastic Container Service or ECS,
in order for it to do batch processing.
Okay.
And then, once ECS processing is done,
it actually writes data back into S3
for our end users to be able to use that data.
OK, what kind of batch processing
does ECS do?
Yeah, sure, so, ECS is actually a
Async RESTful Web API Service
and what it does is that it batches altogether
the output JSONs from the Lambda
into a batch Parquet format for read efficiency.
OK, sure.
And how would SNS offer any other functionalities
that you're trying to achieve?
Yep, for SNS, actually, we use it in two ways.
One is this notification, and the other way is
these Lambdas, right, they actually have
an internal retry mechanism,
and once that retry mechanism, the processing still fails,
we actually trigger, send an event to SNS,
which then triggers another Lambda
to copy that erroneous file back into S3
for us to investigate what's going on with that file.
OK, so your data volume is always 13 terabytes?
That's an interesting question.
The data volume, it doesn't come in
into a stream like this, it actually has peaks and
troughs like that, so at one point in time
it can be as low as one terabyte,
and at its peak it can actually go up to 13 terabytes.
In fact, one of the challenges that we were
trying to solve with this serverless architecture
is because of this.
In our traditional Spark-based processing,
in order for us to be able to process the peak data volume,
we have to provision memory, in order to be able to
handle this, but in cases where
the data volume is at its low, it becomes
very inefficient and very costly.
That's the first, and the second one, actually,
is that as the data volume increases,
this is the data volume right now,
but it keeps growing and growing.
So, our Spark-based data pipelines
are starting to struggle in processing that data,
so the data processing becomes longer and longer
and our users have to end up waiting a longer time
before data becomes available.
Whereas with this new event-based architecture,
data actually is provided to our users
in near real time at a fraction of the cost.
OK, sure, and,
so, what are some of the challenges
or learnings that you have from this architecture,
and can you share with us?
Sure, this is actually an interesting use case
to be using Lambda for such a huge amount of data volume.
As with Lambda, it's not traditionally used for
big data processing like this,
but since it's an interesting use case
we just have to be mindful and smart about
how we're using Lambda, and the application code
that we have should be very efficient
in processing such huge amounts of data volume.
That's the first one, and the second challenge, actually,
that we met, this is not the original architecture,
initially we were using S3
to read and write the metadata.
However, when we were doing that, we found that
the Lambdas were taking around 15 minutes
to process and often times timing out.
And when we dug deeper, we found that, because of S3,
is not a service that is catered to
low latency high throughput operations.
So, we actually did some research and found
that we had to replace it with a service
that is fit for that purpose, which is DynamoDB.
DynamoDB is a service that handles
throughput and latency very well
and once we replaced that service,
we actually bring down our processing time
from 15 minutes down to 20 seconds.
OK.
Are there any enhancements that you're
planning for the architecture?
Yep, one enhancement is actually this,
right now, we plan to use, manage SQS
for better efficiency in handling the data
and handing the notifications.
That's one of the key changes to the architecture
that we want to make in the real near future.
OK, awesome.
So, I enjoyed this conversation,
hope you did enjoy as well.
Thank you so much.
'This is My Architecture'.