Hello everyone, I'm Seema. Today we have Vibin with us from HP.
Hi Vibin. - Hello, nice to be here.
Okay, so can you tell us a little more about HP and your use case?
Sure, so HP is one of the world's largest consumer device manufacturing companies
from PC's, laptops, printers.
And this use case is about ink cartridge manufacturing
and about getting data from the factory into this real-time pipeline system, right.
So from manufacturing data into a real-time pipeline system
where users can make use of it.
Okay, so I see a lot of interesting services out there, okay.
Yeah, so this like I show here is the entry point into this whole system,
behind this are manufacturing machines.
It generates a lot of data during the manufacturing process.
It comes into Kinesis... real-time streams come into Kinesis Data Streams.
From here, we send it to a processing unit, which is a Lambda DynamoDB combination.
Okay.
We also have data from Kinesis Data Streams that is being saved into S3, right.
For further processing later.
Now, this real-time processing unit is as you can see,
it goes through multiple stages.
So I've just shown one unit here, but basically what it does is
this business logic return Lambda that saves data into DynamoDB,
is then triggers the next Lambda.
So these are daisy change so that it forms multiple stages.
Now, what do these stages do? The first stage basically parses the data.
The second enriches it with reference data,
and the third normalizes it so that, you know,
data across multiple lines they all speak the same data model language.
So that's basically what it is, it's a chain of these units.
And how do you get to the second stage from the first one, like follow up?
Is there a trigger?
Yes, so when Lambda saves it to DynamoDB it basically triggers it,
which is DynamoDB Streams.
So Streams then call the next Lambda in the chain
and that does the second... enriching of data for example
among other things.
Saves it to DynamoDB, which calls the third and so on.
Okay, so these three blocks are part of your data pipeline.
I see an additional Lambda there, what is that for?
These are auxiliary Lambdas.
So these alone process the data and send it to the next stage,
but we need other Lambdas, so it's a bunch of (n) Lambdas basically that
take data from the DynamoDB Streams
and then passes it to Aurora as a data store
for some other use cases, it passes it to Elasticsearch as well,
and we also do send it to S3.
So there are multiple Lambdas that accomplish this.
And these three are the primary consumption areas for our business users.
Okay, so you again have that Lambda being triggered from your DynamoDB Streams,
and then your data being sent to Aurora and Elasticsearch.
Any particular reason of using Aurora here and Elasticsearch?
So well, this lends itself easy to be triggered from Lambda,
because Aurora Serverless, we're using the Serverless portion
and the Serverless has data APIs which Lambda can call.
And so that's a big benefit, we don't need to worry about
connection pulling and all for our system that scales up very quickly.
So Aurora Serverless was a perfect fit.
And this is also for our users who are more comfortable with SQL,
they might not be able to process data in S3 like data scientists do.
So this is of use for those kind of users.
Elasticsearch is more for real-time dashboards
because if I need to know where my part is in the whole manufacturing lifecycle,
this provides a good dashboard for users for those kinds of things.
Okay, I also see some Glue jobs and Athena there, what are those for?
Yes, so Glue jobs are... so we do store data in bulk in S3.
This is for later bulk processing by our data scientists and AML jobs.
Glue, what Glue does is it, it's a schedule Glue job that
bulk processes data in S3,
so we have it on a six hourly to a daily basis.
So this processes data, saves it in S3 and this is being used by users...
so there's an Athena layer on top of S3.
It's again SQL or what S3 data for users who actually do access the data in S3.
Okay, so they can write the doc queries
and get some data sets? - That's right.
Okay, that's interesting.
And was this your original choice or you explored any other options?
Well, no. Our original choice was...
It was a real-time platform, but we were using Kafka and Apache Storm.
Kafka was the ingestion portion of things
and Apache Storm was the real-time events engine.
But we found after sometime of operation that we had spent a lot of resources in just
maintaining the environment.
It was not managed, we were doing the management.
So there were a lot of administration issues.
Correct, and this being a manage environment,
it was easier for us to focus on the business logic then.
Okay, any key learnings that you can share with us?
Yes, for example, DynamoDB, we sometimes might go crazy in
streaming a lot of Lambdas to DynamoDB's
but that's not a good option, you had to limit it to two or three
otherwise you might end up throttling DynamoDB reeds.
Okay.
So best practice is limit it to two or three.
Now, in terms of managing Lambdas also, at the time of our development
Lambda came up with a new feature called concurrent batches per shard.
What that enables us to do is for each shard we could invoke multiple Lambdas,
thereby increasing the pararelism factor.
For a system like this, what we need is more pararelism, more throughput.
And so we got almost a 10 times increase in pararelism
when we enable that feature.
As well as, you need to also tune your batch sizes
and now going with the default always isn't the best option.
So tune your batch sizes according to your business use cases
and you might see that your throughput is increased.
Okay, so I think pipeline, and then modularization of Lambdas
that's some of the key things that you explored.
Yes, I think very interesting, I really enjoyed listening to it
and I hope our audience as well.
Thank you, thank you for watching 'This is My Architecture.'