Hi! Welcome to 'This is My Architecture'.
I'm Guillermo and today I am with Sergio Prada, from Lean Grids Services. Welcome, Sergio!
Thanks, Guillermo, for inviting me.
Tell us. What does your company do?
Lean Grid Services belongs to Everis Group
and we are dedicated to maintenance and adaptive solutions
for an important Spanish energy company
associated with maintaining their gas and electricity distribution network.
Seeking efficiency and proper maintenance
and to improve what is continuous, which are their systems.
And what does the architecture you're going to tell us about today consist of?
The architecture we have drawn here
is chiefly focused on data acquisition
for the formation of a data lake and to have a history of information
that lets us make different Machine Learning algorithms and such.
And to have the information organized, and, also,
have a homogeneous and standardized
way to store that information, and also consume it.
You talked about data acquisition, what data sources does this system support?
The data sources are quite diverse.
Actually you can see here that we have Kinesis as well as DMS.
With which, you can see, we have both events
and more static data sources
such as a database
or files, and although you don't see it here,
files are also taken from FTP.
In the DMS case we can also use
keynote captures in case it's necessary, we actually use it.
I see you have a data lake.
How are the data structured in this data lake?
And what tool do you use to transform those data?
The data, independently of the source they come from,
always end up in S3.
Specifically, they're going to end up in a landing zone
so that later this architecture, which we have over here,
lets us collect that landing data
and move it to a raw data layer.
As you can see, this architecture is focused on Step Functions with EMR.
EMR as a processing motor,
and Spark is what we use the most, and Step Functions to control EMR
so it's one piece, and although it's not serverless,
we treat it as serverless.
So, this triggers the processing, mostly batch oriented.
And what it does is obtain the landing information
to return it, normalized, in Parquet format.
This Parquet format, cataloged with Glue,
gives us access to the information and later also permits us
not only to catalog that information and normalize it,
but also to generate data entities
that could be of use to the company in general,
to generate those entities which are necessary.
You talk about processes in lots or in batch,
do you also support streaming processes?
Yes, what we do is take advantage, in this case, of Kinesis.
Here you see the Kinesis piece in general,
but we also use Data Analytics and Firehose.
You can do the analysis, and Firehose as well,
to return the result of that analysis also as one more landing,
one more intake, to what this architecture is.
Understood. Do the system users have some type of mechanism
to control all those processes?
Yes, what we really have is a web
which is developed with React,
what we do is it calls the different APIs from AWS
to get information in real time of how those processes are working.
In that way the user can see how it works,
and even, if necessary, cancel and launch the processes.
You've talked about intake, storage,
processing, management order.
How do you make this data available for the consumers?
This data, what we do is we take advantage,
as I said before, of the use of Glue as a catalog.
So we can have direct consumption from Athena
through the data catalog.
We can also get the information, and if it's necessary,
we can publish it within another data system,
in this case Redshift is the one we use the most,
but it could really be any database,
due to that information being needed in a concrete system,
due to its use case.
And then in turn, Athena
can also access the system, in case it's necessary,
to make Data Hub, access the information
- and not have different connections. - Understood.
And how many data consumers have you implemented in the system?
Mainly, we have three data consumers.
In principle, as you can see, you see here SageMaker and QuickSight.
SageMaker accessing data through Athena,
as well as through Redshift,
as, I'm going to draw it because it's pretty long, I'll draw over here,
even as an S3, depending on the algorithm we can use.
So this is used by data scientists
who need a big volume of information.
QuickSight, accessing mainly Redshift,
although it can also access Athena,
and later the same motor you see here with Step Functions and EMR,
which in turn, can use it to execute algorithms,
or execute any process of data transformation.
Any interesting use case that you've implemented with SageMaker that you can share with us?
Yes. The most interesting one I think we've done
is predictive maintenance.
We've developed an algorithm to detect
when components might fail, the probability of components failing,
developed mainly on SageMaker
and then running the resulting model on this same architecture.
Very interesting. Do you have any medium-term evolution of the system planned?
Yes, mainly we have an evolution
which is to try to adapt this architecture
so that it's more agile in the face of events,
as you know well, EMR has a start-up time
that makes this more event-oriented processing difficult for us,
with which we plan to insert a 2.0
and then take advantage of the improvements they are including within the EMR itself,
like Apache Hudi or Apache Iceberg.
Thanks so much, Sergio, for sharing this architecture with us today.
Thank you, Guillermo, for inviting me to come explain it.
And thanks, everyone, for watching 'This is My Architecture'.
Until next time.