Hello and welcome to This is My Architecture.
I am Walid, and today, I am with
Jean-Baptiste, from Euler Hermes and Lucien, from Ippon.
Jean-Baptiste, tell us about Euler Hermes and your role in Euler Hermes.
So, Euler Hermes is a company specialised in credit insurance.
We are the world leader.
Credit insurance is an insurance protecting B2B companies against unpaid invoices.
In our industry, it means datas on companies,
and in today's use case, we have a lot of data
on companies.
Alright, tell us about your business need
and why this solution?
I am in charge at Euler Hermes
of the monolithic transformation using these company data
to solve insurance-related problems.
And our goal is to move this existing monolith,
and break it into microservices, and then move it to AWS.
So now, you collaborate with Ippon, which is an AWS partner, with Lucien.
- Hello Lucien. - Hello.
Tell us about your role at Ippon,
and tell us also about the technical solution
suggested by Euler Hermes. - Thank you Walid.
So, I am a Cloud Date Engineer at Ippon Technologies,
and I have been working with Euler Hermes for almost a year now
on all the data aspects linked to this monolith to microservice
transformational project. - Great.
The idea is simple, we go from a monolith
to a microservices solution, and we try to make the most of
the agility of AWS, the scalability of AWS in that regard.
Let's hear Lucien about
this process at Euler Hermes.
During the transformational process,
our microservices are developped on AWS,
but are based on data from Legacy
that already exist in an IBM DB2 database.
And in fact, my role during this mission is to
move the existing data from the old system
to the new system,
which are Dynamo DB for our reference database
and Elasticsearch for our search engines,
and to assure a synchronisation between the transformations
that would appear on the Legacy database to the the new system databases.
So we start with the database
construction phase.
So, it means in order to migrate our data
on Dynamo and Elasticsearch, it's crucial to
recover data from S3.
In order to do that, we used the AWS DMS,
that helps — with a few filters
and management policies — to create a copy of the original
DB2 database to a S3 bucket,
like this.
First step is to recover the data batches
directly from the monolith, from the IBM DB2 database.
We use the AWS DMS, Database Migration Services
to migrate the data on S3.
- Exactly. - Now the data are stored on S3.
What is the next step?
The next step is to load this massive amount of data
to our reference databases.
In order to do that, we chose Amazon EMR, Elastic MapReduce,
which is a Hadoop cluster,
that allows us to scale our cluster
based on the amount of data
and based on the computers power
to help with our needs in terms of performance.
The interesting thing about this EMR,
is that we use it temporarily.
So, we execute it only for this phase.
Once the data migration is completed,
this EMR cluster stops, and can therefore really adapt to
our needs.
And behind this EMR cluster,
we have a Spark job which can
write a regular ETL job , so it retrieves data from S3,
(so it's an extraction), that transforms them;
so with the transformations, we have cleanups,
we have data validation mechanisms,
we also have data quality tests;
and behind this, we store these prepared and refined data
in our two systems, which are
Dynamo DB and Elasticsearch.
Great.
Once the data are processed on S3,
we use transient clusters,
it means they shut down
once they are done.
We use Spark to do ETL.
We recover data from S3
and we write them both after transformation
on Dynamo DB and Elasticsearch. - Exactly.
Now, we have retrieved the data, the batch, so to say.
Now, what about the data that change the source database?
You are right.
We need to be able to pass on the changes
that arrive on the source system-- from the source system
to our reference systems of our project on AWS.
In order to do that, we create events
based on the commit log
and the IBM DB2 changes, which are used once again
by the DMS to create, and move an event
to Kinesis stream common to the entirety of Euler Hermès.
So, like this.
For this, we developed Lambdas
that subscribe to a Kinesis stream
and consume the data updates to modify
on Dynamo DB firstly, and secondly, on Elasticsearch.
To modify the data in Elasticsearch,
we use the function Dynamo Streams from Dynamo DB,
which is basically a small Kinesis associated
with a Dynamo DB table, that allows us, with a Lambda,
to react to modification on this table
and to apply these modifications in Elasticsearch.
For the Lambdas, it runs with Node
which allows us to read, transform the messages,
apply the same validation and cleansing mechanisms
we have at the initial loading level
to assure that the data, may they arrive via the initial batch
or via real time synchronization, have the same format
and the same quality.
Great. So, this branch is for real time.
So we retrieve the change logs from IBM DB2.
We retrieved them on Kinesis
and apply the same modifications that we applied
on the batch branch to write them at the end on Elasticsearch.
Exactly. And this guarantees
a synchronization mechanism almost in real time,
which allows the systems to be updated
and if the job runs an operation on the Legacy system
or on the new system, we have the same result
and we don't experience a desynchronization or a difference between the two systems.
Thank you Lucien. Jean-Baptiste, what is the final result?
And how can users use Elasticsearch then?
The final result is a Dynamo DB database
and an Elasticsearch database at a monolithic phase.
It took us five months of development to get there
and today, we are developping an API
that users and clients can use
or fronts that our users and clients can use
to find any data in our database
on all the companies of the world.
What benefits have you noticed?
The response time that took a few seconds
on the monolith now only takes hundreds of milliseconds.
Huge.
Thank you Jean-Baptiste, thank you Lucien
and thank you for watching This is My Architecture.