- Hello and welcome to This Is My Architecture.
My name is Rudy and I'm here today with Franck from ENGIE Digital. Good morning Franck.
- Hello Rudy. Thank you for welcoming us today. - Welcome.
First of all, can you tell us a little bit about ENGIE Digital?
- Yes ENGIE Digital is the entity of the ENGIE Group that designs and develops software
to support the group in its zero-carbon transition.
- OK. You develop a number of platforms, one of which we're going to talk about today in particular.
- Yes, today we're going to talk about Smart O&M Operations and Maintenance,
which aims to carry out hyper-vision, ensure energy efficiency and
also to support people in the field on the maintenance
of energy equipment in the B2B world.
- OK. So, by equipment, you mean there is a notion of IoT, right?
- Yes. Well, we are connected with a number of IoTs, connected objects.
The purpose of the platform is to be able to consume all this data in real time
and to be able to create notifications and send alerts so that people
in the field can carry out maintenance.
But we also process the data in the form of a data lake
and we also perform machine learning and business intelligence operations.
- Are we talking about large volumes of messages? - We're talking about message volumes, currently
producing around 15 million messages per day, which we process
every day, with the aim of reaching
more than 500 million messages within two or three years.
Thus, a big scaling.
- Let's talk more in detail about the architecture, starting with Kinesis,
which I imagine you chose for its ability to scaling, is that right, to ingest?
- Yes, that's right. So the messages reach the Kinesis level, are stored
in the data lake which is based on S3. Then in parallel, each message is processed
by a set of micro services -- we currently have about thirty --
which process the messages and are able to notify
our users in real time. - OK. And so I guess the goal
is to be able to give the results of the calculations here in the form of APIs
to feed dashboards, web pages.
- Yes, that's right. And so we rely on the AWS API Gateway
to provide our customers with all of this data and to be able to interact with
the data, obviously. And we also have a whole web application,
a front end that encompasses all the services around our platform.
- All right.
I see logos for CloudFormation, Systems Manager. I can imagine...
the provisioning and configuration aspect of this platform, is it also important
for you to manage it in an industrial way?
- For us, it's a prerequisite that everything is automated.
So CloudFormation allows us to automate our entire architecture,
to provision all our environments.
Each environment is made up of about 70 stacks.
We are deployed on several AWS regions around the globe.
So, for us, using a central repository and a tool that allows us
to easily update our environments is really very, very important and mandatory.
And we also rely on Parameter Store to configure all our containers
in different ways depending on the environments we deploy them in.
- OK. I also see Spot logos here, Spot instances.
Can you tell us more about the issue of cost control for you?
- Yes, so, as in any digital platform and even more, I would say, in the world of energy services,
cost control is a major issue.
So we have chosen to use ECS with an Auto Scaling Group,
which itself relies on Spot instance fleets; this allows us to have
very low cost machines, at the best cost, thanks to the instance fleets.
- Can you explain to us how you ensure the availability of the platform
because one of the questions we often have at first sight with Spot instances
is how we can react in case instances are removed from the cluster?
- Yes, we do. -...because the cost passes a certain threshold.
- This is often something we fear. And thus, we want to both reduce the cost but
at the same time maximize availability.
And in order to achieve this goal, we use Fleets of instances with different
types of instances inside. This allows us to ensure that if one type of instance is unavailable,
we can eventually use the other types of instances. And the Auto Scaling Groups allow us to distribute
all this in different areas of availability in the region.
- Ok. Great. How much did you win? I mean, what cost savings
did you achieve on these Spots?
- Yeah, so actually the Auto Scaling Group and the Auto Scaling Group scheduling
allows us to turn off the machines at night and on weekends... -- yes, absolutely --
so that we can win on machines that are not of vital importance,
for example, on development environments, for example.
And for machines that are still running, so for example, machines that are critical
and critical services, we use Spots instances.
Overall, we've saved on all our EC2 instances compared to On Demand prices
by about 90% -- which amounts currently several tens of thousands of Euros.
With the scaling that we expect in two to three years, this will represent
several hundred thousand Euros in savings.
- Ok, that's super interesting.
What are the next steps? Do you have any on-going projects?
- Yes, we have a lot of projects in progress and in particular the improvement
of our data lake by relying on Group solutions also developed on AWS.
But also the improvement of our machine learning brick and
a whole topic around edge computing because we really believe
in processing data at edge.
- Ok. A lot of great projects in the pipeline. - Yes, indeed.
- Thank you Franck. Thank you for talking about this architecture with us.
- Thank you, Rudy.
- And thank you for watching this episode of This Is My Architecture.