Hi, and welcome to 'This is My Architecture'.
I'm Andy, and I'm here with Adnan from Flutura.
Hi, Adnan. Hi, Andy.
So Adnan, I see an architecture on here, but I also see some things are going on.
But before we get into this, if you don't mind, can you tell me about Flutura?
Sure, Flutura VR and industrial AI and IoT company,
We provide digital assistance to our customers,
that helps them improve operational efficiency and asset uptime.
Excellent, so if you can, kind of give me an idea here.
So you have an oil derrick, how does that relate back to the rest of this architecture?
Sure, so we had a customer who was facing production deferment in the gas fields.
So they were looking for a mobile- based solution
that can predict anomalies in the gas field before they happen.
So we took the data from there, and we used this architecture
to implement that with the models.
Then how do you get the data from the actual device into your architecture here?
Sure, so the data from the control system and S devices
come to our VPC, that's in the cloud,
through MQTT or MQTT connectors.
And then the data is pushed by those connectors on the Kafka topic,
which is we are using MSK to manage the Kafka topics.
Once the data is in the MSK cluster,
the Lambda triggers the Spark job running on the EMR cluster, right?
That Lambda is set up to trigger every minute, it's on top of a minute, it runs that.
The Spark job is our data parser, which reads the data from the Kafka topic,
cleans the data, perform data validation operations.
And then write over open TSDB, which is our time series database,
that's open source and that sits on top of HBase.
And we run the HBase as well on the same EMR cluster.
So the open TSDB writes to the cluster,
Sorry, the HBase and then that data gets written back to the S3 buckets.
So this is your sanitized data then,
that's been through the processing jobs from your cluster?
Correct.
So I see EMR is really at the center of this here.
So why did you choose EMR?
Sure, so EMR, we were looking for a platform that can run Spark and HBase,
since we use that heavily,
and EMR kind of suited our needs very well, because of the scalable architecture.
So during our peak needs, we can increase the computation power
without changing the storage.
And then during the normal hours, we bring that back to normal cluster size.
And as we are growing in the data,
we are slowly increasing the storage without changing the computation.
So that helps a lot for us. Excellent.
So after you have your sanitized data, what happens next?
Sure, so that parser, that's after cleaning the data writing to the database,
it also publishes the data back to another topic on the Kafka.
From there, our data analytical engine
runs that and reads the data from the next topic.
And that's evaluating the rules, there are thousands of rules that gets evaluated in a minute.
And then the notification of that goes to the user as well, right?
And also, we have machine learning models that are running
and executing in the EC2 instances,
that reads the data from, the real time data from MSK,
and then the historical data from open TSDB, which basically is data sitting in S3.
And the contextual information, which is the equipment information,
such as digital twin configuration,
from the Postgre database that we have.
Using all this information it predicts the values,
and this could be a set point prediction,
for equipment, it could be a failure prediction,
or any prediction that we would like to build a model for.
Then these predictions are presented to the user
through our web application that's running on the EC2 instances.
And also, they can register for notification through SMS or emails.
And after that, some of our customers,
we also write the data back to the field that forms the closed loop.
So control system can accept the data
and change the set points for operational efficiency of the equipment.
So I'm just curious here then.
So for the amount of data that's moving in here,
can you give us a sense of how much that you're dealing with?
Yeah, with one of our customers, we recorded the number of data points.
So we got about a billion records per day.
That's the throughput of the system that we noticed.
And then the rolls that we were evaluating was over 50k rolls in a minute.
Wow, thank you for sharing that story, Adnan. My pleasure.
And thank you for watching 'This is My Architecture'.