Hello and welcome to another episode of 'This is My Architecture'.
I am Sonika, and today I have Vijay with me from WEGo T.
- Hello, Vijay. - Hi, Sonika.
- Thank you for sharing your time here. - Thanks for having us here.
Can you tell us a bit about WEGoT?
WEGoT stands for water, electricity, and gas of things. It's an IoT platform built to measure and monitor utilities.
Oh, that's great. Looks to be something with sustainability, right?
Right.
Okay, so can you tell us what are we talking about today?
So today I would like to walk through the WEGoT aqua which is the water management platform of WEGoT.
Oh, so let's straight dive into the architecture.
Sure, so before we start, I would just let you know that the architecture is split into three phases.
So we have the ingestion layer which takes care of the data coming in.
We've got the pre-processing layer which manages the data and the post-processing layer that saves the data.
So today what we are talking about is close to 100K sensors.
These are water meters out in the field, transmitting data to the AWS cloud.
So the first point of contact is the AWS API gateway.
Which then forwards the information to a bunch of transformers.
These are nothing but Lambdas that take the ingestion data and then process them.
Why are they called transformers?
We have got three generations of sensors running out of the field and each talking in its own protocol.
So we need to streamline them into a single protocol which is what the transformer does.
So what the transformer finally ends up doing is splitting every data into two segments.
A flow data and an alarm data because each data from the sensor essentially consist of these two components.
How much water is actually flowing through the sensor and does the sensor have any alarm?
Oh, so 100K sensors, right? I am assuming it would be a lot of events. How many events are we talking about?
So we are talking about 27 million data points per day, which is what we ingest and process.
Okay, so 27 million events, right? So once this many events of data come, I am sure scalability is a big requirement here.
Absolutely. So we've already taken care of the scale with respect to the ingestion part because these Lambdas scale based on the amount of data coming in.
We are really not worried about how many sensors the business adds.
The second part of the scale is managing them. Once the data is reach the cloud, how do we manage it so that we don't slow down the ingestion?
Which is where the pre-processor comes in. So what the preprocessor does is it reads data from these queues,
these are the SQS free flow queues, and makes an important decision as to what amount of data needs to be really processed.
The reason why I say this is a utility like water is not something we use 24/7.
So not all data points are warranted to be executed.
The pre-processor takes this decision in real-time looking into the data and decides what and what not to process. Now how does that happen?
It works in conjunction with the ElastiCache, which is a high-speed in-memory Redis database that we use.
And once it determines that, hey, this is the amount of water... rather the amount of data points that we need to process,
those are the only data points that get forwarded to post-process of processing.
So by making intelligent decisions as to how much we need to process and by allowing any amount of data to come in,
we essentially have built the architecture to scale for any amount of sensors and any amount of data.
This is a real example of scalability. It looks to be the heart of your system.
So, what percentage of events are really worth sharing to the end users?
In our experience, I would say about 40% of the time series data that we get is what goes into the final stage of processing.
And how does the end user get the final view of how efficiently are they using water?
Yeah, so once this 40% of data is siloed out, they are further processed and pushed into a post-processor.
There are a bunch of jobs that run on an EC2 instance, and they end up getting saved in two data stores.
So we've got two purpose-built data stores.
One is the Aurora serverless version of MySQL. The other is DynamoDB.
The Aurora serverless version stores a day's data. How much water was consumed in a particular sensor in a day?
What are the alarms? So these are basically daily statistics that get stored.
DynamoDB on the other hand is something that stores events. So why events and why daily data?
We cater to both sources as well as inlets in a society.
Sources could be a tanker always coming in. Sources could be board wells.
So instead of the amount of water, it's about when did the lorry dump water into the sump, right?
When did the board well start, right?
And in conjunction with the Aurora daily data, both of these can be provided via web API store - our client's web portals as well as the mobile apps.
So essentially we have a web layer that interacts between both of these and pushes the data to the end users consumption points
which is the app and the website.
Okay, so this is architecture via which the end users really understand how efficiently are they using water, is it?
Correct. So we have visualizations and reports that tell them how are they performing, and we also nudge them to perform better
because the whole platform is about rationalizing use of water and bringing in sustainability.
That's a really good cause. So, we talked about 100K of sensors, 27 million events per day, right?
So, how about cost, how are you managing cost with this architecture?
Yeah, so with scale cost is also bound to increase. Which is at least what we thought.
Just to give you a perspective, when we started this architecture, we were at about 15K sensors, and today we are at about 100K.
The revelation was that the cost actually went down by 25%.
So the cloud cost of a sensor is down 25% because our Lambdas have still not even hit 10% of concurrency
and we know we have another 10X to grow.
And because it is able to scale without a single change in the architecture or code,
we have been actually able to realize so much savings.
That's really great, and I hope that with your scalability you further reduce this cost and the architecture looks great.
Can you tell us about the other two product times, will this architecture also scale for the other two product lines of yours like electricity and gas?
Absolutely, so if you look at this, to me, this is a blueprint of an IoT system.
Today it's water. Tomorrow it could be carrying data for electricity.
The other product is basically about gas. So that's another data point.
So I am convinced that the entire architecture can be leveraged for both electricity and gas.
And we hope to roll those product lines out soon in the future.
I hope, too and Vijay, this is a great architecture with respect to what you are doing, sustainability for key resources like water, gas, and electricity.
I hope the other two products come soon.
Thank you so much for sharing your time and explaining this architecture with us.
Thank you for having us here.
Thank you for watching, 'This is My Architecture'.