Welcome to Solving with
AWS Solutions, your deep dive
into reference architectures
built by AWS that are ready to
deploy instantly in the cloud.
I'm Tom and I'm joined by
Shivansh here to tell us about
Utility Meter Data Analytics
on AWS Quickstart.
Thanks, Tom.
This Quick Start solution
creates a platform that uses
machine learning to help
utility and energy companies
analyze data from
smart utility meters.
Sounds like we have a
lot to explore.
Let's dive into it.
OK, so first questions first.
What is a Quick Start?
So quick starts are production
ready reference architectures
built by AWS Solution
Architects
and partners together.
OK, so includes partners.
Great.
So talk to me about analytics
and data analytics.
Yeah.
So this is the Utility Meter
Data Analytics Quick Start.
It helps energy companies to
deploy this data analytics
platform on AWS.
This is the landing page for
the Quick Start, and there are
a couple of details
going on here.
This deployment guide,
which talks about the
architecture of best practices
and security,
about
this particular architecture.
There's a link to deploy this
through the AWS Cloud
Formation console, and there
are costs
and licenses information.
There are also details about
what resources are being
deployed as part of
this Quick Start.
And then if you want to go and
take a deeper look, you can go
and look into
the architecture.
OK, great.
Let's do that.
But just before we do so,
you said energy companies.
And this is a data platform.
Is it this for a specific set
of data that we're looking at?
Yeah.
So this is for smart meter,
which is at your in my home and
basically get that data
and analytics out.
Got it.
Okay.
So this energy consumption by
all the consumers like yourself
and myself?
OK, great.
Let's look at
the architecture.
Sure.
So there are three main
components in
this architecture.
There's a ETL pipeline,
there is a machine learning
pipeline and then
there are APIs.
OK, great.
So where should we start ETL?
Yeah, let's start with
the ETL.
So in each pipeline, the data
which you get from the smart
meters goes into the
S3 raw bucket.
OK.
You have AWS Glue crawlers,
which crawl the data from the
S3 bucket, and it cleans the
data and put it into the
clean data bucket.
OK.
And then from there, this data
is partitioned into different
partitions across date
and time, as well as other data
sets and put it in the
partition data bucket.
OK, so that's getting the data
cleaned up and ready to be used
through the
machine learning process.
That's right.
So let's look at the machine
learning pipeline then.
Yeah.
So in machine
learning pipeline, there are
two different pipelines.
One is the model training and
the other one
is batch processing.
The batch processing pipeline
uses AWS step functions to
process the data in a batch,
and then the model training
pipeline is actually for the
real time when you are putting
the data in directly through
the different steps of
the machine learning.
Got it!
OK.
And so and the batch processing
you say is scheduled, is that
something that the customer can
define or part of the
Quick Start?
So it is part of the Quick
Start and you when you have to
apply this Quick Start, it is
getting scheduled, but you can
also configure it later
if you need.
Got it!
OK.
So we have we have
our process data.
We're running it through our
machine learning pipeline.
What actual insights are we
gaining from this?
So there are three different
insights you can gain out
of the box.
There is an
energy usage information,
there is an energy
forecast information, and then
there is an outage details
you can get.
OK, great.
So we're taking all of that raw
data we're processing and then
we're able to get those
insights out of that data.
What's the API for?
So the API is to get the data
which you have already
partitioned and trained from
the S3 bucket, and you can
build visualizations and
dashboard on top of that
using the APIs.
OK, so I can query all of those
results and actually
pull those back.
Is that something we can see in
action?
Yeah, let's do that.
So here's the endpoint for
the API, and there are two
things passing it in
this particular endpoint.
There's a start date and time,
and then there's an end
date and time.
And this specific endpoint
provides the outage information
of all the meters between that
start and end date.
So if I click send here.
This is going to return to all
the meta information between
that start and end date,
which have been out for
this particular duration.
Got it.
OK, so this doesn't come with
any kind of dashboard in
the UI, but you could build
those on top of this API.
That's really the idea.
That's right.
OK, great.
So super useful, very focused
on energy companies, but it's
end to end workflow for
processing that that isn't
getting real insights
into energy consumption.
Thanks for sharing.
My pleasure.
There you have it.
The utility meter data
analytics on AWS Quick Start.
Check out this Quick Start and
many of the solutions on the
AWS Solutions website.
Thanks and see you soon.