- Please join us in this video (light music)
as we store data on AWS Timestream
with just one line of SQL code.
And now over to our technical expert, Joe.
- The next piece of this puzzle is getting an idea
of how the data is gonna flow from our sensors,
our weather station in our various fields,
all the way into a dashboard
that we could use to monitor the situation.
The easiest way to understand this
is with a big picture or architectural diagram.
So here we have the architectural diagram.
On the left-hand side,
you see we have a couple of sensors
and they're represented by
a sensor plus application icon on the left
and then an ExpressLink.
So those who maybe you've previously had a weather station
that you had designed for your product
and now you're gonna make it an IOT weather station,
you'll add the ExpressLink.
We've already gone over the video
with the hardware changes for that
and that'll make the new module.
The data will flow over an MQTT,
a secured MQTT connection, to AWS IoT Core.
IoT Core is a broker that's the onboarding place
for all the data through various topics
into the greater AWS Cloud.
The IoT Core can do a number of things,
but the thing that it's gonna do first for us
is it'll hold the topic data.
When the data arrives in that topic,
it'll then be redistributed
out to all the subscribers of that topic.
Inside of IoT Core,
we can have multiple rules that will subscribe to topics.
So we will have a rule that listens
to the specific weather station topic,
and the rule's job is to collect the data
and then disseminate it through various actions
to wherever we may decide
our application needs to have data.
In this case, we want the data to be stored
at a time series database.
So the AWS service, Timestream, is such a database
and we have a rule with an action
that will accept the data coming on the topic
and copy that data into the Timestream database.
Because it's now in a Timestream database,
it's very efficient to sort that data
and bring it out in the order that it arrived
so it's a Timestream.
Grafana is a dashboard tool,
a graphical dashboard that can do
a number of different kinds of widgets for dials.
It can do graphs, it can do heat maps.
It can do a number of things that we would use to build
information dashboards so you can get at-a-glance updates
on what the situation for your sensors are.
It's capable of reading its data out of Timestream.
So now that we have all the data through the rule,
from the topic into the rule into Timestream,
Grafana will reach into Timestream
and extract the parameters
according to queries that you write.
It'll extract those parameters
and then present them to you
with the graphical widget that you desire.
We'll put in more detail
on exactly how that's done in just a few minutes.
The AWS Console, the cloud side connection
and the configuration of our account will be covered now.
This is the main console
when you log into your AWS account.
The first thing we wanna do is make sure we have a thing
that represents our weather station.
So we're gonna go into the IoT Core service.
This is the IoT Core service
and we will make sure that we've got our device.
A device would actually be called a thing.
That's a specific device,
and here we see that we have two things registered.
These long cryptic names
or number strings are the thing names
that come as an ExpressLink.
Every ExpressLink comes from the factory
with some kind of a thing name already attached.
You can see that there's two different kinds
of thing names here.
The shorter one on top was the thing name
registered with the uBlock SARA-R5-based ExpressLink.
They had a different kind of a thing name identifier
and expressive with the C3-MINI-based ExpressLink
had this identifier.
They're all unique, and so it doesn't matter,
but you just need to know which one is which
until you give it some specific identification.
So we can go in here
and we can verify that everything's set up.
It has a certificate.
There's the ARN for this device.
There's the certificate that comes out of the ExpressLink.
It was registered here,
so everything is fine.
We don't have to do anything here.
Normally I would also check
to verify that data was arriving,
but I don't have this device online at the moment.
So on a previous day,
I did verify that the data was arriving.
It's just not running right now
because we're still debugging systems.
The next piece we have to do is set a rule.
The message routing, then go to rules,
and we see that we have two rules.
Let's look at my rule for the IngestJoeData.
The rule is simple.
There's an SQL-like statement
that identifies the parameters.
In this case, it's select the wind speed
of average of two miles an hour,
average of two minutes.
Wind direction, average of two minutes.
The temperature in Fahrenheit,
the daily rainfall in inches, pressure.
All of these parameters are present in the data
that's coming out of the ExpressLink as a JSON document.
So SQL, this SQL statement is able
to parse that JSON document
and extract these parameters.
And it says it's gonna extract
these parameters from this topic,
the weather sensor with this big name,
my unique thing name.
These topics can be anything that we define.
I happen to define this particular one.
Dan chose a different topic.
And the reason we have two different SQL statements
in two different devices,
since we didn't collaborate we just made two rules
and we sent all the data to the same action.
So let's look at that.
We can have multiple actions.
So when the SQL statement matches,
the actions will run.
This action runs a Timestream table right.
Write a message into a Timestream table,
so that's the action
and that action, we have to specify a database name,
we're calling that Weather Data,
and the table name that will receive that data.
We can also add additional dimensions
which are kind of like metadata on the topic.
The dimension in this case is I specified one with location
and one with the value of Joe
so that we can identify later in Grafana
that this was my weather station.
So I'll close that.
In these actions you can actually have multiple actions
and you can create actions that happen only on errors.
So I could add an action
that might be to pass the data to a lambda
where I could then run, I don't know, Python code
that would process that data in the cloud
rather than processing that data on the edge.
So this is the action.
This rule catches the data
and passes it to Timestream by this action.
Let's go over to Timestream now.
So there's the Timestream service.
Let's open up the Timestream service
and we see in the Timestream service,
we have the weather data database.
It's monitoring, we can see
all the different ingestion points,
when data's been arriving
and how many times the rules run.
So we can see the health of our system.
We can look at the table.
We see that we have two tables,
but the one that we were writing to is just data.
So I can look at just this one
and I can see that there's
the critical information about that table.
But this is kind of boring.
Let's look at a query.
So this would be select star from weatherData.data.
So this basically says give me all the information
that's currently in the table data
of the database weather data, let's run that.
So here's a bunch of data.
Now this data is not in any particular order
because I didn't tell the query
to put it in any kind of an order.
Here's another query, select star from weatherData.danData,
order by time, and limit it to ascending order
to test the Timestream
to make sure that our data is arriving
and we can run a query on the data.
We will run a sample query.
In the database weather data,
we'll run the query select star everything
from weatherData.data, order by time
so it'll be in time order,
ascending order, limit of just 10 entries.
If we run that query,
we'll see that here's a bunch of information on the data
and now it's in time order,
and we can see that the data is stored as measure values,
and some of them are doubles
and some of them are big integers.
So you'll need to remember that when we get into Grafana
because the temperature, the floating point temperature
is obviously a double format, 85.196 degrees,
and the timestamp information is just a large number.
There's other pieces of information
like the daily rain in inches is,
it's a big integer
but it's not a floating point value.
So our system gives us whole numbers in inches.
Of course, I haven't had any rain here lately in Tucson.
All right, the data is now in Timestream.
It clearly is recording stuff, so we're live.
The next step is gonna be move into Grafana
and turn this raw data into nice pictures.
(light music)
- Thanks, Joe, for walking us
through the data flow architecture,
showing how to add Timestream database on IoT console
and writing the script.
It is amazing to see the ease of using ExpressLink.