In this video, you'll see how you can stream analytics using a modern data architecture from AWS.
With this approach, you can move data from a data store into a data lake (outside-in),
from one data store to another (around-the-perimeter), and from a data lake to a data store (inside-out).
Let’s begin by navigating to Amazon Kinesis to look at an outside-in data movement.
In this example, Amazon Kinesis is ingesting real-time data streaming from Amazon Elastic Compute Cloud (Amazon EC2).
The stream is collecting data about taxi trips.
While the dashboard is loading, we can go to Systems Manager Session Manager to see the data that’s being processed and sent to Kinesis.
Here we can see that the code is generating a series of events.
The events are then ingested into the stream and sent to Kinesis in real time.
Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data,
and makes the data available for consumption.
On the Monitoring tab, we can see what metrics are being monitored.
The “put record sum” captures how many data records are contained in the data stream at a given time.
Before we continue with this outside-in data movement,
let's use our data stream to illustrate an around-the-perimeter data movement, which sends data between data stores.
In this case, we’ll move data from our Kinesis Data Stream to Kinesis Data Analytics for SQL Applications.
Let’s select the initials-taxi-trips application to view an analytics event that we have already created.
On the Source tab, we can see the application source.
Kinesis Data Analytics is the easiest way to process data in real time with SQL without having to learn the language.
Let's take a closer look at the configuration.
In the SQL code we have two streams, “cleaned_trips” and “trip_statistics,” each with an associated pump.
A pump serves as a continuous query to ensure the stream runs.
The cleaned-trips pump selects type values that have a longitude and latitude value.
The trip_statistics pump calculates the summary statistics in two-second intervals.
Let's save and run the application.
The application has finished running and we can see output streams listed.
Let's look at output for the trip statistics stream.
Every two seconds, a SQL query runs that gives the total trips,
passenger count, and amount spent for trips.
Let’s return to our outside-in data movement pattern and move data into a data lake.
We’ll begin in Amazon Simple Storage Service (Amazon S3).
Amazon S3 is organized into buckets that receive data from Amazon EC2.
We can run queries of our Amazon EC2 data (the outside) from Amazon S3 (the inside).
The NYC taxi trips bucket has been created for our initials-taxi-trips data stream.
The bucket is organized into folders for the year, month, day, and time when data is collected.
The data objects are stored in Apache Parquet files.
We can query an individual file with S3 Select.
We can choose our output format.
Here we see the first five records of input data from our query to Amazon EC2 from our Amazon S3 data lake.
The third data movement in our architecture is from the inside out.
This pattern is used to send selected material from the data lake to outside systems,
like a data warehouse or database, to be analyzed.
We’ll use AWS Glue, a serverless data integration service, to prepare our data and send it to Amazon Redshift.
We have already created a crawler we can run to catalog the latest data in Amazon S3.
Let’s open the NYC taxi trips database table to confirm that the crawler connected to our data lake.
Here we can see our schema definitions.
The table is ready to use.
Now we'll use Amazon Glue to send the data in real time to Amazon Redshift via our Amazon S3 data bucket.
Let's open the Amazon Redshift console in a new tab.
For demonstration purposes, we've already constructed a cluster for Redshift.
Returning to AWS Glue, we’ll use a connection we already created to integrate with Redshift.
Let's test the connection.
Our connection between Glue and Redshift has been successfully established.
Next, we'll build an extract, transform, and load (ETL) job in AWS Glue Studio that will load Glue data into Redshift.
For demonstration purposes, let’s copy an existing job and clone it to run it in real-time.
Here we see the structure of our new job.
We'll leave the data source as our Amazon S3 data bucket and retain the default properties.
We'll also keep the default settings in the Transform and Data target nodes.
Let's review the job details and then save and run the job.
As the job is running, we can monitor its status.
Our job run has a 100% success rate.
This means we successfully transformed and loaded data from the Amazon S3 data bucket to our Amazon Redshift catalog using Amazon Glue.
Now that our job is running, let’s see how an end user can query the data using the Redshift query editor.
From the query editor, we'll connect to our database and select a Redshift schema.
Let's view the table that appears, which is loading the data schema in the column groups.
When we query the table for a count, we'll get an output from the Amazon Redshift job.
Here's our query result, which confirms that we have established the inside-out data movement architecture.
You've just seen how you can stream analytics using a modern data architecture from AWS.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.