In this video, you'll see how you can use purpose-built AWS data analytics services for your modern data architecture.
With this architecture, you can ingest and store data using Amazon Simple Storage Service (Amazon S3) and Kinesis Firehose,
catalog the data using AWS Glue and transform it using Amazon EMR, and query and analyze the data using Amazon Athena.
To get started with our data lake, we're going to upload a dataset into our previously-created S3 bucket 'reference data' folder.
This dataset is a JSON file composed of song artists and tracks.
The destination path is the reference_data folder.
Now that the upload is complete, let's quickly visit the destination folder to see the dataset object.
The track_lists JSON file appears in the reference_data subfolder.
Let's move on and access Amazon Kinesis,
where we'll create a data delivery stream to our S3 bucket using Kinesis Data Firehose.
We'll use a Direct PUT source and specify Amazon S3 as the destination.
Let's name the delivery stream.
Let's confirm that the data transformation and record format conversion options are disabled.
We'll choose the path for the S3 bucket we previously created.
Let's use the prefix "raw" so it’s clear which folder in our bucket receives the raw data from the delivery stream.
We can also adjust the default buffer options.
A lower buffer size will be faster in delivery.
A lower interval sends data more frequently.
In the Advanced settings, we'll check that Amazon CloudWatch error logging is enabled.
Let's create the delivery stream.
While the delivery stream is being created, we can review its configuration.
Here we see the S3 bucket where the delivery stream will dump the data.
Now the delivery stream has an "Active" status.
Next, we need a user pool that can access the delivery stream to generate and produce data.
Let's create a CloudFormation stack to establish an Amazon Cognito user pool with a single user.
We'll use an Amazon S3 template.
Let's specify a username and password for the stack.
We'll use these credentials later when we access the Kinesis data generator.
We'll retain the default stack options.
Let's create the stack.
After a few minutes, the stack is created.
On the Outputs tab, we can find the URL for our data generator.
We'll log in using the username and password we specified for the stack.
We’ll also select the Region where our S3 bucket is located.
Let’s adjust the number of records per second.
We're using a simple script that will pump in the records once we start sending data.
Let's start sending data.
Once the data generator completes its process, we can visit our S3 bucket to confirm the data is present.
Here we can see all the data that was dumped from our delivery stream into our raw folder.
Now that we've ingested material into the data bucket, we’ll catalog the data using AWS Glue.
AWS Glue is a fully managed ETL (extract, transform, and load) service.
Here, we'll create a crawler to mine the data in the S3 bucket.
Before setting up the crawler, you'll want to ensure you've created an AWS Identity and Access Management (AWS IAM) role
with both S3FullAccess and GlueServiceRole policies enacted.
For the purposes of this demonstration, that has already been set up.
First, we'll name the crawler.
We can retain the source type criteria.
Here, we’ll specify the path for the S3 bucket.
We won’t add another data store.
We'll specify the IAM role we previously created.
We'll have our crawler run on demand.
Let's add a database to store the data the crawler captures.
Now we’ll quickly review our specifications and finish creating the crawler.
Here’s the crawler. Let’s run it.
The crawler has stopped running. As you can see, it added two tables to our database.
Let’s look at this table.
This is the raw data captured by the crawler.
Now that the data has been cataloged in AWS Glue, we can query the data using Amazon Athena.
Let’s explore the query editor.
We’ll select the database we just created.
As you can see, the two tables stored in the database are present for querying.
We’ll use this simple code to query a column from the raw table.
Before we run this query, we’ll need to set up a location for the results.
Let's save this query before we proceed.
Now let’s go to the query editor settings to set up a query result location.
We'll choose our S3 bucket.
By editing the S3 path directly, we can create a new folder to store the query results.
Let's return to the query editor.
Let's run our first query.
Here we can see our query results.
Now that we've shown we can query data directly in Athena,
let’s see how we can transform our data using Amazon Elastic MapReduce (Amazon EMR).
Amazon EMR is a managed cluster platform that simplifies running vast amounts of data on AWS.
We'll be creating a cluster on S3 and using PySpark jobs to perform some transformation and aggregations on our raw data.
For the purposes of this demonstration, we have already created two additional folders in our S3 bucket
to store the PySpark script and EMR logs and uploaded the script accordingly.
Let's start by creating the cluster.
We’ll name the cluster and select the path for the log file folder.
We'll use a step execution launch mode.
Let's configure a Spark application step.
We’ll choose the path location with the PySpark script file.
We'll also supply an argument that passes the script to the S3 bucket.
If the cluster fails, we’ll terminate it.
Let's add the step.
Now we’ll review the configuration.
As you can see, the cluster will have three instances. Let's create it.
We can view the cluster's progress as it executes.
Once all steps have been completed, the cluster will terminate.
We can see that the cluster completed the two applications.
Let's go to the S3 bucket to confirm the dataset is present.
We can see that the processed data was saved in Parquet format.
Now let's visit AWS Glue and run the crawler to catalog the transformed data.
Now that the crawler has finished running, let's view the database tables.
Here we can see the mined EMR processed data.
Now let's return to Athena and query the processed data.
Here’s the emr_processed_data table.
We'll run another simple query that returns a list of artist names ordered by count.
As you can see, Data Lake architecture makes it possible to ingest, catalog,
query, and analyze large volumes of data to uncover business insights.
You've just seen how you can use purpose-built AWS data analytics services for your modern data architecture.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.