In this video, you'll see how you can build a big data analytics pipeline using modern data architecture.
With this data architecture, you can populate the data lake using data buckets stored in Amazon Simple Storage Service (Amazon S3),
query data across multiple data stores from Amazon Athena, and build business intelligence (BI)visualizations to develop insights using Amazon QuickSight.
Let's begin by going to AWS Glue to populate our data lake.
We'll add a crawler to extract the data from one of our S3 buckets and load it into the data lake,
which will also be stored in Amazon S3.
Let's give our crawler a name.
We'll keep the default options to crawl all folders in our data store.
Here we'll choose the path for the S3 bucket that we want to populate.
Let's select an appropriate AWS Identity and Access Management (AWS IAM) role for the crawler.
As you can see, there's many schedule options for crawlers.
We'll have ours run on demand.
We'll create a new database to store the crawler's output.
Let's review our crawler specifications and complete the setup.
Once it's ready, let's run our crawler.
As you can see, it successfully crawled our S3 data bucket and added 15 tables to our database.
Let's view those tables.
Here we have the option to edit the tables and their schema.
Let's adjust the column names for our person table, which defaulted to numerals.
Let's move on and access the AWS Glue Studio.
Here we'll set up extract, transform, and load (ETL) jobs
to convert the database tables into Apache Parquet file format so that we can query them later on using Amazon Athena in our data pipeline.
Let's retain the default job options.
In the data source node, we'll select the database we just created for our crawler data.
We'll need a different ETL job for each table in our database. Let’s start with this table.
In the transform node, we can adjust the data types as necessary.
In the data target node, we’ll select the Parquet format and specify the S3 bucket to store the output.
We'll use the same bucket that contains the source data.
On the Job details tab, let's name the job and choose its IAM role.
We'll disable job bookmarks so that the job processes our entire dataset every time.
Let's save and run the job.
From the Monitoring tab, we can confirm that the job run was successful.
The job run succeeded.
Next, let’s visit our S3 bucket.
As you can see, we have a new folder with our Parquet files.
Now let’s return to AWS Glue Studio.
We now need to create jobs for the other tables in our database to convert them into Parquet format.
For the purposes of this video, we’ll do that behind the scenes.
Let’s move on to the next step,
which is to build a crawler to extract and save our Parquet files in the data lake.
We'll create and name the crawler.
We'll choose many of the same specifications as for the previously created crawler that mines our S3 data bucket.
However, this time we'll choose the path for the Parquet files only.
Let's also add a prefix so it is easy to recognize these are Parquet files when reading the database tables.
Let's complete and run the crawler.
Once the crawler has completed its run,
we'll see the new folders with a Parquet prefix in our database.
Now that we've populated our data lake,
the next step in building a data analytics pipeline is to query our data sources.
Amazon Athena is an interactive query service that makes it easy to analyze data from multiple sources using standard SQL.
We'll execute queries via data source connectors that run in AWS Lambda.
We'll create a data source connection to Amazon DynamoDB.
Let's name our data source.
We'll create a new Lambda function for this data source connection.
Let's follow the link to complete our function in AWS Lambda.
We’ll provide a name for the spill bucket where the function will store output data.
Let's also choose a name for the Athena catalog.
This will automatically become the name for this Lambda function.
We'll retain all other default settings and deploy the function.
From the AWS Lambda Deployments tab, we can view the function’s status.
Now that the Lambda function is successfully deployed,
let’s return to the Athena data source connection page and choose it from the list.
Now let’s connect the data source.
Our data source connection was created successfully.
Let's go to the query editor to test the connection.
We'll query our DynamoDB database and a Parquet table in our data lake.
The query produced accurate results,
confirming the data source connector is linked to both databases.
Our data pipeline is now fully set up.
Next, we'll visualize information in the data lake using the interactive business BI dashboards available in Amazon QuickSight.
Let's create a new analysis.
We'll use Athena, which hosts our data source connections, as the source for our dataset.
Next, we’ll select the table with the data we want to visualize.
We'll keep the default setting to import data to the Super-fast, Parallel, In-memory, Calculation Engine (SPICE).
Once the dataset is ready, we can create visualizations from a range of visual types.
Let's visualize the ticket price sum as currency.
Let's add another visual, this time using a vertical bar graph.
It's easy to choose what data to represent on the graph.
We can easily add another dimension, such as seat level data.
We can also change the visual type.
QuickSight Insights interprets the data added to visuals and automatically generates a summary of findings in plain language.
You've just seen how you can build a big data analytics pipeline using modern data architecture.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.