In this video, you'll see how you can accelerate machine learning (ML) pipelines using a modern data architecture.
With this approach, you can easily integrate ML services into data analytics pipelines,
leverage visual tooling to lower the entry barrier for creating and orchestrating pipelines,
and quickly generate and visualize forecasts against custom datasets.
Let's start at the AWS Lake Formation console, where we have data lake registered in Amazon S3.
The data lake contains information about crude oil production in the US.
The S3 bucket contains three objects.
We'll fill the processed and published folders with data as we move through the workflow.
The raw folder contains a public dataset file from the U.S. Energy Information Administration
representing 10 years' worth of monthly crude oil and natural gas production variables.
We also have two databases that serve as repositories for our metadata.
As you can see, we've granted permissions to the various AWS services and Identity and Access Management (AWS IAM) roles
that will access the data pipeline.
Next, let's review a project we have set up in AWS Glue DataBrew.
This project transforms the connected dataset to coordinate with the datatypes used in Amazon Forecast.
We have already defined and run the recipe.
Let’s view the job output.
The transformed dataset is saved as a CSV file in the S3 data lake’s processed folder.
Next, let's go to the AWS Step Functions console to view our workflow.
We can use Workflow Studio to see our data pipeline in a visual format.
The first step runs an AWS Glue DataBrew job to clean the raw data.
Next, the workflow splits into two parallel streams.
The stream we see on the right invokes a data crawler in AWS Glue to run against the processed dataset.
A feedback loop responds according to the crawler's performance.
If the crawler fails, the workflow ends.
If the crawler runs, other steps monitor its progress in 30-second intervals.
The parallel stream imports data into Amazon Forecast.
A feedback loop monitors the status of the import.
With a successful import, the workflow assigns steps to train the Forecast predictor and monitor its status.
Once the predictor is fully trained, we have a step to create a new forecast dataset.
Again, we’ll check the status of that new Forecast job continually.
After the forecasts have been generated successfully,
they are exported to the data lake on S3 and stored in the published data folder.
Finally, another Glue crawler is assigned to collect the published metadata,
populating our data catalog with the forecasted material.
Let's save the workflow and start execution.
Let's name the run and remove this sample input.
While the workflow is running, we can watch its progress.
As you can see, the first step is underway.
Let's check the AWS Glue console to see whether the DataBrew job is running there.
The job is running.
Let's continue monitoring the workflow's execution.
The Glue DataBrew job succeeded,
and the workflow has invoked the parallel functions to crawl the processed data and train the Forecast predictor.
In the AWS Glue console, we can see that the crawler for the processed data has started.
And from our Amazon Forecast dashboard,
we see that the dataset is currently importing target time series data.
Let's continue monitoring our visual workflow.
Now the Lambda function is training the Forecast predictor.
Let's return to the Forecast dashboard to take a look.
Here, you can see our new predictor being created and trained.
This can take several minutes, so for the purposes of this demonstration we've sped up the process.
Let's refresh the page to see if any more steps have been completed.
The right side of our visual workflow has successfully completed.
In the Glue console, we can see that the crawler's status is now "Ready."
The workflow has updated again.
The Forecast predictor has completed its training.
Let's make sure that this progress is reflected in the Forecast console.
The new predictor is now active.
This predictor was configured to use an Amazon Forecast machine learning algorithm called CNN-QR (Convolutional Neural Network - Quantile Regression).
CNN-QR works best with large datasets containing hundreds of time series.
To produce accuracy metrics for predictors, Forecast uses backtesting,
which is the process of testing a predictor against historical data and comparing forecasted values to actual values.
Let’s review the forecasts currently being created against the predictor.
Another forecast is in the process of being created.
Our workflow is continuing its progress.
When the forecasts complete, they’ll be exported to our S3 data lake.
According to our workflow, the data is now exporting to S3.
Let’s quickly confirm that the forecasts finished generating.
The forecasts have successfully generated and are active.
Let's refresh the workflow.
The final step, now in progress,
runs a Glue crawler against the published metadata in the data lake.
Let's check in S3 to confirm the forecasts were successfully exported.
We'll select our bucket and look in the published folder.
The folder has been populated.
We can also review the crawler that coordinated the export.
This page indicates that the crawler exported the forecasted data to the data lake's published data folder.
Back in Workflow Studio, we see that the final steps are being processed.
Let's refresh the workflow.
The entire data pipeline has successfully completed.
Now that our data lake is populated with the forecasted data,
we can produce visualizations and perform data analytics.
First, we'll prepare the datasets for visualization using Amazon Athena.
In the Athena query editor, we can run SQL queries against the tables in our AWS Glue Data Catalog as if it was a traditional database.
Let's preview the data in our processed table, which is a historical aggregate.
We can see the timestamp, state ID, and demand variables that were collected and formatted from the raw data.
No let’s preview our published table.
Here, we see the results of the forecast predictor.
Forecasts were generated at the P10, P50, and P90 intervals.
Next, we'll go to Amazon QuickSight to visualize this data.
We've already produced two visualizations, showing the historical data and the forecasted data, side by side.
We can filter this data.
For instance, let's view data for only the state of Texas.
Finally, let’s review the composition of our dataset.
As you can see, our visualizations dataset joins the two datasets we previewed in Amazon Athena.
When viewing the joined dataset, we can see that the historical and forecasted data sets are still distinct, as they are in our visualizations.
You've just seen how you can accelerate ML pipelines using a modern data architecture.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.