(bright music)
- [Shiv] Hello, I'm Shiv Narayanan product manager
for Glue Data Management Capabilities,
and in this video I'm gonna show you how to set up
data quality checks for your data assets
that are cataloged in your Glue Data catalog.
(bright music)
From the Glue's landing page.
Let's click on databases which is
under the data catalog option.
Let's select billing database.
Here we have three tables, raw trip data,
that contains the source data of taxi rides in the city.
The payment is reference data that is
needed for us to clean this raw data,
and the clean trip data is
a cleansed version of the raw trip data.
The Clean Trip Data is an Amazon S3 based table
with 19 columns.
My goal is to set up data quality checks
for this clean table because we want high quality data
to continuously flow from our raw data sets.
I wanna make sure that the columns have valid values.
All the data from my source table is present
in the clean version of my table,
and by payment codes match to the reference tables
that I had shown you before.
(bright music)
To get started, click on the data quality tab.
Now click on create data quality Rules.
The difficult part of creating data quality checks is
the need to manually analyze the data
and create data quality rules.
Glue Data Quality automatically generates these rules
that reduces your real creation process
from hours to just minutes.
To auto-generate these rules, click on recommend rules,
provide an IAM role that has permissions to the data set,
if needed you can provide a filter condition
to filter data from the source, provide the number
of workers that are needed to process this data,
and click on recommend rules.
The recommendation process has started.
Glue analyzes your data, gathers data statistics,
and auto-creates the data quality rules
so that you don't have to manually
analyze the data and author them.
This saves you days of manual work.
Now the data quality recommendations are complete.
Let's explore the recommendations.
You can see that Glue has generated rules that check
for record counts, valid values, completeness
of data, and standard deviation checks.
These are generated based on your data set
that makes these rules highly relevant for your use.
(bright music)
I'm going to select all of these rules
and insert them into my canvas.
You can see that one of these rules say
that the trip distance should be under 306,000 miles,
which is most likely caused
because of bad data in my dataset.
I want to limit my trip distance to be under 500.
Now, one of my requirement is to make sure
that all payment types in the data set is
present in the master payment data set.
Glue provides several out of the box checks
to accomplish complex rules like these.
To do this, I'm going to add a referential integrity check
by navigating the helper,
and by selecting this specific rule type.
(bright music)
On clicking the plus button, Glue adds
the referential integrity rule type,
with an example for you to customize.
Click the schema tab and select update reference tables
to add the reference data sets
that you want to use in your tool set.
Use the schema browser to easily replace example fields
with actual table names and column names.
Now we have a referential integrity rule
with a threshold of one, which means
that 100% of payment types should be present
in the reference table provided here.
Here I have another requirement to ensure that all my data
from log table is loaded to my clean table.
If it was a small table, I have rule types
to compare two data sets, but my table has
22 million records, and I wanna be
efficient about how I compare my data.
For this, I will use aggregate match,
a rule type that compares aggregate measures
between two different tables.
(bright music)
I have created a rule that compares the total amount
between my raw table and my clean table.
To meet my final requirement,
I'm going to use a custom SQL rule type.
The custom SQL rule type allows you to write
SQL-based rules to handle complex business requirements
that cannot be met by out-of-the-box rule types.
(bright music)
In this scenario, I need to set up a rule
to ensure that passengers were present
when taxi charges were incurred.
Let's now go ahead and save the rule set and provide a name.
(bright music)
I'm clicking run to evaluate the rule set.
I have selected a role that has permissions
to this data set, have selected the options
to publish metrics to CloudWatch and EventBridge.
I have also selected a location
to store the data quality results.
We have options to run this scan on a schedule or on demand.
We will choose on demand.
We have provided the number of workers.
You can choose filter data source
and catalog partition predicate to run incremental scans,
so that you're not evaluating the entire data set,
but for this run, we are going
to run it on the entire data set.
As you can see, the Glue Data Quality job has started.
It is now evaluating the entire data set
against the rules that we have set up.
(bright music)
The evaluation is now complete.
We now have a score of 94% with 51
out of 54 data quality rules passed.
Let's now explore what rules failed,
so that we can take corrective actions.
Our results contain a status message
which tells us whether the rule passed or failed
and evaluated metrics that provides us information
and insight into what the actual values are.
For example, the first rule has a low count validation,
but the evaluated metrics provides us
the exact number of rows in the data set.
(bright music)
On exploration, we can clearly see which
of these rules were successful, which of the rules failed,
and what are the boundary values
that actually exist, the data set.
This allows us to actually start taking corrective actions
on these data sets by talking to our source system owners
so that these values can be improved.
I have requirements to be notified
when data quality rules fail.
I have completed a one-time setup in EventBridge
which invokes a Lambda to receive notifications
that is well-formatted, and it gives me detailed information
of which rules passed and which rules failed.
(bright music)
Our documentation explains this in detail.
After working with the source system owners
and fixing the transformation steps,
our data quality score has improved to 100%.
This is reflected in the train chart.
The score combined with the trends instills
confidence in users of this dataset,
for them to make confident business decisions.
These results are written to Amazon S3 and to EventBridge.
You can use this to build customized dashboards
to look at these trends the way you want to see it.
You can invoke all these actions such as recommendations,
data quality evaluation via API,
so that you can run data quality scans
across multiple data sets in an automated fashion.
Refer to our API guide for examples of how to do this.
(bright music)
Sometimes you may have requirements to run
data quality checks on non-data catalog sources,
or apply data quality checks within your ETL pipelines.
You may have requirements to identify the records
that did not pass data quality checks,
so that you can prevent them from entering
into your data repositories.
For such use cases, you can use
Glue Data Quality on Glue ETL.
You can add a transform called evaluate data quality
within your pipeline that evaluates the quality
of data and identifies which records passed or failed,
so that you can route those data sets appropriately.
(bright music)
There is really no limit on
where you can apply Glue Data Quality.
Glue Data Quality can be applied on your catalog,
ETL pipelines, code, and your notebook.
For more information on how to do this,
review our Glue Data Quality documentation
or watch the video linked below.
Thank you so much for watching this demo.
If you have any questions or feedback,
please contact us or provide us your feedback
by clicking on the feedback link.
Thank you.
(bright music)