evan i see you're drinking coffee is
that a good brew
let me tell you what good brew is let me
tell you about aws glue data bro
do you know that companies spend as much
as 80 time to prepare data data
preparation is time consuming it is
expensive and it is a very much manual
process
my name is amin jinich and my name is
alina yunus in this video we will
introduce you to the basics of aws glue
data proof we'll talk about what data
brew is its core components and then
focus on its data lineage feature
at the end we will also present a
hands-on demo so let's get started
aws glue data brew is a serverless
visual data preparation tool that makes
it easy for data analysts and data
scientists to prepare data
with an interactive visual interface
without writing code
with data brew you can easily visualize
clean and normalized terabytes and even
petabytes of data directly from your
data lake data warehouses and databases
with aws good data bro you can get data
from various data sources like amazon s3
you can get it from relational databases
like amazon rds
your local files like csv or from aws
glue data catalog mapped sources
for input data aws blue data blue
supports commonly used file formats such
as csv json apache parquet and excel
sheets
once you import data into aws google
data bro you can explore understand your
data and then transform it as you wish
the interactive data preparation
workspace in data brew is called project
within a project you can group multiple
transformations together save them as
recipe
and then apply the recipes directly to
the newly incoming data
you can choose from over 250 building
transformations to combine pivot and
transpose data without writing code for
complex transformations data brew
provides advanced machine learning
techniques such as nlp natural language
processing
after that you can run the job to
transform the data according to the
recipe script you just built
and then you can output the data to the
location of your wish
that could be machine learning model
it could be s3 bucket it could even be
amazon quick site to make great
dashboards for your customers for output
data aws blue data bro supports csv
values
json apache parquet apache auro and
apache york
now on to data lineage data blue tracks
your data in a visual interface to
determine its origin called data lineage
this view shows you how the data flows
through different entities from where it
originally came
you can see its origin
other entities it was influenced by what
happened to it over time and then where
it was stored
this allows for data analysts to capture
the relationship between the different
data sources how data gets transformed
converged its upstream dependencies and
its downstream usage moreover it allows
to trace incorrect sources of error
all right
now that we've gone over the basic
concepts of glue data brew and our
lineage we'll now move over to a demo to
see how this works in practice
for the purpose of this demo we have a
fictional retail company called octa
tank
octa tank sells pet products online they
want to find out which product is not in
demand in which area and target them to
increase awareness and product sales
the data analysts theme is fairly new
they want tooling that doesn't require
them to code and manage infra
additionally they want to be able to
track data flow so that it is easy to
trace errors along the pipeline
let's take a look at how octa tank can
use glue data brew to identify zip codes
where specific products are not very
well
we have three kinds of datasets the
sales dataset in amazon s3
this contains information about the
product that was sold the quantity of
the product as well as the customer who
bought it
the other two data sets are customer and
product data sets
the customer data set is residing in
amazon rds relational database service
we have it extracted as a glue data
catalog table which acts as a metadata
table containing references to the data
in rds
finally we have the product data that
exists as a csv file on our local
machine
we'll be joining and transforming all of
these in glue data brew
and then finally outputting them to an
s3 bucket
now let's move on to the demo
in the services search for our type data
blue click on aws glue data blue
and this will take you to its console on
your left side you have all the
components we discussed earlier
click on create project and give it a
meaningful
name a recipe as you can see is
automatically created
now let's move on and create a new data
set
give it a name
this will be the data set that will
correspond to the sales data set and
within it we have the cs.csv file
keep the rest as default and scroll down
to permissions
select the appropriate role and click on
create project
now you can visualize all the different
columns of the data set
let's take a look at the data lineage
tab on the top right here we can see
that cs.csv was loaded as a data set
from s3 it is part of a project which
has a recipe attached to it we will
populate this in a bit
switching over to the cloudtrail view we
can also see an audit trail of the
different actions that were performed on
a resource the time at which they took
place and the user that performed them
in order to develop a recipe let's head
back to the project view the next step
is to join customer data with the sales
data set
click on join on the top menu and you
will be now able to create a data set
from the glue data catalog for customer
data
give it a name
select data catalog rds tables
and then click on octa tank customer db
and select customer table
click on create dataset
click on next
next again
from this screen select customer id
and also customer id from table b and
create an inner join while only
selecting zip from the table b
click on finish
now we have to create another join with
product data we will achieve that by
repeating similar steps as before
starting off by creating a new data set
and giving it a name
selecting csv upload is an option
then storing it in an s3 destination
click on create dataset
here you can preview the data
for the columns to join on this time we
will select product id and product key
so we select those here
and this time from the product table we
only select product type
now octa tank has all the data they need
to calculate total sales by zip and
product type at the penultimate step
we'll remove all the unnecessary columns
like txn or transaction number customer
id and quantity
to finish off the recipe we need to
identify total sales by product id sip
and product type
for that we create a grouping
so now we choose product id zip product
type and total sales that's the column
names and choose aggregate functions
for total sales the aggregate function
would be sum
click on finish
now that we have all the steps in our
recipe we will go ahead and publish this
version
after publishing the version we will run
a job on it which will apply the whole
recipe to in the entire data set and
persisting the transform data so in
order to do that let's create a job
we will give it a name
you need to provide an output location
for the job and for that we can select
an s3 destination here
select a role from the role name drop
down
and then finally create and run job
after the job is run successfully we can
take a look at the data flow and see how
the lineage has changed with more data
sets and a destination being added
note that this is the lineage for this
project and similarly you can look at
lineage for each one of the data sets as
well
now where we understood what data brew
is and what it can do let's see where it
fits customer asked me hermine we have
so many services in aws we already have
aws glue studio we have sagemaker data
wrangler now you come to me with aws
glue data blue what is the right tool
for the job
so let's look at this small diagram to
help you here
if you're focusing more more on running
etl jobs
ongoing moving data between various data
sources you should use aws glue or
google studio
this is something your data engineers
are going to use
if you're preparing data for machine
learning and you would like to explore
data
then use
sagemaker data wrangler this is
something your data scientists are going
to use
but for your data analysts who are there
to explore understand and prepare data
for reporting use
aws glue data room
of course there is an overlapping
scenarios where you can use different
tools to accomplish the same tasks for
example data transformation you can use
blue studio and data brew for data
exploration you can use both data brew
or data wrangler
i hope this diagram helps you to select
the right tool for the job cool let's
summarize now we discussed glue data
brew and looked at its data lineage
feature in detail in the demo we saw how
octa tank's data science team can use
this to have a no code tool to perform
analysis while keeping track of the
lineage
so that's it from our side now it's time
for you to try it out on your own trust
me it's going to stick like glue bye