Welcome to 'Back to Basics.'
I'm Orit Alul and today we're going to talk about data.
The amount of data generated by IoT, smart devices,
cloud applications, and social is growing exponentially.
You need ways to easily and cost-effectively analyze all of this data
within minimal time-to-insight, regardless of the data source.
Therefore, I'd like to talk to you about the basics of data lake and lake house patterns.
You're probably familiar with the data lake pattern, but what is the lake house approach?
The lake house approach is about connecting your data lake, your data warehouse,
and all your other purpose-built analytic services.
Let's start with the source of the data, the data ingest.
Your data is valuable, and you'd like to put it in a persistent safe place,
such as Amazon S3, as quickly as possible and as simply as possible.
What you don't want to do, is you don't want your application to gather the data in files
and copy these files to S3.
This will require your application to buffer the data in memory or on a local disk,
which can result in failure and data loss.
Therefore, you should use a stream, such as Amazon Kinesis Data Firehose.
With Firehose, it is simple to reliably load your data into the data lake.
Your application can write the data it collects straight to the stream,
and Firehose will deliver the data to S3 as quickly as you choose.
When you use a data streaming mechanism,
your application can write even small amount of data as it arrives, directly to the stream.
This will keep your application lean and simple.
You can configure Kinesis Data Firehose
to persist the data according to a time interval between one to 15 minutes,
or as data is accumulated between one to 128 megabytes.
This is your raw data.
Now that you've ingested your data to S3, you'll probably like to start querying the data.
But before doing that,
there are three things that will make your reads or queries, more efficient;
the data format, partitioning, and the size of the files.
Now let's look at each one of these topics.
Let's start with the first, data format.
In most of the cases, your application will write data in a textual format,
such as JSON.
This is a nice, human-readable format,
but it is wasteful in terms of data size and processing.
Since most of your reads are going to be analytical queries,
it can be beneficial to transform the data to a columnar format,
such as parquet, or ORC.
The columnar format enables better compression of the data,
and the ability to access only specific columns,
which reduces the amount of data scanned during query time.
Let's talk about the second topic, data partitioning.
Data partitioning is about the ability to search for the answer in the right place.
When you run a query, you'd like to scan as little data as possible
in order to get results.
Data partitions will navigate the query engine to the right place or places
in which the data resides.
For example, if I'd like to run a query on the last three days of data,
then if I partition the data in days,
I can scan only the data of the last three days.
But if I'll partition the data in months,
then I have to scan data of the entire month and filter only the relevant three days.
What if you have really fast velocity data and partition data in hours or minutes?
Would that be a good approach? Probably not.
Which leads me to the third point, data compaction.
If you write data too frequent, or partition the data to small files,
you might end up with large number of small files.
Large number of small files will also lead to inefficiency during query time
due to overheads.
The general rule when using S3 as a storage layer
is to generate files in the size of 128 megabytes or larger.
As always, we have to consider trade-offs,
between partitioning the data and the file size within the partition.
You can use AWS Glue jobs to create scheduled jobs
that will transform the raw data to a columnar format,
compact the data to large files, and write the transform data to the relevant partition.
You can leverage Glue Studio to build this job in a visual way.
One more thing that we need to do, is we need to update the data lakes catalog
with a schema of the data and the partitions we created.
This is where AWS Glue Crawler comes in handy.
We can use the Crawler to populate the Glue Data Catalog automatically.
Now we are ready to start querying the data.
You can use Amazon Athena to run standard SQL ad hoc queries
straight on the data lake on S3.
Athena is serverless, so there is no infrastructure to set up or manage,
and you pay only for the queries you run, based on the amount of data that was scanned.
Amazon Athena is really popular with data engineers and developers
due to the ease of use and cost efficiency.
If you prefer to analyze your data in a more visual way, or produce reports,
you can leverage Amazon QuickSight.
Amazon QuickSight is integrated with many data sources, including Amazon Athena,
so you can analyze the data in your data lake,
and also join other data sources,
such as relational databases, Amazon Elasticsearch, Apache Spark, and more.
If you are a BI engineer, you'd probably like to run some complex and sophisticated queries.
You'd probably like to use a data warehouse, such as Amazon Redshift,
to run your queries on large amount of data and get quick results.
Amazon Redshift is a fully managed, petabyte scale data warehouse service.
This leads me to talk about the lake house approach.
The lake house approach combines the things that we discussed so far.
The data lake allows you to have a single place
where you can run analytics across most of your data.
While the purpose build analytics services
provide you the speed you need for specific use cases
like real time dashboards and log analytics.
The lake house architecture enables you to use the right tool for the right job.
You get the flexibility to evolve your lake house to meet current and future needs
as you add new data sources, discover new use cases,
and develop newer analytics methods.
Check out the resources below for more details, and see you next time.