if you work with data you may come across a file format called parquet,
which is a very popular file format for data science and data engineering project.
Hey everyone I'm Riz Ang and I make videos about data engineering and
Microsoft Azure if you like videos about this consider subscribe like and make sure
you comment down below if you have any questions. Parquet which sounds like a french word actually
means a wooden floor in English dictionary unlike Excel or CSV file parquet is designed with big
data analytics in mind, meaning the file is designed to handle large amount of data and also
being able to be read or queried frequently and efficiently, now to show you what that looks like
let's go to some slides, let's start with some basic we'll know a CSV file or excel
which is actually a row-based data and column-based data slightly different from
that and it's parquet is one of them if you see this one example of a table with three columns
title date and a chart for a CSV file the way it's stored in the system on disk, it will look
like this there are basically one next to another the title name the date and the chart after that
it's followed by another title date and a chart it goes like this so on and so forth. Parquet which
is a column based data it looked a little bit different it will actually keep all the titles and
then followed by all the dates and all the charts, and this is how a parquet file or column base file
looks like. Under the hood, parquet file looks a bit like this, so it has a file metadata which
contains the start location of other metadata for the file which is the role group metadata,
the column metadata and page metadata and in the file we have one or more row groups which is
basically a logical grouping of columns and the columns which I gave an example in the previous
slide and within a row group there is one or more column chunks essentially and in each column
there are pages one or more pages and the page contains a header and a metadata
of the page which is essentially statistics like a sums and count and min and max
around the data we've got R and D values which stands for petition level values and definition
level values, they are purposed for nested data structure or nest list and just briefly repetition
values counts how many repeated fields within that nested list, whereas the definition values, counts
how many optional fields within that nested list in order, let me know if you have any questions
in the comment below so I can explain further. And also lastly the data itself
encoded, which I'll explain with some examples. To give an example of what data in encoded format
looks like or encoded data, I have some examples here to show you so encoding by the way is
essentially finding repetitive data and replace it with something smaller or more compact, now I have
two examples here, dictionary and delta encoding there are other kinds of encoding obviously as
well used by parquet but I'm just going to give you some illustration on what it is trying to do
for dictionary encoding is quite straightforward so it looks for repetitive values and replace
those values as repetitive values with integer or an index and as you can see here zero is for
Atlantic one is swan song so and essentially we replace those strings with zero one which save a
lot of space. For delta encoding is a little bit different it will actually look for the position
of the character for this some of these these strings they are quite similar to one another
so Led Zeppelin in here which is about I think 11 characters is we know that it's from the position
zero and the next the next one is actually gonna remember that or from the after 11 which is 12
this is the next values and then the next one is actually using the the value from the 12 which
is after these two ones here this is the the next one which is just one here character 15
but the next one is interesting because this one is picking up the value from the back to the the
12th one here which remembers that after the the underscore there's a V, but again this is just one
example of what the encoding for the incremental or delta loading looks like and just based on
these examples alone you can see the benefit of replacing the repetitive values into something
smaller which is less storage which is yeah money. The other great thing about parquet
is compression, now compression is a little bit different from encoding because compression
looks at the file as a whole remove any unused or redundant parts encoding is a little bit
different from that because encoding is looking at the data itself and finding
repetitive values and make it smaller now just to give you some comparisons here CSV versus
parquet in different compression methods as you can see here just how dramatically reduced
both the query time and also the size in gigabytes now just to give you an idea the different
compression methods that can be used in parquet, there are three popular ones which is ls lzo
gzip and snappy the default is generally snappy as a compression method and I normally go with the
snappy because it's just best for general usage and it's actually slightly better performance than
lzo, just to point out though gzip is actually a good idea if you want to store the files or
data as a long term or cold storage, again another benefit is less storage more money yeah in there.
To give you another example about the query time I have a comparison here on the left which is a
selected columns query essentially selecting some columns with some filtering and on the
right i have full table query which is selecting all columns and with some filtering as well,
if you notice CSV takes a lot longer to query for both types of query here and what that means is,
if you use parquet your query time is dramatically reduced you save time that way and there you go
that's one of the great benefits of using parquet. Parquet is also great when you want to use
all these big names Azure AWS Google Cloud Spark Hadoop and also Databricks,
so it has great integration with many other solution providers and frameworks.
To conclude parquet is a binary file format great for big data analytics, it's able to
store data very efficiently, you can query that data that parquet file format very quickly and
also it's able to integrate with many other solution providers and frameworks quite well.
That's it for today's video I hope you like it if you do subscribe like and also ask any
questions if you do have any questions I'm more than happy to help that's it for today I'm out.