two of the most important Concepts to understand as a data engineer are data warehouse and data
Lake and maybe more importantly what's the difference between them so in today's video
that's what we're going to talk about we'll talk about the difference between a data
warehouse and a data Lake when you're going to use the two so that ultimately you can walk
away with a better understanding of them and be able to engage in discussions around them
so before we dive into each of those individually I think it's important to take a step back and
actually put in perspective what are we even doing here what's the purpose of a data architecture to
begin with and to me there are three big reasons that we even do this to begin with and I'm sure
there's more but let's just stick with these three first of all a lot of companies pretty
much every company is generating data from various different sources all over the place
you know they have different applications different third-party tools that they use
all sorts of stuff and they need to understand it so the first goal is to just consolidate all
that information we have information all over the place get it in one place and second is
trying to figure out what happened with that data what are all the events that happened you
know what can we analyze from that and the third reason which probably summarizes the whole thing
is to have all this data so that we can make better decisions our whole purpose here is
to take all of this data that's generated so that ultimately the business can make better
decisions otherwise why are we doing any of this right so with that in mind let's talk talk about
where does a data Lake and where does a data warehouse fit into this and why do we use them
while data warehouse is probably a more well-known term I actually want to start first with data Lake
because if we think about what the whole purpose is and how we're taking all of these Source system
information and moving it over to make decisions a data lake is going to be where all of the source
system data gets put it's a central Landing zone for all of your data it's unstructured
it's just dumping data constantly into this place it's not in a consistent structure it's going to
be data in all different formats so Json video files text files CSV it's everything just dumped
into one location and this could be maybe in an S3 Amazon bucket it could be Azure storage you
could use modern Cloud databases like Snowflake and just load it all into a database server you
can pick wherever you want to put this and this has become more of a requirement in a
modern data architecture because we're constantly generating more and more data in a modern world
right we don't want to wait an hour or a day like in a typical ETL fashion the idea instead
is to just constantly be loading into this data Lake and then from there you know maybe
once an hour you pull from that data Lake and do some Transformations or whatever you want to do
so now let's talk about the data warehouse in my mind the data warehouse comes after a data
Lake if you want to think about it linearly the data is being loaded in its raw form into
a data Lake and now like we said we want to be able to make decisions and report off of it and
make sense of this data all that data in the data lake is completely unstructured it's in
all different forms it's not relatable in any sense so we can't really report off it that's
where the data warehouse comes in the goal of a data warehouse is to take all that data and
turn it into a single source of clean formatted organized data some sort of structure of it's
it's probably a separate database with separate schemas and tables all this stuff that is separate
and structured in that way for you to use and there's a lot of strategies that you'll find for
using this you know Common ones are dimensional modeling Kimble star schema normalize denormalize
all this stuff and you can get on and on about which approach is best but ultimately the goal
here is to create a single Source organized form of your raw data and you're typically going to
update take this on a Cadence so so maybe once an hour once a day whatever you're pulling from this
data Lake updating your data warehouse cleaned up models by running Transformations adding
business logic all this stuff so that you have something clean and easy to work with not only
is it more organized easier to interpret kind of a consistent look but performance might be
better as well because you're going to have keys and relationships and things that make
more sense this is also where you'll see the concepts of facts and dimensions possibly or
the idea of slowly changing Dimensions again these are things outside the scope of this
video but at a high level that is the purpose of a data warehouse you are taking this unstructured
all over the place data and creating a single source of organized Truth for companies to make
decisions and use their data however they want so hopefully now you have a better understanding of
the difference between these two important topics of a data lake and a data warehouse and these are
two really critical components of a modern data stack and one thing that might be helpful for
you to kind of visualize lies this more is to see the whole modern stack maybe in one place
and if that's something you're interested in I do have a free guide that you can download and
just use for yourself it's called the starter guide for the modern data stack Auto link below
so again I hope you found this helpful thanks as always for watching and I'll see you next week