smart companies know that they need good data engineers and there's a good reason
for that data management is going to separate those companies that excel in the next decade
versus those that start to fall behind a little bit and that's why it really shouldn't be a
surprise that data engineering in particular has become one of the most in-demand jobs
so in this video i want to discuss how we even got to this point and then start to think about
where things are headed for us in the future now just as a heads up this one is going to
be more of me just talking as opposed to the usual tutorial style that i do but i'm going
to try to cover like 20 years of progress in about five minutes so let's get started
if we were to travel back in time to the early 2000s a time of innocence with aim and myspace
in the corporate world we'd see a very clear divide between the it department and then pretty
much everyone else most database related things were on premises and probably managed exclusively
by a few database administrators or dbas for short spreadsheets were used a lot because it really
wasn't straightforward to get custom data from the databases themselves the rise of the internet
at this time brought about a lot of new business applications but that also meant new data and data
that was scattered and wildly inconsistent the solution then at that point was to focus mostly
on centralization and data modeling into what we all refer to as data warehouses by the end of the
decade companies started to focus a lot more on digitizing everything i.t and business functions
started to blend together more than in the past and people started slowly realizing just how much
value this data stuff could really have in making decisions so those companies that kind of leaned
in heavily into using data around this time were able to gain a really big competitive advantage
this increased interest was great but a traditional it department couldn't really
keep up with this new demand or custom data all by themselves so by this time a lot of companies
began to focus on more structured etl processes and data pipelines that could help automate some
of the work the mid 2010s saw a pretty major shift towards even more data-driven apps and technology
in general i mean just think about the explosion of social media apps around this time we saw the
rise of modern cloud databases like snowflake and amazon redshift full-on data teams were becoming
a thing at companies where it may not have been really considered before there was also
a ton of really important innovation happening at the biggest tech companies so we're talking
amazon facebook google who unsurprisingly were some of the first ones to see the real impact
that big data could have and was probably already having at that time so this is where we started
to see software engineers being called upon to help build some of these more complex components
to make it all work but at the same time all this innovation also had other side effects one
in particular being the huge increase in the amount of new tools flooding the marketplace
we also started to see more data silos beginning to pop up within companies and security becoming
obviously a bigger concern but of course there's always some sort of response and this time the
focus began to shift towards what we now refer to as modern data engineering by the mid 2010s it was
pretty much standard practice across the board to have an internal analytics team that managed
a data warehouse the data pipelines and provided custom reporting many companies still followed etl
processes but this elt format started to become a lot more popular because of the wider adoption
of cloud databases with all these new tools and integrations that were required the data engineers
were relied on really heavily to create and manage the back end of this constantly growing tech stack
you also started to see more people outside of just the core data team begin to develop more
advanced data literacy and then because of that business users now wanted more control and so you
started to hear this term self-service analytics become a really big buzzword and something that
people really wanted all right this brings us to today the early 2020s we've officially gone from
myspace to tick-tock and from spreadsheets to data streaming today more companies continue to migrate
to the cloud which makes that elt approach much more common architectures consist of
multiple tools many of which are open source now and the influence of software engineering
principles can be felt pretty much across the stack data operations has officially evolved from
more of an ad hoc type of workflow 20 years ago to a more continuous development kind of thing
with a lot of automation in a very similar fashion as traditional software development we even have
an entirely new focus area known as uncreatively data ops so overall now that hardware and storage
is less of an impediment as it was maybe in the past the focus has now shifted to development
best practices and automation okay so i've covered a lot of changes here especially the stuff that
happened in the last five to ten years but the technology itself isn't the only thing that's
evolved the job titles have changed as well and obviously every company is different so there's
no really one-size-fits-all definition but based on my experience here's the current lay of the
land of the main roles close to data engineers so first and foremost we have the data engineer
itself these people are going to be mainly focused on the end-to-end integration for extracting and
modeling data using a variety of tools they'd also be expected to have an understanding of
general programming cloud platforms infrastructure management scheduling amongst other things they
might also be involved in report development but sometimes it may stop a little bit before
that next is the data analyst now the analyst will typically sit between the engineers and
then business end users they'd be expected to create dashboards and have a good understanding
of the content of the data itself but they're not going to be as technically focused as let's say
an engineer next is the analytics engineer and this is more of a recent role that's popped up
and it kind of acts as a hybrid of an engineer and then an analyst they may not be as involved in the
integrations or back-end work like an engineer might be but they would be able to use sql-based
transformation tools you know things like dbt to create custom data models they'd also be expected
to understand software development best practices like version control ci cd and because they're
also leaning towards more analysts they would be able to maintain the documentation train business
users etc kind of like a hybrid and last but not least is the data scientist and they're not going
to be as involved in the back end integrations and the pipelines modeling all that stuff they're more
involved in creating predictive models based on the data already provided by the engineers
there'll be more statistics and math driven rather than let's say pipelines integration
storage et cetera when i hear things like machine learning i typically think data science
so where do we go from here obviously i can't see into the future but my guess is we'll continue to
see evolutions within the operations part of the work i think writing custom scripts will always be
part of the job and personally it's my favorite part but i also think we'll continue to see
more things being abstracted away as new tools are able to handle a lot more under the hood for us
and we don't even have to worry about it as data engineers our value will be more on understanding
how to properly configure and manage all of these different systems that are interacting with each
other but regardless of what happens i still believe that understanding the fundamentals of any
technology whether it's data or web development or anything else is what's going to make you better
at using any new technology that comes out and using it to its full potential for that reason
i would always still encourage people to master sql database design and basic programming concepts
so these are just my thoughts but what do you think where do you think things are headed for
us over the next decade thanks as always for watching and i'll see you at the next video
you