Being a data scientist is cool but sometimes it's overrated. Today let's talk about the most
underrated job in data science and it's in way higher demand than a data scientist. I'm Samanvay,
a data scientist living in Sydney, Australia. Data science is huge and growing really rapidly
especially in Australia there are many roles that you can do. In data science you could be a data
scientist, a machine learning engineer, a data engineer or a data analyst. Today let's talk about
the most underrated job in data science and I'm honestly surprised as to why it doesn't get enough
air time. Feel free to pause the video and write down your guesses in the comments
section below. So without further ado let's talk about the role itself. So if you guessed
data engineer then give yourselves a pat on the back because that's the correct answer.
According to a data science interview report done by interviewquery.com in 2020, there was a 40%
increase in data engineering interviews as opposed to only 10% increase in data science
interviews. In another tech job report done by DICE, data engineer was listed as the fastest
growing profession with a 50% year-on-year growth as opposed to a 32% percent year-on-year growth
for a data scientist. Now let's talk about what data engineering is and why it's in such a high
demand right now to the point that it's now more popular than the sexiest job of the 21st century,
being a data scientist. I actually started my career doing a lot of data engineering so
I'll be speaking greatly from my data engineering experience and also relating to the trend that I'm
seeing in the industry and throughout my career so far. Firstly what does a data engineer do?
In summary a data engineer looks at how data is collected, transformed and loaded. Data engineers
will readily build scalable data systems that execute on these three domains and let's talk
about why these skills are in such high demand right now. So the trend that we're seeing in the
industry is that more and more organizations are reaching higher levels of data sophistication.
With this comes the need to build reliable data systems that won't falter when it's being pushed
into the limit and more organizations are dealing with a greater amount of data and this is only set
to increase going into the future. This explains why there's a huge demand for data engineers that
are well versed in building robust data systems. Now let's talk a little bit about the specific
tasks that the data engineer does so it gives you a little bit more detail as to what a data
engineer's day-to-day looks like. As I mentioned data engineers are concerned with data collection,
data transformation and data loading. So what do I mean by data collection? An example of this is
integrating with third-party data providers in order to ingest their data into our databases.
Let's suppose that you're a data engineer that's working at a company
that provides personalized news feeds based on the user preferences the first step is to get the news
this involves connecting with existing news providers and downloading their news so that you
can build a recommendation system in more detail the data engineer will be responsible for building
ingestion pipelines that hook into the apis of these news websites and ingesting their data
into databases and the data engineer will be responsible for setting up the entire system
so hopefully you can see how data collection is such a crucial role and how important it is
with data of all kinds in all different types of industries so after you've connected to these news
websites you've downloaded the data so next step is data transformation so you might realize that
the different data sources from the different news websites are not standardized and they're all in
different data formats and this difference in data format can cause a lot of issues down the line as
data analysts and data scientists are looking to use that data for their model building or
reporting purposes so your role as a data engineer is to create a uniform data model for all of the
different data formats so that's where the data transformation step comes in so this will involve
you doing a lot of data cleaning and building transformation pipelines for this purpose then the
final step is data loading once you've transformed the data then you need to make the data ready for
use by data scientists and data analysts so for example in the case of data analysts they might
want to analyze the data which means that you might have to build another pipeline that's able
to take that transform data and shift it into a analytics tool this might be loca or power
bi or even tableau so you need to make sure that the data is easily ingestible by users down the
line another example of this data loading process is loading your data into another data system or
loading it into another software platform that the customer will be directly interacting with so this
entire process i've just described is known as etl extract transform and load the main responsibility
of a data engineer is to take care of that etl process and this might involve them working
on all of the pieces of the pipeline as long as everything that's needed to facilitate it such as
curating a database to making data integrations to coming up with effective data models so what kind
of tools does a data engineer use on a day-to-day basis so the main programming languages that a
data engineer uses would be python and sql so all of the etl pipelines are usually written in
python sometimes data engineers also use scala to create effective data pipelines in my experience
i've also done quite a bit of work in golang to make apis for data systems but that can be on a
case-to-case basis in terms of applications you'll be using git for source control using tools like
kubernetes docker and cloud technologies such as aws and google cloud platform for deploying
applications you might also use etl pipelining tools like airflow or event streaming tools
like apache kafka or and finally you might use snowflake or luca for your data analytics and
building bespoke data products this is obviously not an exhaustive list but hopefully you get a
good idea as to what are the tools that data engine use on a day-to-day basis so who should
become a data engineer so i would definitely recommend you to become a data engineer if you
truly love programming and software engineering so to be a great data engineer you definitely
have to have a great attention to detail because majority of the time you'll be in the weeds of
building data systems and also looking at a lot of data in its more granular and detailed form so in
that case you have to be very good at picking up anomalies and errors in the data like i mentioned
before you also need a good grasp of software engineering fundamentals and on the flip side
you won't be interacting too much with customers or stakeholders definitely not as much as data
scientists so data engineers will definitely be spending more time coding as opposed to data
scientists so you've decided you want to be a data engineer so what other steps you can take to head
down that direction so if you're just graduating from high school then i'd recommend you to do a
computer science or a software engineering degree to be a data engineer and it's also pretty useful
if you choose to do a data science major because that'll expose you to the data science cycle and
really understand how data scientists think about data and how machine learning algorithms work
there are also some data science degrees that are very heavy on the engineering side of things but
in my opinion right now i think it's best to stick with pretty vanilla software engineering
or computer science degrees because that'll give you the fundamentals that you need to grow
as a data engineer now let's talk about the spicy details which is what is the salary of
data engineers so currently in australia the average salary is around $110,000 to
$120,000 AUD and that number varies obviously based on your experience and the industry that
you're in. Going into the future i can definitely see the demand for data engineers only increasing.
So that's it for this video if you enjoyed it then smash that like button and i'll see you next time.