hi friends welcome back to another youtube video in today's video I'm going to
share my journey and my reasoning for leaving data engineering as a career for some of you who are
familiar with my career journey you already know that i worked as a daily engineer for about two
years before i decided to become a self-taught data scientist before we jump into the video as
always make sure to give this video a thumbs up if you like what you hear and subscribe to my content
subscribe to my content subscribe to my channel if you want to hear more about data science tech
lifestyle career journey and everything else in between and i'm also starting a new series on my
channel where i'm going to invite other industry professionals who are in the data science domain
and in tech domain i am super excited to share those content i'm super excited to talk to all
those people and share more insights with all of you i think you'll love that so with that let's
not waste any more time let's get started okay so let's step back back in terms of my
journey as a data engineer a brief history i started my first job as a data engineer if i
don't include my internship experience and my pro bono experience my first full-time paid job
was working as a data engineer and i did that for about two years i was working at one of the
fan companies what are the manga companies oh my god i've been meaning to say that
okay the new name is mama i don't know like what what is happening
yeah for those of you don't know facebook changed their name to meta so now the fang
whole fang thing is all messed up there is a twitter meme going on some people are calling it
manga and the original guy who this who basically founded the term fang has called it mama which is
i don't even know if that's how you pronounce it i need to hear somebody
else say it but apparently that's the new name anyways i got a little bit sidetracked
so i worked as a date engineer for two years at one of the mama companies that's so so weird
mama companies okay sure i guess i'll get the hang of it eventually
um and um i did that for two years okay it was it was great learning experience
there's a lot of work that goes behind the scenes of like how the data pipelines are built how you
actually get to see those like pretty dashboards and stuff like there's a lot that happens in the
back end and i was basically part of that back end um of like building those pipelines making
sure the data is flowing in and we have all the correct set of data that we need to understand
where whoever is like doing the analysis whether those are like financial analysts data scientists
product managers whoever is accessing the data so i did that for about two years
and then after two years i decided to make the transition out of the data engineering domain
and um became a data scientist so there were three reasons why i left data engineering domain and i'm
going to talk about them one by one number one for me was obviously the type of work i was like
i supported one of the mama companies i just can't get it one of the mama companies who has like
presents globally and i basically supported all their data pipeline basically how customers are
coming to the site and how they're engaging with the site um and where they are coming from where
how they're entering the site what do they do after they enter the site so a lot of like traffic
data um i was ingesting and building pipelines around that whole whole basically domain after
doing that for two years i realized like a lot of the work was somewhat repetitive for example i
would build one pipeline it would go to production people will start using it then i get assigned on
another project and then i'm doing pretty much the same thing i guess like i was i was like in the
entry level role so that's why maybe it did not i did not get to do like a lot of like thinking work
but um most of it was like very repetitive work i would say on top of it all the data pipelines that
i created supported the team's weekly business review which goes out on monday 12 pm that means
whoever is going to prepare the data monday one morning has to has the data by sunday night so
i for that two years i actually was working on sundays trying to make sure that the the pipelines
are running and everything else is is is in in the database that it needs to be and if it's not
there then i'm like opening ticket talking to the data warehouse teams and trying to figure out like
what how can we get that data in time by monday morning or what is the reasoning that the data is
delayed because if data is missing i need to give reasoning that was my reason i got actually kind
of bored i felt pretty comfortable in the space i got kind of bored so i was like i need to try
something different and that was the time when i actually did a project um which was more like
data science focused project and that's where i got the flavor of like what the interesting
word could look like which means like i don't do the same thing over and over again so that like
brings me to my second reason visibility so one of the things that i struggled with like working
in the data engineering space was it was it's a back end role like you work in the back end
you prepare data and then you're putting in a database and somebody else is either accessing
it directly from the database and presenting it or if you are working on a data poll you are actually
pulling the data and then you're giving it to somebody else like a product manager or somebody
else so for me i realized like i want to be the front-facing person because i was realizing that
i was doing a lot of work like i was pulling a lot of data i was like doing a lot of like work in the
back end but i wasn't getting the visibility in the front end by the time somebody takes that
data or writes a paper on it and presents it to leadership my name was lost in that process
so i wanted to be in the front i didn't want to be in the back so that was like my other reasoning
because i wanted that visibility i wanted to be part of those decisions that were being taken
with the data so that was my second reasoning of the lack of visibility which led me to
um make that decision the third and the last one was it's it's a bit of open-ended and
some not all of you are gonna agree with me on that that data engineering overlaps with
software engineering so let me say that again the the data engineering work overlaps with data with
software engineering in some companies they don't even have data engineers they only have software
engineers and software engineers are doing the data engineering work that data engineering
the role itself exists in very big companies but it does not exist in all companies
even if you just take example of like the big tech companies and you just search role by each company
data engineer role by company you won't find it in every company which means you either have to do
that maybe they have some other role that is that is considered a data engineer but they're calling
it something else so you have to like do your own kind of research and figure out what that is
or you're stuck where you are pretty much the create opportunities i did not find that
in data engineering like there is like entry level data engineering one data engineering
two senior data engineer or principal and then there were not like many principal data
engineers like actually i don't even know if there were there were any data principal data
engineer in that company where i was so i didn't see a lot of people in the leadership position
who were data engineer previously and now are in the leadership position and for me like i
want to be in the leadership position i uh so for me like that was kind of like
also like a reminder to self like what is my what is my trajectory in this role if i were to like
invest 10 more years in it all those things combined basically led me to transition
out of the data engineering role the money was good i'm not going to complain the money was good
pretty much same pretty much same as i made as a data scientist in that company but the type of
work was obviously is very different between the data engineering and the data scientists work i've
been doing data science work for about like seven eight years now and i love it like i'm not bored
i know this is like the right field for me my work is super interesting even though like
probably from my last videos you saw you saw like how frustrated i was with like the job search
specifically yeah there are frustrations but i love this space i don't think i'm going anywhere
i love working on data science projects i love working on experiments i love working on documents
i love writing my analysis i love storytelling and i love being the person who is in the front
presenting those insights and being part of the whole decision i think that just that's just like
more of my work style and what i enjoy most this video is not supposed to bash a specific career
choice i think data engineering is great for somebody who enjoys that type of work for me
personally it did not make sense because i had different career aspirations and i wanted to do
different type of work that's all I wanted to say and if you have data engineering experience let me
know in comments how has your experience been what are your thoughts on the data engineering field
in general like i would love to know your thoughts but that thank you so much for
watching i hope you're having a beautiful day and i will see you in the next video bye