hello and welcome back to another video series in this
series we will be looking at dbt which stands for data build tool and dbt is an open source
project for handling the transformation piece of an etl or more specifically an elt type of process
it's really interesting it really blends software engineering and the automation aspects of that
with something that most data engineers and bi professionals are familiar with which is sql and
i want to give a huge shout out to the people at fishtown analytics who have created this
um it's it's really something that the more you get into it the more you realize
how how much thought really has gone into the development of this so so the first video here
will be strictly on just installing dbt and just getting the foundation set up on your machine so
we will be using the cli so i'll go over here to dbt cli installation i am using a windows machine
and there is a note here that says you should have git as well as python
before you install this so as it says here we will open our powershell and just simply do a pip
install dbt it's as simple as that so i will open it up so open our command prompt pip install dbt
now that went really fast for me this is because i've done this in the past
before i already have it if you don't it might take a couple minutes to go through and
install all the dependencies that are part of this project but even still once that
completes that's it it's installed and we can check it by running a dbt version
and there it is it says this is the installed version we're up to date we have these various
plugins and we're ready to go so the next step here is to create our first project
you know we have the dbt installed but we don't have any files or anything to really work with
so what's the next step there's two ways to go about this we could either one create our github
repo first that we're going to be building with or two start to create the files and then push
them to a repository for this tutorial we will start first with creating a github
repo and then moving from there so let's go to github create a new repository i'll call this demo
dbt just keep no description keep it private just a basic repo because this is where we
will push everything once it's initiated so now we have our blank repository here
and now i'm going to clone this into just my documents folder so documents git clone
okay yes i understand it's empty
and here we go
so now we have a blank repository let's go into that
let's change directory into that
and right now there's there's nothing in here it's just an empty um empty folder
but what we'll do now is we will use this to initialize a dbt project and the command for
that is dbt init and the name of the project it could be whatever you want
in this case we'll keep it the name we'll keep the name the same as our folder so we'll just say dbt
or oops we'll just say demo dbt
enter and that's it it's really as simple as that so now this has created a skeleton
project for us with a couple examples in here it also created
a user profile if you go to your users and the name that you're on i'm on my admin on my machine
and we have this dot dbt file folder i should say with profiles let's go into visual studio
and open up and see what this looks like
so if we look in here
we have our analysis these are a couple empty folders that can be used for various purposes
maybe you have some one-off scripts macros macros are really important aspects of
of dbt you can think of them almost like user-defined functions that can be reused in
different places so we will touch on that later models this is where the core foundation of dbt
lives in terms of creating data models these are just examples that are put here we will get
into the specifics of each of these files later snapshots is another empty one as well as tests
so really it's just skeleton there's not much in here the dbt project is an important one this
is where we can give it a name of our project let's call this keep everything the same here
demo dbt you can call whatever you want but what you put here needs to match down here that's
again well we'll get into the specifics but just understand if you change this this needs to match
now the profile here this is what will match
this profile right here so let's open this up as well with visual studio code and the default here
i'm going to do side by side the default here is matching this default here the placeholder
that was put in is redshift and it's split it up by dev and production which is really nice because
you can split up the credentials based on you know your environment needs and you can easily split
back and forth between what it's looking at by changing the target so in this case it's
saying use these credentials if you change this to broad it would look at these credentials i
actually have to set this up to match the profile for a snowflake so let's go here and maybe you
have something else but i will go to profiles go to the profile set up here find snowflake
and here's an example this is what it should look like what it should look like for me
so i'm going to copy this
and paste it
my snowflake db i'll just shorten this to snowflake
which has to match right here that used to be default now it's changed
target is dead this is the only option that we have right now i could call this
you know i could call this anything but we'll just keep it dev type is snowflake the account id is
this part of the url everything from the account number to the location
again this is specific to my snowflake instance
username
this is my username and i'll put in the password that i have for my trial account
after i publish this i will be removing this the role
the default role i'm going to put it as i'm actually going to set this to account admin now
this is something in a production environment you likely wouldn't do you'd want to create a
specific role that is just for dbt to use that has all the permissions that it would need to
create and view any table and any table and view but for now we'll just use account admin
just for the sake of simplicity here
the database name we'll use this demodb that comes with the snowflake account
warehouse name compute warehouse again these are defaults and all these things you could customize
and we can customize as we get further along into something that's a little more realistic schema
we can do public threads one or more we'll put this at 10.
you can play around with that but 10 just to allow it if you have a lot of tests or a lot of
models it can run them in parallel together which is which can really help performance
okay so now this is set we have our base project created we have our profile set it is referencing
the correct profile and this profile is set up to match our credentials on snowflake
okay so let's open up a terminal and one of the commands that we can run here is dbt
dbt debug and that will attempt to connect with the credentials we have and will
really let us know if we did this correctly or not okay what do we got here error not found
ah that's because i'm running it in
i'm running it in the wrong place here so let's
let's open up here do the same thing
okay there we go again that was because the terminal was trying to run at the wrong
granularity of this folder so once you run it at this level you should be fine okay so connection
is okay it can find that profiles file the dbt project file all of our connection information
so we're good to go so let's attempt to do a run now run our first example dbt run
okay so it says where target is dev we know it's running here and it'll give you a log output of
everything that's happening so it created this model created this model successfully
we didn't dig too much into what these are basically they're sample models that are just
selecting one and just creating one row now if we go into our actual database here
here they are dbt has created these models in here
and they should just have one row with one id that's it so it was successful we were able to
successfully initialize a project create a profile and run our first set of models
now you can imagine once you start to really grow this out now you can literally build your whole
warehouse using this tool it's really powerful once you start to get into it so now the next
step here the final step is let's add this to our github repository right now it's just empty
let's see i think we need to add a remote branch
so let's do
git remote add origin here
oh i may already have this okay all right let's
okay so i must have created this already but if you don't you can add that origin in there
we will get add everything git commit add a message my first commit
and now let's push
okay this should have pushed right right to master and let's refresh this
and here we go
maybe you don't want this extra file this extra folder structure here i actually
want to change that i don't i'm actually not crazy about how that looks so i'm going to do
something a little funky here let's let's do it through here it'll be a little easier to see
yeah we got too many levels so i'm going to i'm actually going to put these in here
and then get rid of
get rid of this folder
now let's go back to visual studio code
it should recognize yeah we're changing all this up here a little bit
get add all these changes commit message
remove extra folder push now that whole last part maybe not necessary if you're not super
comfortable with git but for me i don't i don't want to have that extra folder there
and now yeah refresh this and when we come right to this repo it's right on here so we
don't have that extra layer of a folder so now this github is ready to go and if you have a
team that you're working with you can all start to make branches off of this and push-up changes
thank you for watching we'll continue on this tutorial in future videos and if you found this
useful drop a comment if you have any questions or feel free to subscribe as well thanks again