what's going on everybody and welcome to
a video about cloud adverse local GPUs
and when should you use which one or
which one should you use I think it's
pretty clear the one that you should use
is most likely cloud or local or both so
so on which circumstances should you use
which one and so on so I think you know
most of the hosts are gonna vary in
price at least a little bit but in
general a good average number to look at
is gonna be about two dollars an hour
per four thousand dollars of GPU so
you're gonna pay two dollars an hour for
$4,000 of GPU that you rent so it's kind
of regardless of which GPU it is that's
gonna be in general the the ratio so
point what would that be point zero zero
zero five I think that's correct
anyways dollars per hour so whether you
rent a smaller GPU or even more power
than a four thousand dollar GPU that's
close to the same ratio right is going
to vary significantly as we've seen we
can see hosts literally cost twice as
much for the same amount of VRAM so
anyways keep that in mind but that's the
number I'm going to be working off of so
let's say it's two dollars an hour for a
four thousand dollar GPU well it's
pretty simple math how many hours is it
going to take to equal four thousand
dollars well it's just four thousand
divided by two so two thousand hours so
that's gonna be about 83 days which is
like two point eight months so once we
also factor into so so basically we take
two point eight three two point eight
months to equal four thousand dollars so
let's just say no matter what the GPU is
that you're going to rent let's assume
you were going to consider buying it you
would have to spend two thousand hours
on that GPU constantly training with
that GPU to equal the cost of that GPU
if you were to buy it you know locally
as opposed to hosting it now of course
there's other considerations like if you
buy the hardware you could later sell
the hardware so there would be you
usually I don't I try to budget about a
50% depreciation per year just because
one it's old but also like one it's used
but then too usually by the time you're
ready to sell it there's newer bed or
ones out so it also gets depreciated for
that reason so generally a 50% hit per
year tends to be the case on high end
GPU some of the mid-range ones hold
their price better but anyway so there's
that also but the other thing you have
to consider if you run locally is it's
not just the cost of the GPU right you
you could in theory take that GPU and
put it in your main machine and so
that's what I did when I first started
out and I'm actually gonna recommend
against doing that if you're actually
starting out because it's not cost
efficient the only reason I did that is
there were no cloud GPU hosts when I
first was getting into machine learning
these GPUs in the cloud are actually
kind of a new thing that hasn't really
been around for very long so I think if
you're just starting out you can't
possibly fathom actually training models
for 2,000 hours so just rent in the
cloud that's where you should go now I
will throw in one more kind of caveat
and that is in slash warning to watch
out for these like value-added hosts so
I see them recommended or people ask
about them all the time an example this
might be Floyd Hub now nothing against
Floyd Hub I think they have a decent
enough service it's just that I think
some people mistakenly go to them when
they shouldn't so if you're trying to
learn how to do machine learning and
actually program it all yourself and run
it yourself go into production yourself
a place like Floyd Hub kind of becomes a
trap again they're not trying to become
a trap it's your fault not their fault
but you'll get stuck there because you
get used to that specific platform and
leaving becomes a significant hardship
because you've got to learn how to
actually do hosting rather than using
they're pretty gooeys and they're
pre-built models and all this stuff so
you you want to be careful about those
services also the problem and the reason
why it becomes a trap is Floyd hub or
other sites like it or going
to just always be a little bit more
expensive their value-added so like
comparing the cost of Floyd Hub to other
hosts even some of the more expensive
ones you're still gonna pay like a 33%
premium over what you would pay if you
hosted it on a more traditional host
that does not scale well so I would
recommend against making that mistake
there's also like free tiers that you
can get honestly when you're just
learning and you don't even want to rent
a GPU or you don't want to buy a GP or
you don't have one powerful enough just
go off of your CPU your CPU is powerful
enough to learn deep learning you you
really don't need a GPU you start
needing a GPU when datasets are like
millions large and obviously you've got
how many features per sample or
something like that but but still you
get the idea you don't really need a GPU
to learn you need a GPU to actually
produce things so anyway and you also
don't need a GPU for like production
like once you actually go into
production with a specific model chances
are you're doing it on a CPU just
because you don't need the GPU we use a
GPU so we can feed in batches of these
samples unless you're running a big
business or you're processing things for
Facebook who needs to classify it just a
ton of stuff every second you're I just
run on a CPU I mean every model I've
ever put into production ran on the CPU
even though we trained on the GPU so
anyway getting back to our let's say our
$4,000 GPU for $2 an hour you know the
other costs that you have when you're
local is just electricity so let's say
you are running a four thousand dollar
GPU 24 hours a day it's going to draw
like your entire machine maybe 50 to $70
per month just empower for the machine
the machine also is going to generate
heat which has to be accounted for with
air conditioning usually unless it's
like in the winter but here in the south
I tend to budget about a hundred dollars
per machine per month that I run if I'm
running it 24/7 training models so again
that adds up so again it kind of
basically becomes about three months so
if you are training the model non-stop
for three months straight or
accumulative of three months and let's
say
three months over the course of a year
so 25% of the time you are training some
sort of model on that GPU then it starts
to make sense to buy this GPU and have
it local but there's other reasons you
might want to buy a GPU and have it
local right first of all computer
hardware is just cool like I understand
that so sometimes you know you might be
trying to justify buying it locally
because it's gonna save you money but
that's that's not true just be you know
realistic and honest to yourself just
admit you want to buy cool stuff and
that's okay like if you just want to buy
it to own it that's also cool but you're
not saving any money like it the cloud
just makes more sense because you have
options rights opportunity cost if
you're in the cloud at any point you can
change to a different cloud provider you
can also come down you could at any
point decide I'll just go local now
right you have that opportunity to do
that but if you go local and you spend
$5,000 in your and then you train for
only a hundred hours over the course of
a year because I'm gonna be honest like
it's actually pretty difficult to hit
that 2,000 hours of training mark in a
year I've done it but I've also gone on
long periods of time where I don't do it
and it really depends like the only
reason I'm able to do it is because I
have clients and then I also do like
tutorials and so I'm always like working
on something but I think if you're just
like a hobbyist and you're just trying
to like learn you're like a student or a
hobbyist chances are a local GPU is a
huge fiscal mistake right if you're
doing it for business purposes it makes
a little more sense like if you're
earning a profit in some way you're
getting grant money or you're being paid
by some corporation or whatever you're
you've got clients or whatever it makes
a little bit more sense plus if you're
running it through a business it's
tax-deductible either through like
depreciation but also just the purchase
price so it starts to make more sense to
have things locally again but anyway
they're there's so many things to think
about but honestly for me personally I
end up with a bit of a hybrid approach
so I've got a Titan r-tx
in my main machine prior to that I had
the 1080 Ti which I strongly recommend
that was like the best bang for your
buck card to date before that was the
970 was it a TIA I think was
970 TI anyways that was like an amazing
one we haven't had any amazing bang for
your buck since like the new r-tx cards
are pretty awesome for their price but
the 1080 I would have kept my 1080 TI I
just happen to get a Titan r-tx
so I use that now but I think for the
typical average person you you would
really want to you want to have
something that you can like develop
locally on so like a 1080 TI or maybe
one of the new RT X 1080 s or the 220
ATS or whatever something you can at
least develop models locally on but then
when you go to actually start training
for long durations you'd probably just
do in the cloud the cloud is way more
convenient also if you're local you kind
of like for me I train models locally
but then sometimes I want to Train like
many models at the exact same time well
to do that to train many models at the
exact same time I just expand into the
cloud so maybe I've got one locally but
then I expand out into the cloud I'm
doing everything like a bunch of other
models in the cloud so I just kind of I
kind of do it that way but I'd say 95%
of my GPU time nowadays happens in the
cloud it's just more convenient there's
more power it's as much power as you
could possibly afford and desire so so
anyway I hope that was clear
like I said I recognize most people are
not going to be buying maybe a four
thousand even dollar GPU and then maybe
some people want even more than that so
for example at some point I'm gonna
review this Lenovo data science
workstation which is a thirty thousand
dollar computer that you can plausibly
buy well even this three-month thing
extrapolates to high-end machines even
like that where if you want 96 gigabytes
of vram it's going to be like $13,000 a
month from Asher or AWS and then it was
like about $10,000 a month from Google
or it could be as cheap as $4,000 a
month from Linode so yeah I'm quite the
Linode shill lately if you want to check
out Linode by the way for cloud GPUs
that's just low note comm slash syntax
it's a $20 credit honestly I think the
other benefit to cloud
P use is like what we're seeing with
windowed right now they're coming in
with the same like let's say four per
dollar vram and tensor cores they're
coming in with fit faster tensor T flops
and the same amount let's say V RAM for
half the price less than half the price
it's less than Ave and it's not because
Linode is amazing it's because all of
these cloud hosts providers are going to
be just clawing at each other and just
fighting it out for the the actual users
because right now this cloud hosting
GPUs thing is this kind of new market is
this a brand new market with a lot of
money that's coming in and if you ask me
I think you know I think GPUs just like
GPU rentals is going to be a far larger
market because the average person who
rents a GPU right to do data analysis
and like all companies are starting to
need to do this these people are
spending way more money than they were
ever gonna spend on CPU and RAM because
it's a completely different challenge I
mean we're talking like 10x the hosting
cost so I think GPU hosting especially
as we move forward is so valuable that
these companies are going to just be
competing the heck out of each other and
in the meantime you and me get to enjoy
really really really cheap cloud hosting
prizes like I'm pretty sure I I don't
know I can't imagine how even how I
think Linode has to be losing money on
these are this offering and it's not
Linode is not gonna have the last word I
don't think someone else is gonna come
in with even better prices so anyways
stay tuned for that but as long as these
companies are fighting each other for
for actual you know users the cloud is
it just becomes even more enticing to
use over local but there are reasons to
have local it is it's definitely easier
it can be can be sometimes more
convenient to have local so you can just
develop right at your main machine and
then you push it to the cloud and let it
run while you forget about it for a few
days so anyway hopefully that was
helpful for some of you guys
want to get too deep into the numbers
but basically no matter which way you
cut it it's some number of thousands of
hours that you need to spend on a GPU
and most people I don't think are gonna
spend that number like thousands of
hours on a GPU per year and if you just
start in the cloud and you find that you
are spending thousands of hours you
really haven't lost that much money
right you've retained your opportunity
costs therefore you can then think about
further getting some sort of local local
GPU power I think in the meantime you
can either get by with CPU and RAM or
get like a mid-range GPU that you can
just dev on locally and then just use
the cloud so anyways that's it for now
questions comments disagreements
whatever feel free shows below come hang
out with us in the disc or this disc or
GG slash Centex otherwise I'll see you
guys in some other video