what's up everybody and welcome to
another deep learning with Python
tensorflow and chaos tutorial series in
this video we're going to be continuing
along on our little mini project of
implementing or recurrent neural network
to predict the future price movements of
a cryptocurrency
based on the sequences of that
currencies prices and volumes along with
another three cryptocurrency prices and
volumes I think I got that right all
right so let's get into it so so far
we've we've got the data we've merged
all the data and we've created targets
now the next things that we have to do
we do have to make sequences out of that
we have to balance normalized scale all
that fun stuff but the next thing I want
to focus on is out-of-sample so before
we start you know shuffling and doing
all these things we need to immediately
separate out our out-of-sample data now
in the case of temporal data and time
series data and really a lot of
sequential data I would argue you can't
just shuffle and then take a random 10%
and the reason why I can't do that is so
our sequences are you know 60 minutes
long and we have a six you know sixty
units in each sequence right and we're
predicting out three minutes the the
problem is if we were to shuffle this
data and add it and then just pick a
random you know 10% the out-of-sample
samples would all have very very close
examples in sample and it would be
relatively easily
relatively easy for the model to as it
over fits on the in-sample data to also
just that that overfitting pour over
into out of samples so instead what you
have to do is sequential data in most
cases is take a chunk of the sequences
and separate them away now in the case
of Time series data I would take that a
step further and say really
out-of-sample needs to be a chunk of
data in the future so in the case of our
data we want to take
let's say the last five percent of the
historical data and separate that out as
our out-of-sample data so that's what
we're gonna do next and I think that's
the most realistic way to do
out-of-sample testing it aside from
maybe doing like a true forward test
like using all your data and then
testing it forward
in this case this is basically the exact
same thing it's as if we built the model
you know 5% of time ago and then forward
tested it so anyways that's what we're
gonna do here so I can't tell you how
many times I've seen you know finance
stuff or even just any sequential stuff
but it's usually with finance that
people do you know time series data
where either they don't do out-of-sample
and they're like wow look how good my
model fit and it's like whoa you know
with a big enough neural network and
enough epochs you will absolutely over
fit your data to a perfect or almost
perfect fit and but they don't do true
out-of-sample or they screw up on the
out-of-sample doing a method like I just
said so it's really important to keep
that in mind all right
so let's go ahead and say like our last
underscore 5% of time that's gonna be
equal to and in fact let me just do this
let's first take x and we're just gonna
say x is the sorted they should be in
order but we want to this is really
important that they are in order so
we're gonna sort them sorted and then
main the F dot index values so dot index
just references the index values
converts it to an umpire rate so those
are the times now what we want to find
is where's the threshold of like the
last 5% of times like what's the what's
the actual unix time that is the
threshold of 5% so the way that we're
gonna grab that is we're gonna say last
fought last 5% is equal to x and then we
want to get the index values so the
index value should be something like you
know let's say the it should be
something like negative I forget how
many things we have it's probably like
let's say if you had a hundred thousand
it would be about the index of negative
five thousand right or it would be the
index of 95
thousand something like that so the last
5% will be x - or the index is a
negative not - int the int value of 0.05
times the Len of really anything it
could be main DF but we'll say x so
that'll give us the threshold so let's
just go ahead and print that out last 5%
I'm gonna comment out that now and let's
just run that real quick and see what we
got okay so we actually have a timestamp
now so now we can actually separate out
our validation data or out of sample
data and our training data and we're
gonna do that now before we do anything
else we want to separate that out that
it can kind of create a problem with
normalization stuff like that I'm not
gonna worry about that right now and
scaling but not gonna worry about that
the mo it's far more important that we
get this right than anything else and we
can figure out a better way later but we
definitely want to make sure we get this
right so we're gonna say validation
underscore main underscore DF equals
main oops that's main DF where main
underscore DF dot index is we'll use
greater than or equal to last 5% so that
is sort validation data is the basically
anywhere where the time stamp is greater
than this value so it should be the last
5% and then we want to do basically the
exact same thing for the actual training
data which will retain the name main DF
and main DF equals main DF where main DF
index is the less than last 5% so now we
split up the data and the next thing
that we'd like to be able to do is now
you know we need to create our sequences
we need to balance we need to scale and
probably other things I'm not thinking
of right out my head yet and so what we
want to do is make a function to do all
those things but we're going to do all
those things on both of these are
basically we want to make some code to
do all those things we need to do
both of these so we need to make a
function so what we want probably is a
function that's gonna do something like
allow us to do this like train X train y
equals some sort of function I'm gonna
call it pre process D F and then you
just pass a D F so tray next rain Y will
be mean D F gets passed and then we'll
do the exact same thing for validation
validation X and validation Y and we
want to pass the validation main D F so
that's that's what like what we're
hoping to be able to do so I'm gonna
comment those out for now and now we're
just gonna go up to the top here and we
are going to start creating our
pre-process D F function so define
pre-processed
DF it's going to take in a DF as a data
frame as a parameter and I think the
first thing that we'll do is the scaling
and to do that we're going to from SK
learn import pre-processing again if you
don't have this it's a pip install SK
learn I think it's a scaler it might be
a sidekick learn I actually don't know
what the C you know I pull this down I
think I need that to be running let me
do pip install SK learn okay yeah so pip
install SQL urn I didn't know if it was
psych hit you know - learn or what
that's what the package is called
anyways okay so we get the
pre-processing and now we'll come back
down to this data frame that are the
function that we're gonna build and
first we're gonna say DF equals gift a
drop the what we want to drop is that
future column we don't need it it
carries no significance to us anymore we
only needed that column to generate the
target and you definitely wouldn't want
to accidentally leave in the future
column because that would be that would
allow the network to learn really really
quickly
how to predict the future movement so we
want to drop that the next thing that
we're going to go ahead and do is we're
going to iterate over the columns and we
want to scale those columns so what
we're gonna do is for call in DF columns
what do we want to do well as long as
column is not target because we don't
actually we don't actually mmm target so
there we go we don't actually want to
why are you angry at me we don't need to
scale we don't need to normalize when I
do any that to target target is done
it's a completed column we don't need to
do anything more to it now for the other
columns we do need to do some stuff one
of the things we need to do is so DF Col
now is going to become DF Col dot
percent change and the way the reason
why we want to do that is this
normalizes all the data so whether the
price it moved from ten thousand to ten
thousand and five or it moved from you
know a few pennies to a few more pennies
percent change normalizes that because
you know bitcoin is a different poison
then litecoin is is a different place
price than aetherium and so on so this
helps us to do that same thing is true
with the volume the volumes are you know
different in terms of magnitude but
they're actual movements you know the
trends of their movements or what are
what we're trying to analyze for their
relationship to other prices and stuff
so anyways we normalize everything with
a percent change then we're gonna do a
DF drop in a in place equals true
percent change will definitely spawn at
least one at the very beginning but just
in case these things love to creep in
and they're gonna cause a heck of a lot
of trouble for us so we're gonna drop
them if they appear then we're gonna say
now DF call is equal to pre-processing
dot scale and what do we want to scale
we want to scale DF call dot values so
what is this so pre-processing is from
SK learn there is some pre-processing in
scaling with Karros I have used it I
don't know how it works in general I've
been relatively unhappy with what how it
scales the values for some reason I
don't see it actually perfectly scaling
from 0 to 1 which is the target I am
sure there's a way to do it it's just
that this works just as well so so I'm
gonna use that I like I said I'm aware
it exists in Karros and I think the cool
thing with Karros is you can you know
multi axes data you can scale as well
and I don't think you can do that with
scikit-learn
but again I've not tried anyway many
ways it's gonna cat you also could scale
like it's like the current value / min -
max I think or something whatever that
is there is there's a really simple
formula that you could scale with so
however you want to do it just scale the
data try to get it to be between 0 & 1
so this is how I'm gonna do it for now
if that bothers you feel free to do it a
different way okay so now that we've
done that at the very end
let's do you have to drop in a again JA
same case so if this creates you know an
na which it definitely does the percent
change we're going to drop it and then
this here just in case for whatever
reason that creates a not a number we're
gonna make sure we drop it again okay
let's do I think we got time well stuff
in the sequential data as well so now
we're gonna say is sequential data is
equal to an empty list and then we're
gonna say priva
is equal to and we're going to use a day
Q max Len actually should be one word
max align equals whatever the sequence
length is we need to go ahead and import
dick use so I believe it's from
collections import day q now day q is a
fancy little dude what it does is so
previous days is dia q with a maximum
length of 60 so it's based you can think
of it like a list and as all you do is
you just keep appending to this list but
as this list reaches the length of 60 as
it gets new items it just pops
the old items and so we don't have to
write that logic it just does it for us
so what we want to do is we're gonna
wait until predates at least has you
know sixty values and then from there
we're just gonna keep populating it and
that's going to be our sequence so so
now what we're gonna do is for i ND f
dot values so what is d F dot values
going to be so probably well I think
what I should do - let's print D F dot
head just so everybody you have to head
as well as for c in d f dot collins
print see i just wanna make sure
everybody knows because at this point
we're starting to get a little crazy
pre-process DF we need to at least run
i'm just gonna come down here hopefully
i don't forget to undo this and I just
think hopefully this will help to kind
of visualize what's happening here do
this so we've got our target
everything's been converted to percent
change as well as been scaled so and
then these are all the columns that
we're working with and these are the
columns in order so for input data to
measure what the target is we're going
to use all of these which is litecoin
prices theory and prices Bitcoin prices
Bitcoin cash prices and so on and then
we've got target so the sequences will
be sequences of prices and price and
volume for all of those but it can't
contain the target because again that
would be cheating right so what we're
gonna do is come back up to where we
were working and in fact let me I guess
I'll leave that for now but I'm gonna
delete this stuff so what we want to do
is iterate over the columns so for I in
DF top values so DF top values just
convert your data frame to a list of
lists so it won't contain time anymore
but it's in that it's in the order of
the index still but it is going to
contain target so we need to be careful
so for I so
is just the row of all the columns so
you know Bitcoin clothes Bitcoin volume
if they are enclosed
aetherium volume and so on so what we
want to say is as we iterate over this
data for I and D F values what we want
to say is Priya's preview score days dot
append and what we want to append is a
list because it's because the sequence
is going to be a sequence of lists what
we want there is n for n in I up to
negative 1
what's going on here and for n is each
value in that list a list which again it
is each of the columns right so the
close the volume the close the volume
but also includes targets so we're gonna
say n for n in I up to the last I so we
are not taking target hopefully that's
as clear as possible so that gives us
previous days now if previous day's is
equal to the sequence length that we
require so if Len prove day days equals
60 and actually not see 60 but see clang
whatever that is it is 60 right now
we're gonna say sequential data dot
append and now we're gonna append our
exes and our Y's so our features and our
labels that's going to be enough it's
going to be a numpy array of the
previous days so it's that sequence of
60 features or 60 feature sets I guess
we could call it so append that and it
really this needs to be a list so it
though it's the features and then the
label so the label is pretty simple it's
just I negative one so that's the
current label right now what's the
current label so based on the last 60
minutes of all that data we are hoping
the model could predict yes that label
either a 0 or a 1 okay so we build
sequential data and when we're done we
are going to do a random not shuffle
sequential data
we probably have random we do not so
we'll import random let's run that see
if we hit an error we do numpy not
defined we're gonna need that as well if
you don't have numpy installed
do pip install numpy we will try again
this point this step can take a little
bit because you're building sequences
but it's done okay so I believe we're
done I don't think I'm gonna continue
until the next video this one's probably
pretty long okay so now we have our
sequences we've got our targets we are
closing in on the ability for us to feed
this through a model we still have a few
more things we need to do but we are
getting close so this this one's a
little dense so as the other ones have
been kind of dense if you've got
questions on whatever is happening here
or if you think there's a better way to
do something please by all means let's
talk about it in the comment section let
me know otherwise I will see you guys in
the next tutorial where we're hopefully
gonna get closer we probably won't train
the model the next one I'll probably the
tutorial after the next one but we'll
continue trudging along so anyways
questions concerns whatever feel free
lean below otherwise I will see you guys
in another video