when you hear that 70% of trading volume
in the entire stock market is generated
by algorithms you might think that
you're missing out something big are we
the only fools in the market who still
trade the traditional way if the
missions dominate the market today you
ask mere mortals even stand a chance
against the mighty machines while a big
chunk of that automated 70% is a result
of high-frequency trading algorithms
trying to predict only milliseconds into
the future and those algorithms usually
use very very simple methods nothing
fancier than a chain of hard-coded rules
or a simple linear regression model so
yes if you race against milliseconds
then you do need to be a machine but for
casual investors like us that shouldn't
really matter or should it deep learning
models can learn much more complex
patterns in data so as it possible to
predict longer-term price movements in
the market using deep learning nobody
knows for sure many large financial
institutions are hiring data scientists
machine learning engineers and deep
learning experts with hefty salaries so
does that mean it works for sure well
that might give us a sign about the
trends in investing strategies but
institutional investing can be different
from individual investing for example
unless you are buying a very low volume
stock the shares you buy or sell barely
have any impact on the price but if
you're buying and selling in large
amounts how you execute your trades can
make a big difference a machine learning
model can help decide how you should
split up your sales over time to avoid
causing big price movements to actually
predict the price movements you can try
a lot of things from very simple things
such as training LSD amps or temporal
convolutional networks on historical
prices to overly complex models such as
training convolutional elysium zhan
satellite imagery to predict
macroeconomic movements any predictive
model you may build practically tries to
find some inefficiencies in the market
so in a fully efficient market none of
these models should work for example
your model can analyze text from various
sources such as financial news websites
and social media to decide whether a
particular stock will go up or down you
can do sentiment analysis at character
level text or audio you can analyze not
only what's in an earning statement but
also the
it's announced however the efficient
market hypothesis states that the stock
prices reflect all available and
relevant information immediately it's
likely that any new information that can
impact the prices is already
incorporated in the price by the time
your model gets to parse it from the web
if you think this hypothesis is true
then using any data beyond the prices
would be redundant then should we just
use nothing but the historical price
information to build a predictive model
that saves us a lot of trouble
technically that's the opposite of what
the hypothesis says but for now let's
forget about that and talk about what we
can do with historical price data trying
to forecast the direction of prices by
finding patterns and past market data is
a form of technical analysis these
traders do this all the time they look
at the charts and name the patterns they
say things like head and shoulders cup
holders shooting stars etc neural
networks are very good at finding
patterns in data if there are really
such patterns and neural network with
enough capacity will be easily able to
pick up any patterns that might lead to
profit give a neural network a price
chart then it will fit a function to
that chart as closely as possible the
problem is that being able to find
patterns and past data doesn't mean that
those patterns will generalize and hold
in the future as well a neural network
can even find patterns in completely
random data but what the model actually
learns wouldn't be any more useful than
a lookup table a table that holds price
information for the past data while
having no predictive power this is
called overfitting you can check out my
earlier videos to learn more about it
machine learning models are not the only
ones who find patterns in data that
don't exist
humans do too we see faces in the COTS
and inanimate objects we see patterns in
price charts and assume that the prices
will regress to the mean I generated
this chart for example completely
randomly by changing the price by some
random percentage at each point yet
there seems to be some pattern in the
chart so I will be cautious about
technical analysis there are so many
books and success stories about how some
authors got rich by doing technical
analysis but honestly most of those
authors are getting rich by either
writing those books about how they got
rich or they just happen to be lucky yes
the odds of
such luck is not very high but there's a
lot of survivorship bias because losers
hardly ever write books about their
failure stories if you gather 1000
people in a room and ask them to predict
coin flips the outs with at least one
person predicting the outcome of eight
flips in a row is over 98% so it takes
only a room of people to find a
clairvoyant and there are many many more
people trading in the market earlier I
mentioned that according to the
efficient market hypothesis any publicly
available information that might have an
impact on the price of an asset has
already done it past market data is
publicly available information so any
information it might have such as some
patterns is likely to be already in the
price in an efficient market not all
markets are efficient or rational though
for example something as simple as the
log periodic power law model was able to
predict the 2018 Bitcoin bubble eight
days before the bubble burst it doesn't
mean that it'll predict the next month
or whether there will be another
cryptocurrency bubble at all so it's an
interesting observation in hindsight
okay let's say you want to experiment
with machine learning models to predict
the stock market just for fun so where
do you start you can start with defining
the goal of your model and pick a
corresponding loss function for example
if you want your model to pick the best
stocks among some options you can treat
this as an N wait classification problem
and use a soft max cross-entropy as your
loss function if you want your models to
give it a rating between 0 and 1 to any
given stock that you can use sigmoid
cross-entropy
next you can go on to design your model
architecture it doesn't have to be a
sophisticated model you can stack one or
two layers of LS CMS or gated recurrent
units for example you can even use
temporal convolutional networks which
are simple and easy to train you can
check out my earlier video and recurrent
models to learn more
what about Hardware GPUs are one of the
first things that come to mind when the
topic is machine learning in this
particular case you probably won't need
any special hardware if you already have
a GPU then use it otherwise unless you
plan to use some extraordinary large
volume data any model that is too big to
train a CPU will wildly over fit so
whatever hardware you already have will
probably be enough to get started if you
want to speed up your experiments using
GPUs you can always use
hot services and pay as much as you use
one of the problems we have with using
stock market data in deep models is that
we don't have enough data to train a
large model without overfitting to
reduce the risk of overfitting you can
do all sorts of data pre-processing and
augmentation for example you can add a
small amount of noisier data and you can
pick a random subset of stocks for each
time interval at every epoch and
generate new samples by taking random
linear combinations of existing stocks
those samples should essentially behave
like randomly managed mutual funds or
ETFs if you want to use data other than
just price and volume information you
can look into Google Trends to see how
much people are searching for particular
keywords however any signal you get from
there is likely to be lagged none of
these will probably work so I wouldn't
expect too much but give it a try anyway
you'll learn a lot while you try even if
it doesn't work if you find a trick that
no one has found yet and it does work
have fun and enjoy the profits anything
I say here is clearly not an investment
advice I don't have financial background
I'm a computer scientist who played a
little bit with financial data just for
fun I do have a decent amount of
experience and machine learning but my
area of expertise is its applications on
imaging data alright that's all for
today I hope you liked it subscribe for
more videos while keeping in mind that
this is not the type of videos that I
usually make and as always thanks for
watching stay tuned and see you next
time