hello world it's Suraj and today we're
going to learn how to prepare a data set
before feeding it into a machine
learning model our example code will
predict if someone is meditating or not
by training on a data set of brain scans
data is raw information it's a
representation of both human and machine
observations of the world everything can
be represented as data all science art
literature all of it can be represented
as ones and zeros on a computer when we
enter a virtual world we are literally
surrounded by data since it is the
fundamental building block of everything
we see and when we observe something
physical in real life it becomes data in
our brain unless our universe is a
simulation then everything you see is
already data if you don't work at a tech
giant how are you supposed to get that
data that brings us to step one in
preparing data deciding the right kind
of data to use the data set you use
entirely depends on the problem you're
trying to solve if I want to build a
chatbot that comes up with new
innovative product ideas I'm not going
to use a data set of Tim Cook dialogue
data is a means to an end and the good
news is there is a public data set for
almost any topic you can think about a
couple of sites I like to use to find
cool data sets are Kaggle since I love
the format of their website and how they
explain each of their data sets in
detail also the data sets subreddit is
great for requesting data sets you want
and there is this awesome list of public
data sets in the readme of this github
repo that I'll link to in the
description Google's advanced search
feature is also super helpful
usually combining a few keywords with
the word data or database is enough to
find what we need and to make it easier
we can specify the type of file we want
like CSV and a type of domain like edu
or gov usually a website has an API that
makes it easier to get the data you need
but if it doesn't you can use a library
like beautifulsoup to take a raw HTML
web page and just scrape the data
directly DIY data died once we decided
the type of data we want our second step
is to process it we're going to write a
function to extract data from a brain
scan data set then we can feed that data
into a single layer neural network
created in tensorflow the network will
create a separator line between two
classes so that given a new person's
brain scan data it can predict if they
are meditating or not let's take a look
at this data this is a list of
neurological metrics collected via an
EEG device for a set of human volunteers
there are two possible classes either
meditating represented by a 1 or not
meditating represented by a 0 and there
are 3 features for this data a measure
of mental focus a measure of calmness
and the volunteers age want to format
our data properly data could come in the
form of a text file or a relational
database or like what we have a CSV and
there's a library to convert pretty much
any file type to another so make sure
you have your data formatted to a file
type that you most feel comfortable with
once it's in the right format we'll want
to clean the data sometimes we have
instances in our data that are
incomplete we can iterate through each
row and delete those instances by
checking if the value is empty or not we
should also decide what features to use
deciding what features are important is
one of the key parts of data science if
we don't use the right features our
model will make bad predictions we only
want to use features that are relevant
to our problem their gender has nothing
to do with their meditative state so we
can totally disregard that feature so
let's first create two arrays one array
will hold our class labels the other
array will hold our features we can
iterate through every line in our CSV
file using this for loop we'll define a
row which is a single instance as an
array of values by splitting the line by
the comma separator using this row we
can first get the Associated class label
by retrieving the first value in a row
array converting it to an int then
adding it to our labels array now we can
do the same thing for our features array
we'll take each feature and convert it
to a float since we want precision in
our values our feature array is now an
array of arrays now that we've pulled
our data from our data set file into
memory we've arrived at the last step
transforming the data
one possible transformation is
decomposition sometimes we have features
that are too complex like the date if
we're trying to predict which day in
October is most likely to get rainfall
this year we don't really need the month
and the year if we decompose that
feature into just a day of the month
that'll make our model more accurate
since we're satisfied with the features
we have in their class labels will
perform the only transformation we need
we'll transform them into vectors
vectors are numerical representations of
features all features can be represented
as vectors words images videos all of it
we can take these vectors and feed them
into our neural net directly we'll
convert our array of arrays into a 2d
matrix using num pies matrix function
and set the type to float this is a
matrix of feature vectors each vector
contains a list of features for an
instance we'll also want to transform
our class label array into a numpy array
because a numpy array can easily be
converted into a one hot matrix then
we'll return our fully processed feature
matrix and one hot label matrix so what
is one hot encoding yo DJ drop out bump
it
I was bored so I made in the right I put
words in like cat dog duck and fillet if
i encode them as numbers it's a ok one
two three four all on display but if 4
is more than one is fillet more than cat
no but I'm not gonna leave it at that
cuz I'm not trying to rank differentiate
yeah differentiate so I'll make each a
vector of ones and those different but
not right that's just how we go 3 no 2
no one hot and code yeah once we have
our data processed we'll want to feed it
into our graph in tensor flow the
placeholder object is considered the
gateway for data into our computation
graph so we'll want to initialize two
placeholders one for our class labels
and one for our associated feature
vectors and when we finally run our
training step using the run function we
can feed our data into the graph using
the feed dictionary parameter the label
placeholder gets the labels and the
feature placeholder gets the features
when we run our model it will show the
classification line that
created to separate the meditating from
the non meditating people and if we feed
it a new instance boom it will classify
it just like that so to break it down
the steps to prepare a dataset are
selecting the right data processing it
and transforming it you can easily find
public datasets on the web via a number
of sources that I'll link to in the
description or use a web scraping tool
like beautifulsoup to create them
yourself and once we have our data we'll
convert them into vectors which are
numerical representations that our
machine learning model can understand
the winner of the make a game bot
challenge is Karl BIC Eisen he created a
bot that used deep q-learning his bot
fed the pixel data it received to a
convolutional neural net that updated
its policy over time through trial and
error also this was his first github
repo badass of the week and the
runner-up is Rohan Verma while my demo
bot couldn't finish a lap his could in
just two and a half minutes the coding
challenge for this video is to write a
script that classifies a pokemons by
their element using a data set that I'll
provide post your github link in the
comments and I'll announce the winner in
the next video please hit that subscribe
button and for now I've got to
decentralize the web so thanks for
watching