okay so artificial intelligence machine
learning data mining data analysis
clustering classification data
pre-processing big data it's hard to go
anywhere now without hearing about AI
and machine learning and data data
particularly it's everywhere research
we've suggested that every two years we
generate more data than ever existed
before so the amount of data is doubling
every two years now that isn't
absolutely am you know astronomical
amount of data but the thing is of
course that this data doesn't
necessarily mean anything the fact you
can create tables of data but unless you
understand what's in them and what they
mean you haven't got any knowledge right
so there's a distinction between having
data and having knowledge so all very
well saying yes as a species we're
producing a huge amount of data but
actually a lot of it doesn't get used a
lot of it sits there on a hard disk
waiting for someone to look at it and
that's kind of what we're talking about
here if we want to extract knowledge
from data we're going to need some tools
and processes to do this in a formal way
and that's that's what data science is
right and things like machine learning
and AI have a place within it so perhaps
if you do this for your job then data
analysis is going to be useful for you
maybe your company's generating data and
you want to analyze this data but on the
other hand perhaps you're just a
consumer and companies are using data on
you they're generating data on you and
actually they're profiting from data on
you these are sometimes life-changing
decisions that are being made on your
data and so it's empowering to know how
this process works
and I'm a very simple example which you
might even do yourself suppose you go
online to book some flights for a
holiday and then you decide that
actually two flights via an intermediate
Airport is cheaper than a single flight
right you're doing data analysis say
you're taking lots of different data
sources and working out the optimal
route and this of course happens
automatically as well depending on the
flight website that you're using all
right so this kind of stuff you're
already doing it it's just a case of
trying to formalize this process so what
do any of the things I listed at the
beginning mean well one problem is that
everyone's definitions differ slightly
but also I think that a lot of these
terms are used completely
interchangeably AI is the classic
example so AI is everywhere right
talk you can't buy a product without it
having been having a I added to it a lot
of the time you CAI we're actually
talking about machine learning so
machine learning is the idea that we're
training a machine to perform a task
without explicitly programming it to do
so a good example of a IVA isn't machine
learning would be lets say a mouse in a
maze where all you're doing is telling
it to turn left or right at random not
learning anything it doesn't understand
what the maze is but it will eventually
get to the end right that's a kind of
rudimentary artificial intelligence that
doesn't involve learning anything
machine learning is about not giving it
conditions not saying if you're here
turn left if you're here turn right it's
just giving it examples and hoping it
will learn to perform most tasks itself
right so machine learning is a subset of
AI but they shouldn't be used
interchangeably if we're using machine
learning often what we'll do is we train
it based on samples of data so we'll
have some existing data set that we're
trying to train on and we're trying to
use the machine learning to either tease
out information or make predictions on
this data the problem is that not all
data is sort of made equal some of its
noisy and messy maybe we don't know what
it is and don't know whether we can
apply a certain technique to it right
and so we need to clean this data up we
need to take this data understand what
it is and extract some knowledge so that
we can then apply these AI or machine
learning techniques to it so this
combination of things that can take data
and prepare it in a way that we can then
use it or understand it that's data
science there are quite a few ways we
could do this data analysis right
throughout this course we could use our
we could use - we could use MATLAB they
all have their pros and cons we're gonna
use our because it's free and it's
really good for statistical analysis
it's got loads of great libraries if
you're really familiar with Python then
maybe that's what you want to start with
for this kind of stuff but we know we're
going to be working with R we have our
script area here where we can write
scripts and run scripts you can save
them and then come back to them later
console where we're going to be putting
in you know specific commands we have
our environment which is where all our
variables and our data is held and we
can look at them there and then we have
plots any plots of which you can do
quite a lot of different plots in our
very versatile that's going to appear
down here
okay so you've probably got everything
you need to get started with data
analysis in my opinion the best way to
get into is just to kind of have a go so
it's going to look at a few of the most
obvious things and it does it had
a little bit of a learning curve only
because it's syntax is slightly unusual
if you can program you'll be fine but
even if not you should get there pretty
quickly most of our time in our we'll be
using live a matrices or vectors or
which are kind of a special case of
matrices or maybe data frames data
frames a really nice aspect of our which
you can kind of think of like a table
that you might have in in Excel except
you've also got headings for your
columns so let's have a look at some of
these things and just a few of the
things we can do with them before we
perhaps go into a little bit more detail
in other videos so for example we might
look at our variable X which I've
created and X is a sequence going from 0
all the way up to a few multiples of Pi
which I used to create this plot that
was only one line of code that produced
that and I've used that to create my
plot by essentially saying y equals sine
X and then just simply plotting that if
you wanna get a little bit more
complicated we can start looking at
matrix data
so I created a CSV file with a Gaussian
function in it so essentially a two
dimensional array of values that get
bigger in the center very
straightforward the CSV file is
essentially a text file with commas
separating those values very easy to
read and write these out of Excel and
other packages and so they're off you'll
often find data is passed around in this
way at least moderately sized data if it
isn't too you know to it too huge I can
load this in using my Mead CSV function
so I can say name data now the arrow
operator is essentially equivalent in
our for the assignment operators or
equals equals will often work but I tend
to try and use this one so name date I'm
going to assign read dot CSV and the
file is going to be norm dot CSV and
I've got no header for this file so I
don't want it to use the top row for the
labels so I'm going to say header equals
false and that's loaded in name date and
we can have a look so I'm gonna click on
name data here and if we click on it you
can see we've got the rows and the
columns of our data in here we can look
at individual elements in this array so
we can say data at position three four
right and that's going to be the third
row down and a fourth value across we
can also leave one empty and just have
an entire row or conversely an entire
column like this and so it's very easy
to take ranges of values you've got a
huge table of data selecting certain
columns looking at certain columns
plotting certain call
this is one of the reasons why I was
very popular quite often when you're
looking at data will actually be looking
at something called a data frame now a
data frame I've got a load one up is
simply a in essence a table of values
but it will have to be the same type so
in an array normally they'd all be
floats or they'd all be integers in a
data frame there can be different things
so you could have first and last name
next to age for example so I've just
created a tiny little CSV file with some
random people in it so let's load this
up so I'm going to say name data sign
read CSV names dot CSV and if I look at
name data you can see that it's got
three columns it's got first name
surname and age and five rows and
there's five people in this dataset and
then you can do just like I did before
but now we can also index by the names
of these columns so I could say I want
all of the first names for example so I
can say name data dollar first-name and
I can see all the different first names
so you can start to look at this data
set and more in more detail obviously
this isn't absolute tiny data set but
you get the idea you could also look at
individual instances so we could say
name data and I want just the second row
for example name data the second row
there we go Bill Jones and he's 18 years
old as we move through these videos it's
going to be very common for us to load
in datasets like this in this format and
then start to process them based on
these data frames so perhaps an example
right so so let's imagine you're an
online retailer and someone comes into
your shop and buy some things and maybe
they you trying to understand what it is
what they do so that you can let's say
send them emails to try and get them to
buy more products or show them
recommended products and things like
this so you want to try and build up a
pattern of their behavior right and all
you've got is what they click on what
they add to their basket and what they
buy right so you've learned that they're
looking at these kinds of items and they
look at these ones regularly and then
sometimes they just buy something
completely random seemingly and that
goes in their basket and gets bought
straight away maybe it's a present right
so maybe it's not tied to them as a
person so you're taking all of this data
all of these purchases all of these
products are they're looking at and
you're turning this into a kind of
picture of this person and you're
clustering that person in with other
cuttance you
that bought similar things and trying to
predict what they want to buy next right
and that's when you send them an email
say you should look at this one because
this one's really good and you didn't
buy it last time but you'll definitely
want to buy it this time
so we've got some data we want to
extract some knowledge what's the first
thing we do well we have to start to
look at it and try and tease out some
kind of information right or analyze
this data the data analysis is the idea
of using statistical measures to try and
work out what's going on this is kind of
a cycle we're going to analyze the data
so we're going to do a data analysis and
perhaps sometimes just using statistics
to analyze the data isn't enough you
can't really learn everything about it
yes you can learn you know
mathematically how it works but you
might not understand about what it all
means so visualizing the data can be
really helpful so what we'll also do is
we'll visualize the data visualization
so that's going to be charting it
plotting it trying to work out trends
and links between different variables
and things like this and these are kind
of being back and forth right you could
do both of these things numerous times
and work out what we've got right so
you're gonna do something like this and
then what we're going to do is we're
going to pre-process the data often
you'll be finding your recording much
more data than you actually need right
this is certainly true of an online shop
I'm going to be looking at a lot of
products but I don't end up buying and I
was never really going to buy I know
maybe a pipe dream and they've got a
sort of weed out this information to
work out what it is that they might
actually better convince me to buy right
so this is going to you and a
preprocessor data and remove a nonsense
and drill light down to the stuff that's
really useful so this is pre-processing
and this is going to be a kind of cycle
of analysis and visualization and
pre-processing and we can repeat these
things and then we can really drill down
and whittle down our data into the most
usable sort of core of knowledge that we
can and get the most out of it now it
may be that just analysing the data is
enough right you've now sort of you've
obtained some knowledge you kind of
understand what the trends are and maybe
that was all you wanted to do that's
sometimes the case maybe actually what
we want to do is take things a little
bit further we're going to use machine
learning or modeling to try and model
this system and predict what's going to
happen next so for example in the case
of an online shop we might want to start
predicting what people are going to buy
next and if we can do that that's when
we can send out these emails or flag
things in their recommended items and
get many more sales as an example let's
imagine that someone has spent a lot of
time
DIY tools my I've you know recently
moved house I spent a lot of time doing
DIY and I'm always trying to buy new
tools because it just seems like a good
idea so you know maybe I buy a certain
kind of saw and then you know a few
months later they're starting to
recommend me it's like a different kind
of solver so it's a slightly different
purpose that suddenly I definitely need
to be doing and I think another yeah
maybe I will buy that and then the end I
have 10 saws and I don't know how to use
any of the source but you know the
retailers job is done it's if we want to
extract this data we're going to use
machine learning or modeling to put to
model this system and make predictions
right now so for example we could
customer data together we could link my
purchase history with similar people
what are they buying can I be tempted to
buy those things as well right maybe I'm
very different from someone else and so
it's not a good idea to recommend me
certain products because I'm unlikely to
buy those things perhaps use a different
example in the medical domain it's quite
common to classify people into kind of
risk categories right so that we can
maybe use preventative treatments so
every time I go to a doctor they're
going to collect data on me on what I
can't cope what's currently one with me
and what was wrong with me before and
combine that with with you know standard
data like how much exercise someone does
and you know their family history and
how what their stress levels are and
things like this we can combine all
these things to make a prediction as to
what they were at risk of in the future
so you know heart disease or something
else like this it could save someone's
life if you spot that they're at risk of
a certain thing and you can really
advise that person to you know increase
their level of exercise or alter their
diet there are two other terms that we
come across you know a lot right so
there's data mining and big data right
now I'm not really sure what data mining
is because I don't think anyone is it's
a bit it's a bit of a buzzword really
what data mining is is a combination of
pre-processing your data and maybe using
clustering to extract some knowledge
from it right so that's our sort of it's
a word that's come to be used in place
of those things right if someone says
they're doing data mining that's what
they're doing they're pre-processing and
extracting some knowledge from their
data it's a night it's a cool sounding
word you're not actually mining anything
right you're just doing what everyone
else does on data big data is the idea
that maybe we've collected a lot of
examples of something you know a huge
number or each of our examples is quite
complicated and
a lot of variables right in that case
the amount of data we've got is sort of
unwieldy right so I would argue perhaps
that big data is not data that you can
run on your laptop like you might be
using cloud compute infrastructure or
certainly parallel processing in some
way to to pre-process and analyze this
data right so exactly where the light
how big is big I don't know but exactly
where we do over line in some ways it's
not really important right the idea is
just that the amount of data we as a
species are now producing more and more
of our data is becoming big data but you
know exactly where the cutoff is isn't
it's not doesn't really matter what is
data right I'm pretty sure that's data
right is this data you know Chur or
that data is this data what what is data