today we'll be summarizing articles that come together to help us understand
different aspects of big data we will explain how bad data can affect your
machine learning tools secondly we will look at the top ten learning algorithms
for beginners but before digging in I want to quickly go over what machine
learning is machine learning is a method of data analysis that automates
analytical model building the models can learn from data identify and predict
patterns and make decisions with minimal human intervention
first we start with the bad it has been a saying I have heard and used many
times as a quality data analyst garbage in garbage out and I stated in the
article if your data is bad your machine learning tools are useless by thomas
reddmann poor data is the number one enemy when it comes to widespread and
profitable use of machine learning bad data can negatively affect machine
learning by affecting the historical data used to train the predictive model
which in turn affects new data in the model that is used to make future
decisions to avoid training a particular model using bad data the historical data
must be exceptionally broad and high quality standards one must also have the
right unbiased data data should be correct properly labeled des duped and
biased unlike most data quality work which
focuses on one criteria or another the machine learning you have to work on
multiple criteria simultaneously mostly two today fails to me data are right
standards some examples of why this happens are listed here for example
video creators not understanding what is expected of them poorly calibrated
measurement gears overly complex processes and of course human error data
scientists spend up to 80 percent of their time cleaning data before training
a predictive model to compensate for these errors that might occur and even
with all that time consuming tedious work not all errors are detected or
corrected an undetected minor error in one predictive model that feeds another
model and another and so on it can cause unforeseen errors and go from a minor
error to a large error across an entire process
to avoid this from happening five steps should be taken according to Thomas
Redmon's article if your data is bad your machine learning tools are useless
first clarify objectives in assess whether you have the right data to
support such objectives ask yourself does the data falls short of the
objective if so find new data or scale back objectives or both build plenty of
time to execute data quality fundamentals into your overall project
plan for training for every person month of building a model you should spend
four person months of cleaning for implementation it is best to eliminate
the root causes of error the article suggests to start this work as soon as
possible but at least six months before you let your predictive model loose
maintain an audit trail as you prepare the training data it is good practice to
maintain a copy of your original training data the data you use to train
and then the steps taken to get from one step to another charge the specific
individual or team with responsibility for data quality as you turn your model
loose this pressure should know the ins and outs of the model they should
enforce standards for quality data of incoming data and they should lead
ongoing efforts to find and eliminate root causes of error that should have
started prior to the model going live obtained independence rigorous Quality
Assurance an internal QA department or team from outside the department or a
qualified third party should carry out this process of ensuring that the
Quality Program provides the desired results even after following these five
steps the article states that you will find that your data is still not perfect
and this is when you should pair your scientists with the most experienced
business people when preparing the data and training model Laurel corn off of
stratify Inc mentions partnering data scientists with business people as they
have dealt with that data forever and so doing this it is possible to build their
expertise into the predictive model there are three types of machine
learning algorithms supervised learning algorithms unsupervised learning
algorithms in reinforcement learning algorithms supervised learning
algorithms are labeled training data to learn the mapping function for the input
variables to the output variable there are two types of supervised learning
problems in the classification problems the output variables in the form of a
category such as male female or healthy sick while on the regression problems
the output variable is in the form of real values such as a high or weight
unsupervised learning algorithms use unlabeled training data to model the
underlying structure of data and is only associated with an input variable and
not with a corresponding output variable two types of unsupervised learning
algorithms are Association which is used to discover the probability of a
co-current of an item in a collection and clustering used to group samples
such that objects with the same cluster are more similar to each other than
others in another cluster reinforcement learning algorithms allow agents to
decide on the best next action based on the current state by learning behaviors
that will maximize the reward learning is usually done through a trial and
error process the first steps in the article 10 taught machine learning
algorithms for beginners by Renisha are all supervised learning algorithms the
first we're going to see is linear regression linear regression is
expressed by the equation shown the goal is to fit the line that is nearest to
most of the points and this will reduce the distance an error between the Y
value and the data point the logistic regression is expressed by the equation
shown and its goal is to use a training data to minimize the error between the
predicted outcome and the actual outcome logistic regression is best suited for
binary classifications cart or classification and regression trees are
an implementation of decision trees which is made up of non-terminal and
terminal nodes non-terminal nodes are the root and the terminal nodes are
known as leaf nodes non-terminal nodes represent the single input value along
way a splitting point on the variable while
terminal nodes represent the output variables naivebayes
call naive because it assumes that all variables are independent of each other
is used to calculate the probability of an event will occur given that another
event has already occurred and nearest neighbor algorithm abbreviated Canon
uses the entire data set as the training set rather than splitting the data set
into training set and test set the Kanan algorithm goes through the entire
database to find the k-nearest instance when an outcome is required for a new
data record the next three algorithms fall into this unsupervised learning
algorithms first is the pori algorithm it uses a
translational data base to mine frequent itemsets and then generate Association
rules the pori principle guides the support measure for this algorithm it
states that into the item sets is frequent then all of its subsets must
also be frequent for example in the article it states that if a person buys
milk and sugar then the same person is also likely to buy coffee powder the
k-means is an interactive algorithm that groups similar data into clusters the
principal component analysis known as pca
captures the maximum variance in the data into new coordinate systems with
accesses called principal components this algorithm makes data easy to
explore and visualize by reducing the number of variables assembly learning
techniques combine results from multiple learners to improve results by voting
during classification or by averaging during regression baggers with random
forests use multiple trees which are constructed using a random sample of
records and each is split by constructing using a random sample of
predictors the random samples are components from the original data set
boost e with Eddy boost known as adaptive boosting uses sequential
assembly where each model is built based on correcting the misclassifications of
previous models so models are built by assigning greater
weights on previously misclassified instances of previous models