[MUSIC PLAYING]
JASON MAYES: So before you have any chance of creating a custom
model, you first need to find some training
data that the untrained model will be able to learn from.
In supervised learning, this is a set of labeled data
for which each entry has an example answer of what
you're trying to predict.
And you might need thousands of examples to get good results.
You may remember from chapter 2, in order to classify something,
you will need to define some features or attributes that
may be meaningful in helping you make a prediction.
Previously, you saw an example for classifying fruit using
weight and color, for example.
But how do you use this collected data?
When looking at real-world examples,
you may see engineers reserve 80% of the available data
for training the model and 20% is set aside
for testing it once trained.
In this case, the flow of training a model
would look like this.
First, you train the model on the training set.
You then evaluate the model on the test set.
And finally, you keep tweaking the model's parameters
based on how it performed on the test set.
You would do this a number of times,
and then finally pick the model that
works best on the test data.
Now, this is a good starting point.
But there's a chance the model will try and change itself
to simply perform well on the test data alone.
Instead, you can randomly split your available data
into three sets.
One for training-- maybe 70% of our data.
One for testing-- say, 15% of our data.
And the final one for validation,
the final 15% of the data.
Exact splits will depend on how much data you have available,
so you may see variations of this.
Now, in this case, the training process is slightly different.
First, train a model on the training data.
Next, evaluate the model on the validation data.
Tweak the model to improve itself.
Repeat this a certain number of times,
and then pick the best model that
works on the validation data.
And then you can test that on the test
data set to see how well it works
on data it never saw before.
This allows you to spot when the model's overfit itself
to the training data, which is where it essentially
learned examples perfectly and not generalized well.
Instead, you test it against examples it never
saw before to check for this.
And this data is never used to make changes to the model,
so it can never accidentally learn from it.
Ultimately, you want a model that
can generalize well and not just learn from the training
examples, as the real world will contain unseen examples.
But how do you obtain such data in the first place?
First, if you're lucky, you may already
have the data contained in your existing systems
that are recorded in a form that can be directly used
to train a model.
Common sources for this include databases or server
logs that you have control of or are allowed to use and contain
the features and attributes that you require in the right form.
For example, maybe you're the owner of a recruitment business
that has a database of users who have
used the site for many years.
During this time, you've managed to collect
the users' attributes, like years of experience someone
has, the city they are from, and their current salary
expectations.
With this data, you might be able to train a model that
can guess the expected salary of a new user who joins the site
before they even tell you what it is, just
by using their experience and location as inputs.
As everything was recorded manually and you can verify it
all exists and is accurate, the data quality
is likely to be high, the cost to access the data is very low,
and at time to obtain the data is very short.
This is essentially the ideal situation to be in.
And it should be noted that very few people have
enough data in the right form that they
can use to train a model like this out of the box.
A more common situation is that you
have to start recording the data you need
once you define what it is.
And the first way to do this would be to do it yourself.
You could update your system to start recording such features,
or you could use new sources, like a questionnaire sent
to your users, recording the results
in a spreadsheet or database for the attributes that matter.
In this situation, do remember to record such data
in a machine learning-friendly way.
After all, everything must be represented numerically
to be sent to the machine learning model,
and you may have to preprocess the data before it's
usable to train such a model.
In this case, the data quality is, again, potentially high
as you still have control over how the data is collected.
However, the cost would increase as you
will need to spend engineering time to do this.
And it might take a long time to gather enough example
data for your needs too.
Now, if the problem you're trying to solve
is something that research communities are also
interested in, there's a good chance
that an open source data set may exist
that was put together by passionate folk for a given
problem.
One popular website, known as Kaggle,
often sets challenges to the machine learning community
whereby they provide data sets for a certain domain
in the hope that people will compete
to make the best performing model in return for a reward
or status on the website.
Many other companies, governments,
and scientific communities also release
public data sets like this too, and a quick search online
will reveal many.
Google even has a data set search engine specifically
for this at datasetsearch.google.com.
It should be noted, however, that in this case,
the data quality can be of variable quality
and some data sets are well-validated and accurate,
but others may contain mistakes or are missing entries
for certain features and will need
to be cleaned up before using.
On this note, do you look out for verified data
sets from data science communities,
like those on DataHub.io, which are
often of a high quality and well-suited for machine
learning tasks.
Therefore, the cost in using such data is low to medium
depending on how clean the data is but the time
to get the data is short.
Finally, you could simply hire a third-party company
to gather the data for you from a whole army of people
dedicated to finding or creating the data.
In this situation, the data quality
should be high, as you have precisely defined
what you need them to collect.
But the costs might also be high as you need
to pay all of these people.
And the time taken will vary depending
on the complexity of the task that you're trying to do.
And many people will need to be trained to do that task too.
Now, the next question you may have
is how much data do you need to gather?
Well, there's no one answer that fits all situations here.
However, generally speaking, the more good quality examples
you have that can account for the variations
and the edge cases you might expect
to see in the real world, the better the chance
the machine learning has to learn from that.
Now, one thing to consider is the granularity of your data.
Imagine you are training a model using sales data.
If you only use data from days of the month of January,
you may not learn about seasonal variations in sales,
like the holiday spikes or the other times of the year.
At the very least, in this case, you
want example data from each day of the year,
so at least 365 examples.
And realistically, you probably want more than a few examples
for each day over multiple years,
bringing you to over 1,000 data points pretty fast.
It's not uncommon to need thousands of examples
to get good results for machine learning systems.
And some systems made by larger companies
can use millions or billions of data points to train on.
In fact, most of your time will be
spent gathering suitable data and cleaning it
so it can be used to train the model well.
And on that note, you must ensure
the data you use has been cleaned
of any errors that could disrupt the training process.
Once you've got thousands of examples,
this can take time if manual checks are needed for things
that cannot be automatically checked.
In this fictional example I've made for house data,
you can see how line 2 contains data from a different country.
Here, the price and size are input
using a different standard, one using British pounds instead
of dollars and square meters instead of square feet.
On the third line, someone used uppercase letters,
but the house types are the same as the first two.
This could cause issues later if you
assume it's a different type just because the case didn't
match.
And on the last line, one field was not filled out at all,
and someone described the apartment
as having 2.5 bedrooms when you're
expecting an integer value.
These issues come up more than you may imagine.
And if you don't fix them prior to training,
your model may not perform well when you try and use it.
After your data is cleaned up and you
can see what you're left with, you
can now set the features you want to use for training.
It's tempting to use as many features as possible,
and sometimes, more features can help separate the data better.
But choosing features wisely can often
mean a smaller, more efficient model that can work just
as well as a model that uses more features.
Here, I selected price and size to see
how well that works first before trying with more features
as I feel size is a good general indicator of house price.
If this yields acceptable performance,
I might not need to consider other features at all.
Finally, let's talk about data bias.
Imagine you were tasked with collecting
sound samples of people speaking English sentences for a voice
recognition model.
You go ahead and ask everyone to contribute samples
and even hire a company to collect more from all
over the country.
What went wrong here?
Well, if you are based in the USA,
you now mainly have samples of US accents.
If someone from the UK or India tried
to use the trained system, it may not
be able to understand their accents.
In this case, you've unwillingly introduced a bias
into your data collection.
You should be extra careful to try and identify any biases
you have before collecting the data or defining the features
and attributes you want to use when training the model.
All right, you now know the essential concepts
around gathering data, so it's time
to go ahead and learn about the building blocks of machine
learning models themselves.
See you there.
[MUSIC PLAYING]