[MUSIC PLAYING]
JOSH GORDON: Hey, everyone.
Today I'd like to make a quick video
that I hope will be concrete and useful for you.
It's about the very first machine
learning library I ever tried.
And it's called Weka.
What's great is that Weka comes with a GUI that
makes it easy to visualize your data sets
and train and compare different classifiers.
And this is a really handy tool to have
while you're learning ML.
I'll give you a quick walkthrough of how to use Weka,
from installation all the way to running experiments,
and show you some of what it can do.
I'll demo training models on two different datasets.
First, we'll predict if a patient
has diabetes based on attributes like their glucose levels.
And next, we'll predict if a congressperson
is a Democrat or Republican based on how
they voted on different bills.
I'll also show you how to evaluate
the results of these experiments and how
to do things like feature selection
to discover which attributes are important.
OK, let's dive right in.
The first thing we'll do is download and install Weka.
And what's neat is that it comes as a nicely
packaged application you can run on Mac, Windows, or Linux.
There's also a Java API.
Here, I'll download and install Weka.
And now I'll start it up.
There are different interfaces, and we'll use the Explorer.
There's a lot on this screen, but don't worry about it.
You'll get a feel for how this works in a moment.
The first thing to do is open a dataset.
So we'll hit Open.
And now would be a good time to download one.
You can find a bunch of prepackaged datasets
on this page.
And we'll start with the UCI repository.
It contains about 37 problems.
And when you download it, you'll get a JAR.
Now, you might be familiar with these
if you're a Java developer.
But if not, don't worry.
You can treat them as a ZIP.
Here I'll unzip it.
And now we can see a directory of datasets.
Let's return to Weka and open one of these up.
And we'll start with diabetes.
All right, what do we see here?
Let me walk you through it.
First, let's learn about the dataset.
At the top, you can see there are
768 examples, or instances, and nine attributes, or features.
The best attribute to start with is class, or the label we
want to predict.
And usually in Weka, that's the last attribute in a dataset.
Clicking on that, we can see a histogram.
The blue column on the left shows
the number of people who tested negative for diabetes.
And the red column on the right shows
those who tested positive.
Now let's look at the attributes we'll
use to predict if a patient has the disease.
The descriptions here are pretty short.
But we can open up the dataset in Sublime
or your favorite text editor to learn more about
what they mean, as well as how the dataset was collected.
Now, Weka datasets come in an ARF format.
And this is just a CSV with some metadata included at the top.
Scrolling down a bit, we can see a description
of the attributes.
And the first tells us the number
of times a patient was pregnant, and the second
tells us their plasma glucose.
For diabetes, I imagine one of these
is more predictive than the other.
Let's see if Weka can tell us that, too.
Back in the GUI, let's click on plasma.
And what's cool is you can see a histogram of how
different values correlate to the class we want to predict.
Recall that blue is negative, and red is positive.
And right off the bat, we can see
this is a useful attribute, meaning that if plasma is low,
say below about 100, then it's unlikely the patient has
diabetes.
Most of these values are blue, whereas as the value increases,
it's increasingly likely that a patient has the disease.
Now let's look at pregnancy.
And to me, this doesn't look like a strong correlation.
There may be one here, but it's less obvious.
Now here's where things get really interesting.
I've so much to show you, but I want
to jump in and classify this data.
So let's head over to the Classify tab.
There's a whole bunch of built-in classifiers.
Let's start with the decision tree.
And J48 is one type of tree that does pruning.
We'll hit Start, and bam.
We're done.
We just trained a decision tree on the diabetes dataset.
Let's say we also wanted to train a linear classifier,
like logistic regression.
To do that, we can go into Functions, Logistic,
and hit Start.
And there we go.
This is great, because we can flip
back and forth between the two and compare the results.
There are many types of classifiers in Weka
if you're interested, everything from naive
Bayes to basic neural networks.
Now let's head back to the tree and see the results.
And there's a lot of information on this screen,
so let me walk you through it.
Let's scroll to the top and start there.
First, you can see the trained tree.
As always, you read it from the top down.
It's telling us to start by looking at the value of plasma.
And this happens to be the most predictive attribute
in the dataset, but we'll return to that later.
Scrolling down, we can see the accuracy was about 73%.
But what exactly was the accuracy evaluated on?
Well, here you can see Weka gives you three options.
The first would be to compute the accuracy on the training
set.
And if we do that, of course it will be higher.
It goes up to 84%, because we've tested the tree
on data it's already seen.
Of course, this isn't useful in the real world.
As always in machine learning, our goal
is to generalize from the training data.
Ideally, we want a model that performs well
on data it's never seen before.
So how can we know if it does?
Well, one way to simulate that is to have a separate test set.
Basically, you can divide the diabetes ARF file
into two separate files, one for training and one for testing.
Use the testing file only rarely to see how
well your algorithm performs.
Another thing we can do is use cross-validation.
And this sounds fancy, but all it does
is iteratively divide the dataset into two chunks.
The larger chunk is used for training,
and the smaller one is used for testing.
We train a model, evaluate it, and repeat
this process a number of times, then average the results.
And Weka automates this process for you.
Now, let's look more closely at the evaluation.
You'll see stats like these for any classifier you train.
And importantly, notice that there
are metrics like precision and recall in addition to accuracy.
Why?
Well, although accuracy is the first thing
we think of when evaluating a classifier,
it doesn't always tell us the whole story,
especially in datasets where one class is rare.
For example, let me show you how to write
a 99% accurate classifier that doesn't really
do anything at all.
Imagine you're writing a program to assist a doctor, like we're
doing with this diabetes dataset.
Now imagine that the disease we want to predict is very rare.
Say only one person in 100 is sick.
So how can you train a 99% accurate classifier
without using any ML at all?
Well, it's simple.
It turns out you can write one line of Python.
Def diagnose return healthy.
Because most people are healthy, just
by predicting that everyone is, or the majority class,
we're 99% accurate, but not useful.
Our model will always be wrong when the patient is sick.
That's why when we evaluate classifiers,
we have to look at accuracy both on the positive and negative
cases.
And there are different ways to do this.
A confusion matrix like we see below is one of my favorites.
And you can find a link in the description
to learn more about it.
Now onto another topic.
Imagine we had asked the question, which attributes
in the dataset are important?
Here we don't want to train a model.
We just want to explore the data.
There's a technique we can use called feature selection which
can help.
And the first thing we can do is rank the attributes
by their information gain.
Let's head back to the diabetes dataset for an example.
We can hit Filters, Supervised, Attribute, Attribute Selection,
then select Info Gain as the method and Ranker
as the search.
And when we run this, the attributes
will be sorted by how useful they are to predict the label.
If you could know just one feature from the dataset,
you'd probably want to know plasma.
If you could know two things, you'd probably
also want to know mass.
But keep in mind we haven't done a search.
It's possible that these two attributes are not
the best combination.
There are other methods of selecting
attributes, like this, if you want
to find the best subset to use.
An exhaustive search can be computationally expensive,
though.
Now let's look briefly at the vote data.
I'll move faster this time, because training and evaluating
a classifier uses the same pattern.
This dataset is from the US Congress,
and the goal is to predict if a representative is
a Democrat or a Republican based on how
they voted on different bills.
As before, let's start with class.
Here, blue are Democrats and red are Republicans.
And this dataset was collected back in the 1980s,
so these ratios are different than they are today.
Each attribute describes how a congressperson
voted on different bills.
And many are predictive.
If you flip through them, you'll see that many votes
are divided along party lines.
As before, you can read details about the bills in the ARF
if you're interested.
And if we train a tree, these are the rules
you can use to predict the political affiliation
of a member of Congress based on their voting history.
It's still amazing to me how easy this tool is to use.
And I find it helpful on a regular basis.
I usually begin my experiments with a decision tree
to learn more about the data and as a sanity check
for a baseline classifier before I
move on to more complex models like neural nets.
OK.
I hope this was helpful and that Weka makes it easier
for you to learn ML.
Thanks, everyone, and I'll see you next time.
[MUSIC PLAYING]