LAURENCE MORONEY: Hi, and welcome back
to Machine Learning Foundations, where
you can learn the basics of machine learning
with computer vision, natural language processing, and more.
I'm Laurence Moroney from the Google AI team,
and I'm here to be your guide.
Over the last few episodes, we haven't
done much machine learning.
Instead, we looked at how you can preprocess
text data to get it ready for training machine
learning models.
In this episode, you're going to put that knowledge
to use in training a text classifier, a model, which,
when given a piece of text, will understand
the contents of that text.
You'll be working with the Sarcasm
in News Headlines data set by Rishabh Misra, which is
available on his website here.
This is a really fun data set, which collects news headlines
from normal news sources, as well as some more comedic ones
from spoof news sites.
The data set is a JSON file with three columns.
The is_sarcastic one is 1 if the record is sarcastic.
Otherwise, it's 0.
The headline is the headline of the article,
and the article link is a URL to the text of the article.
We're just going to deal with the headlines here.
So we have a super easy data set to work with.
The headline is our feature, and the is_sarcastic is our label.
The data in JSON looks a bit like this.
Each entry is a JSON field with the name-value pairs showing
the column and associated data.
To make it easier to use in Python,
I've edited the data just a little.
By wrapping the whole thing in square brackets
and putting a comma between each line,
I can then write some super simple code to load it in.
And then we can start tokenizing and sequencing.
And here's the code to load it in Python
when it's structured like that.
I'll go through this piece by piece.
First, we'll import json so that we can
use the json parsers in Python.
Then we'll open the sarcasm.json file.
I've hosted a version with my edits that you can use.
You'll see the URL for that later in the notebooks.
Using json.load(), we can load and parse the entire thing.
I'll initialize arrays for the sentences, labels, and URLs.
And I can now simply iterate through the datastore.
And, for each item, I can append its headline, the sarcasm
label, and URL to the appropriate array.
And that's it for loading the data.
In previous videos, you may recall
that we had hard-coded sentences into an array of strings.
We now have exactly the same data structure for the headline
sentences, despite that there are now over 25,000 of them.
So, for the next code, despite us using this real data set,
it will look very familiar.
So let's dive in.
So here's the code to tokenize and sequence
the sarcasm data set.
We create a tokenizer and fit it on the sentences.
In this case, the sentences are the large array of 25,000-plus
sentences that we read from the sarcasm data set.
We can use the tokenizer to show us
the word_index so we can see what words it
learned from the data set.
And here's an example of some of the words.
Things like skillingsbolle, whatever that is,
was tokenized at 23,055.
Remember from earlier that the words with the lower number
tokens are the ones that are more common,
and the ones with the higher numbers,
like our friend skillingsbolle, was less commonly
used in the data set.
So, of all of the words here, hurting
is the one that was found most often.
We can now turn all of our sentences
into sequences where, instead of words,
we have the tokens representing those words.
We'll pad them post, which means that all of the sentences
will be the length of whatever the longest one is.
And anything shorter than that will
be padded with zeros at the end of the sentences
in order to keep them all the same length.
If we want to inspect them, we can then print out one of them,
and we can print out the shape of the entire padded data
structure.
You'll see output like this.
This is the first sentence in our corpus
after tokenizing and padding.
It's a shorter sentence.
So it ends with a bunch of zeros.
And this is the shape of the data structure for the padding.
This tells us that we have 26,709 padded sentences,
and each of these is 40 values long.
With just a few lines of code, you've
loaded the data from sarcasm into sentence arrays,
tokenized, and padded them.
Before we can get feeding them into an ML model,
we'll take a step through a code lab
where you can see all of this code running.
After that, you'll be able to try it for yourself.
So here's the code.
The first thing that we're going to do
is just download the sarcasm.json data set.
And, once we have that, we're going
to start slicing and dicing it.
We'll see that we import json, and then sarcasm gets
downloaded to /tmp.
And we'll set datastore equals json.load() this file.
Then we have a arrays for sentences, labels, and URLs.
And, for each item in this datastore,
we'll just take the the headline will be a sentence.
The is_sarcastic will be the label.
And the article link will be the URL.
So this, because it's in json, is divided line by line
into these three things, and we'll
be able to see those within the datastore.
Then, finally, we can start importing things
like the tokenizer and pad_sequences.
We can set the tokenizer to have the out-of-vocabulary token
OOV.
And then we're just going to fit it
on text sentences for the sentences with the headlines
that we loaded.
We can do the word_index to be the tokenizer's word_index.
And we can print out the length of that
and print out what that looks like.
And we'll show that in just a moment.
The sequences are just going to be
tokenizer's .texts_to_sequences method with the sentences
passed to it.
And then we'll pad those sequences using
padding='post', and we can print out the first couple.
So, for example, our first print here
was the length of the word_index.
And we can see that it's 29,657.
That's how many unique, independent words there
are in this word_index.
The word_index itself we print out.
And we can see OOV is the first token because we
specified it that way.
And then the words are in frequency order.
So, as you can imagine, things like to, of,
the, in, for are the most common words that will be printed.
We then used the texts_to_sequences() to pad
these and to create the pad_sequences().
So we now have sequences that are
padded to the length of the longest one.
And here is an example of our first one
that is actually padded.
And you can see there's 40 characters in it,
and the shape of our padded is 26,709 comma 40.
That means we have 26,709 headlines,
and the maximum length of each one is 40.
The first one we can see has been encoded like this.
And because our padding is post, then it gets padded with zeros
after the text.
So that's a quick look at how you
can do this type of text preprocessing.
Next, you will see the URLs.
So you can try this out for yourself.
Well, that was pretty straightforward, I hope.
And here's the URL for that code lab.
So pause the video, and give it a try for yourself.
After that, when you come back, we'll
take a look at the next step and what you've really
been waiting for, building a model that understands
this text enough to classify future sentences as
sarcastic or not.
First, in the code, you'll see a number
of commonly used variables.
Each of these will be used throughout the code.
You've seen many of them so far, but others,
like the embedding dimension, will be clear later.
The training_size of 20,000 will be used next.
We have a corpus of many thousands
of sentences and labels.
And, moments ago, we specified 20,000 as the training_size.
So that many sentences and labels
will be the training set.
And we'll hold back the other 6,000 or so
as a validation set.
So our training_sentences will be the complete corpus from 0
to the training_size.
And our testing_sentences will be from the training_size
to the end of the set.
We can do similar with the labels.
The training will be the first batch,
and the testing will be the last ones.
As we've split the data into training and testing sets,
we should do the same for the padded sets,
instead of having that one large master one
that we had earlier on.
First, we'll create a tokenizer, and we'll
specify the number of words that we want
and what the out-of-vocabulary token should be.
We'll fit the tokenizer to just the training_sentence corpus.
This will help us accurately reflect any real-world usage.
Our testing_sentences can be tested
against the vocab that was learned from the training set.
Now we can create a set of training_sequences
from just the training_sentences.
And we can pad these to get a set of padded training
sentences.
And then we can just do the same thing for the testing_sentences
and for all the labels.
So before we can train a model with this,
let's take a look at the concept of embeddings, which
help us turn the sentiment of a word
into a number in much the same way
as we tokenized words earlier.
In this case, an embedding is a vector pointing in a direction,
and we can use those directions to establish meanings in words.
I know this is all very vague.
So let me explain it visually.
For example, consider the words bad and good.
Now we know they have opposite meanings.
So we could draw them as arrows pointing
in opposite directions.
We could then describe the word meh as being sort of bad,
but not really that bad.
So it might be an arrow like this.
And then the phrase not bad, it's not as strong as good,
but it's more or less in the same direction as good.
So we could draw it with an arrow like this.
If we then plot these on a chart,
we could then get coordinates for these arrows.
These coordinates could then be seen
as embeddings for the sentiment of those words.
There's no absolute meaning, but, relative to each other,
we can establish sentiment.
To do this in code, we can simply
use a Keras layer called an Embedding.
Our Embedding should be defined as a vector for every word.
So we're going to take vocab-sized words
and then specify how many dimensions we want the arrow
direction to use.
In this case, it's 16 that we created earlier.
So the Embedding layer will learn 10,016 dimension vectors
where the direction of the vector
establishes the sentiment of the word.
By matching the words to the labels,
it'll have a direction that it can then start learning from.
Once we've defined the model, we can then train it like this.
We simply specify the training_padded features
and labels, as well as the validation ones.
Now that you've seen the full thing,
let's take a look at a screencast of the sarcasm
model being trained and tested.
After that, I'll give you the URL
so you can try it for yourself.
OK, let's start looking at the code.
First of all, I just want to make sure TensorFlow 2.0 is
being used.
Then I'm going to do the imports.
We're importing json.
We're importing tensorflow and, of course, Tokenizer
and pad_sequences so we can preprocess the text.
Next, we'll just set up some of the variables
that we're going to use like the embedding dimensions
and the vocab_size, et cetera.
And next up is here I've stored a version of the sarcasm
data set where I've just made it a little bit more
Python-friendly, as we saw in the slides.
I've downloaded that, and it's now ready to go.
So I'm going to open that into datastore
and then iterate through datastore
and add the headlines to the sentences
and add is_sarcastic to the labels.
This can then be split into training_sentences,
testing_sentences, training_labels,
and testing_labels using the training_size variable
that we created earlier on.
And that was 20,000.
So our training set is going to be 20,000,
and the remainder will be used for testing.
I'll then use the tokenizer to fit it
on the texts for the training_sentences only
and then get the word_index out of that.
Our sequences and our padding can then
be created off of that tokenizer for both
the training and testing sets.
So note that the testing sets are
going to have a lot of out-of-vocabulary tokens
in them because they'll have words that aren't
in the training data set.
And that's OK because that's the kind of thing
that we want to use to emulate real-world scenario.
I'm just here going to convert them into numpy arrays,
and now I'll define the model.
And we can say our model is an Embedding, followed
by a GlobalAveragePooling, followed by a Dense layer,
which has 24 neurons in it, followed by a Dense layer
for the output with a single neuron.
And this is a binary classifier.
So we have one neuron that's activated by sigmoid.
I can summarize this, as we can see here.
And now I'm just going to train it for 30 epochs.
Now one of the things you'll see when it's training
is that models like this tend to overfit a lot.
So we'll notice, as you're looking at the training,
that the accuracy will start climbing very high.
And it will go to, like, 99% maybe in about 10 epochs.
The validation accuracy, as you see, is also doing very well,
but it's going to top out at something like at about
85% percent, but take a look at the validation loss
as you're training it.
It's initially going down, but then it will eventually
turn around and go up.
And, by the time we've done 30 epochs,
our loss on the training set it going to be tiny.
Our loss on the validation set is
going to be relatively large.
And that's indicating overfitting.
There's a number of techniques you can use here
to avoid overfitting.
And one of the best ones to use is actually the vocabulary.
If your vocabulary is much too large,
then you're going to be overfitting to relatively
few words in the vocabulary because, if you think about it,
the words aren't going to be distributed evenly.
The most common words-- you'll have a hockey stick curve.
The most common words will be used a lot.
The least common words will be use very seldom.
And, if you're training set is training
on many of the least commonly used words,
you could end up in that overfitting scenario.
So it's good for you to experiment with your vocabulary
size in order to avoid overfitting.
But just to keep it simple, I'm training a pretty naive model
like this one.
And we can see now that we're greatly beginning to overfit.
Our loss on the training set is 0.03.
Our validation loss is 0.95.
So even though the validation accuracy looks pretty good,
it is leading us somewhat into a false sense of security.
So at the time we're done, the accuracy
is 99.3% on the training set.
It's 81% on the validation set, but keep an eye on the loss.
The loss on the validation set is enormous,
where the loss on the training set
is very small and a clear indication of overfitting.
But let's start taking a look now
at some of the data in there.
So, if we take a look at training_padded number 2,
we can see that the words in it have been tokenized.
And it's been padded with a ton of zeros
because it's much shorter than the maximum length sentence.
And that's another way to avoid overfitting
is to take a look at the lengths of your sentences
and try and pick something optimal so you don't end up
with situations where you've got tons of zeros like this.
But I'm just using the default behavior.
If we want to see what the original sentence was,
we can do it by just looking at the training_sentences,
and we can see it's this one.
And then we can look at the label for that.
And it looks like this is a sarcastic sentence
because it looks like a sarcastic sentence.
So let's test with a couple of sentences ourselves.
So I have a couple of sentences--
"granny starting to fear spiders in the garden might be real,"
and "game of thrones season finale showing this sunday
night."
And we can see that the first sentence
is 9.7 times 10 to the minus 1.
So it's 9.7.
So it's a very--
sorry, it's 0.97.
So there's a very high indication
that this is sarcastic, which is true,
whereas this one is 5.48 times 10 to the minus 5, a very
small number.
So it indicates that this probably is not sarcastic.
And maybe this is dated a little bit.
Maybe people will think that it is sarcastic now,
but at the time this was created, obviously, it wasn't.
That's just a little joke.
So, if we now go--
a little thing that we can do is to visualize
the words and the embeddings that have been learned.
And I've added this code to allow us to do that.
So what I want to do here is create these things called
vectors.tsv and meta.tsv.
And these are just the words with the embeddings that
were learned from them and the metadata around that.
And then I can download these vectors.tsv and meta.tsv
from Colab.
Once I have them, then I can go to the Embedding Projector
at projector.tensorflow.org and actually load them in.
So if I load--
and you see it's asked for a TSV file of vectors
and a TSV file of metadata.
Make sure you choose the right ones.
So vecs.tsv is the vectors.
meta.tsv is the metadata.
And once they've loaded and I've clicked outside,
we can see now a visualization of the words.
Now, one of the neat things about this
is if we spherize it, because this is a binary classifier,
we can actually see the words clustered according
to the classification that they were driving
or the classification that they were labeled with.
So a lot of these words here are for our sarcastic
and non-sarcastic.
And these ones are the same.
It's up to you.
By looking through some of the words,
maybe you can find out which ones they are.
Also, when you can click on the word,
you can actually see the nearest words in that vector space.
So there are words that were established
have a similar meaning in the context of sarcastic or not
sarcastic.
It's a really neat visualization to show how
it has learned the embeddings.
And then, based on the embeddings,
if it finds a lot of words on this side,
then it's most likely that those are classified sarcastic.
And if it finds one on this side,
then they're not, that type of thing.
So have fun playing with it.
Take a little look around, and see what you can discover.
Great.
Now you can try it for yourself.
Here's the URL.
Pause the video.
Take a look at training the model.
And then come back when you're done.
Hopefully that was an interesting exploration for you
into the beginnings of NLP with TensorFlow.
And that brings us to the end of this 10-part series
on foundations of machine learning.
I hope you've enjoyed these videos,
and I hope you've been able to learn from them.
And if you have any questions, please leave them
in the comments below.
And if you want more videos like this, just let us know.
Thank you.
[MUSIC PLAYING]