LAWRENCE MORONEY: Hi, and welcome back
to "Machine Learning Foundations," where
you can learn the basics of programming machine learning.
I'm Lawrence Moroney, your host and guide
to this wonderful new world of developing software.
Last time, in our Introduction to Natural Language Processing,
we looked at how you can tokenize words
with simple APIs.
This allowed you to turn words into numbers or tokens
so that they can be more easily represented
in a computer's memory.
It's the first step in processing language.
The next step, which we'll look at in this video,
is to turn sentences into sequences of tokens.
And we'll explore the tools that make this very simple
to do in TensorFlow.
So let's get started.
Here's the code that we were looking at last time
where our sentences are represented
as elements in an array.
We use a tokenizer to turn the words in those sentences
into numeric tokens.
Then we can inspect those tokens by looking at the tokenizer's
word index property.
But there's a few differences.
First, I've added another sentence to the corpus,
just so that we can get a variety of sentence lengths.
All of the others were four words long.
So I've added one that has seven words.
Additionally, I've added this call--
tokenizer.texttosequences-- which does the hard work
of turning the array of sentences
into arrays of tokens.
As the tokenizer already fif the job of tokenizing the words,
it's nice that it can handle the sequencing, too.
But this is just the first step in what
you'll need to do when you're preparing text for an LP.
The results of this, if we print out
the word index and the sequences, will look like this.
We have a lot more words now.
And we can see how some of them have been added.
Remember, the order of words is the frequency of them
in the corpus.
So my is the most common word and then love, et cetera.
We can see that the new words that we've
added, such as amazing, think, is, and do are there also.
And the sentences have been encoded into numeric sequences.
So, for example, 4, 2, 1, 3 is our first sentence.
If you substitute these numbers for the associated words,
you'll see I love my dog.
So think about this in machine learning terms for a moment.
All along, we've had data that we've
trained neural networks on.
And then we showed those neural networks
new data with a view to having the network predict
what that data is.
In the case of pictures, for example,
we had images of horses and humans.
And after the network was trained
on lots of images of these, we'd show it a new one.
And then it would tell us if it thought it
was seeing a horse or a human.
With NLP, we have similar approach.
We'll train a network with lots of sentences.
And then those sentences will be labeled.
Later, you'll see a data set of headlines, some of which
are sarcastic and some of which are normal.
And we'll train a network on those.
But think about what happens with a trained neural network
when you show it new data.
It tries to use what it knows, what it's been trained on,
to understand that new data.
In the case of words, the network
is trained on the words in the corpus.
But what happens when you want to show it new data
and have it predict from that?
The new data will need to be encoded with the same tokens
as the training data.
And it will have to be sequenced in the same way
with the same rules.
So let's go back to our code and see what this might look like.
So while we haven't trained a neural network yet,
we have started to prepare imaginary training data.
Using the tokenizer, we were able to get tokens
for the words in our corpus and create sequences out
of our text.
So imagine now that we've used this to train a network.
And we want the network to understand this test text--
I really love my dog and my dog loves my manatee.
We should use the same tokenizer that we
used on the training set because we
want to have the same tokens.
For example, the token used for dog
should be the same when you train the network and then
later on when you're using and testing it.
So if we were to encode these sentences into tokens
and sequences, we'd get this result.
I really love my dog would become
4, 2, 1, 3, which, if you do substitution,
would be recognized as I love my dog.
Not bad.
But my dog loves my manatee would become 1, 3, 1,
which is my dog my.
And it's really lost all meaning.
The network will not be able to understand the sentence
reasonably because words like loves and manatee
just aren't in the corpus.
You might think a simple solution
to this would be to add the test data to the training data
so that the training data will contain words
like loves and manatee.
But that isn't really feasible.
You can't train a network with new words every time
it sees a not previously seen word.
And you'd, of course, end up overfitting,
where the network is only able to parse data
that it's previously seen.
There's no simple solution to this.
But there is a simple thing that you can do to begin solving it.
We'll see that next.
Here, I've updated the code, and I've added a new option
on the tokenizer.
The OOV token parameter allows you
to specify a token type for Out Of Vocabulary.
Choose a string you don't expect to see in the corpus, like I've
done here with OOV, and pass back to the tokenizer.
Now, there'll be a new token list,
with OOV being number one.
It'll always be number one regardless of how many times
it's used.
So the rest of the tokens from number two onwards
are in order of frequency.
And the sequencing of the sentences will use it.
So now our two sentences are encoded as 5, 1, 3, 2, 4 and 2,
4, 1, 2, 1.
If you swap the words back in, you'll get I OOV love my dog
and my dog OOV my OOV, which is a small step
in the right direction.
One important thing when training neural networks
is to get the input shape of your data uniform.
With images, you saw that we resized all the images
to be the same size so that we could have a network fed
something of that size to produce a prediction.
With language, we generally have to do the same thing.
There are exceptions with something
called ragged tensors.
But I'm not going to be covering those here.
Given that our sentences are different lengths,
we can get them to uniform lengths
using something called padding.
We'll see that next.
Here's the updated code to handle padding.
Let's step through it piece by piece,
and we'll explore what's new.
First, we'll import the pad sequences APIs
from tf.keras like this.
Now, we can pass our sequences to pad sequences
to get back a padded list like this.
If we print them out, we'll see the sequences
have been padded for us so they're now
all the same length.
There are also no longer lists of comma-separated values,
but real tensors in an array of tensors.
So 5, 3, 2, 4 become 0, 0, 0, 0, 5, 3, 2,
4 with a bunch of zeros at the beginning.
And our longest sentence, 8, 6, 9, 2, 4, 10, 11
gets into it without any padding of zeros.
You saw the first sentence was padded
with zeros at the beginning.
And that's the default behavior.
If you want them at the end instead,
you can just say padding equals post.
The other default behavior was that each padded sequence
was the length of the longest sequence.
So the longest sequence had no padding at all.
You can change that to another length using the maxlen
parameter.
And here, I've set it to five.
You might wonder what would happen
to all of the characters in sentences longer than five.
Well, they're going to be truncated.
And with this parameter truncating,
you can specify if it's cut off at the end
of the sentence using post or at the beginning using pre.
You've now seen the steps involved,
not just in tokenizing all of the words in your sentence,
but also in sequencing them into sentence arrays
and using padding to get the arrays
into the same shape and size.
Now, let's take a look at how all of this will work in code.
OK, so here's the code for this code lab.
You can try to put yourself in a few moments.
What we're going to do is we're going to import the tokenizer.
But we're going to extend it now to do sequences.
So we're importing pad sequences.
Here's our set of sentences.
And we'll use these with the tokenizer
to fit on texts with these sentences
so that we can get our word index.
The sequences, we'll then call text
to sequences with sentences.
And they'll be turned from words into sequences
of numbers where the numbers are the tokens
for the particular word.
We'll print them out so we can see what they look like.
But there are also words that the tokenizer wasn't fit to,
like manatee, for example.
So when we look at I really love my dog
and my dog loves my manatee, we can also
use the tokenizer to turn these into sequences
to see what those sequences would look like.
And here it is.
So first, I'll see my word index.
And my word index is generated from these sentences.
After that, I'll see the sequences for these sentences.
So 5, 3, 2, 4 is 5 is I. Three is love.
Two is my, and 4 is dog.
So I love my dog gets turned into 5, 3, 2, 4.
Next, we can look at the padded sequences.
And we can see that they've been sequenced into five tokens.
So when we look at the fit on text,
we did say max length equals 5.
And that's why.
So these three sentences of four tokens each
get padded with a zero.
This sentence of seven tokens gets truncated to just five.
So now if we look at our test sequence,
our test sequences become 5, 1, 3, 2, 4.
And that's because we're using an out of vocabulary token
as 1.
So 5, 1, 3, 2, 4 becomes I, out of vocabulary,
which was the word really.
Three is love.
Two is my, and then dog.
And when it comes to my dog loves my manatee,
well, we have 2, which is my, 4, which
is dog, out of vocabulary, my again, out of vocabulary.
So it's my dog out of vocabulary,
my out of vocabulary-- that type of thing.
And then, of course, we can pad our test sequences.
And in this case, we said the max length was 10.
And as a result, we get these test sequences,
which are much longer.
Give it a try for yourself.
Experiment with the words, and see the kind of results
that you get.
All of that code is at this URL when you can try it out
for yourself.
Pause the video, give it a try, and come back when you're done.
Up to now, we've just been using hard-coded sentences
to experiment with tokenizing and padding.
Before you can train a neural network,
you're going to need to read in text from a data source.
In the next video, you'll see how
to do that, reading thousands of news headlines,
tokenizing and sequencing them.
After that, you'll be ready to start
building your first models that understand language.
So stay tuned and don't forget to subscribe.
Thank you.
[MUSIC PLAYING]