YUFENG GUO: Natural language has many challenges that are unique
and separate it from other data types like images
and structured data.
So it requires a slightly different approach.
Today, we'll explore a foundational piece
of modeling natural language, called "bag of words."
What does it mean?
And how do we use it to process text?
Stay tuned to find out.
[THEME SONG]
Welcome to "AI Adventures," where
we explore the art, science, and tools of machine learning.
My name is Yufeng Guo.
And on this episode, we're going to look
at how to use bag of words to classify natural language.
Natural language is special because it
has structure inherent in the language while at the same time
being very free-form.
There are many ways you can say the same thing.
And you can also say very similar words,
and yet mean very different things.
So in much of machine learning, we
aim to turn our data into matrices or tensors.
This is very natural for images since that's already
their inherent representation.
Structured data often meets a similar fate,
with numbers in a spreadsheet mapping very directly to input
matrix values.
But with natural language, we need
to somehow find a way to turn words into numbers so we can
stick them into those matrices.
There are many ways that we can do this.
And today, we'll focus on an approach called "bag of words."
Let's pretend for a moment that we're learning English
for the first time ever.
And for some reason, the first words
we have chosen to learn in our entire vocabulary
are these 10 shown here--
words like "dataframe" and "graph," "plot," "color,"
and "activation."
And so we want to be able to identify,
given some arbitrary text, whether that topic
is about pandas, keras, or Matplotlib.
How might we do that?
Perhaps if we looked at a sentence, like "how
to plot dataframe bar graph," we would
recognize just the words "plot," "dataframe," and "graph."
The rest of the sentence would look like a foreign language,
just gibberish.
Knowing only those three words in this sentence,
though, we might still be able to get
some sense of what it's about.
And the way you might capture this information in an array
or matrix would be to first make an array that represents
your entire vocabulary.
So in this case, we have an array of just length 10.
We'd set all those values to 0 and turn
on the array indices that correspond
to the words in the sentence by setting them to 1.
Notice that this has nothing to do with the order
the words appear in the input sentence,
but everything to do with the order of the words
in our vocabulary list.
So now we've encoded or translated the English sentence
into an array of numbers based on our somewhat limited
understanding of English.
The words we don't recognize, we'll just ignore.
Notice that this has the effect of scrambling up
the order of the words, like, say, a bag of words.
Of course, we should do the same for our labels.
This is much simpler, since there are only three of them.
In our case, we have some sentences
that have more than one label attached
to them at the same time, however,
since a sentence can talk about multiple topics at once.
In that case, we want to set all the relevant indices to 1,
leaving the rest as 0, just like we did for the words
from our training data.
Now we've turned the inputs as well as
the outputs, which both used to be words,
into arrays of numbers.
And we can let machine learning do what it does best--
map one set of numbers to another set of numbers.
All the heavy lifting is done in the preprocessing
as we transformed or encoded that text
into numerical representations.
Bag of words is a pretty simple approach for doing this task,
though it's worth pointing out that it might surprise you
how well it works in some situations.
How might we build a bag of words modeled with code?
Keras has a convenient preprocessing library
that we can use to handle much of this for us.
Using the Tokenizer class, we can
select the size of the vocabulary
we'd like to utilize.
In our example, we just had 10 words, which is quite small,
but in our code, let's choose something bigger, like 400.
This will then be fit on the entire body of the text
from your training data, selecting out
the most common 400 words.
With the tokenization process complete, building the model
becomes quite straightforward and similar
to working with other structured data.
Since each row is now just an input of 1s and 0s,
using something as simple as a standard, fully connected,
deep neural network can be quite effective.
If you're planning on having multiple label classification,
where more than one label might be true for a single input,
as we do here, be sure that we choose a sigmoid activation
instead of the more common Softmax activation function,
and pair it with binary cross-entropy loss.
So there you have it--
the bag of words model in a nutshell.
Understanding how bag of words works
and its advantages and drawbacks can
help you build your foundation in natural language processing
as you move on to more advanced approaches to encoding text.
For more details and examples, be
sure to check out the expanded blog post I have linked below
in the description.
Thanks for watching this episode of "Cloud AI Adventures."
And if you enjoyed it, please like it
and subscribe to get all the latest episodes right
when they come out.
For now, get started on your natural language processing
journey by checking out the tensor flow
word embedding tutorial I've linked below
in the description.
[MUSIC PLAYING]