[UPBEAT MUSIC]
JASON MAYES: In this chapter, you'll
be discovering how to create your very own website that's
capable of detecting comment spam
live in the web browser using a pre-made TensorFlow GS
machine-learning model.
But before you can use that, it's
time to learn the basics of how to interact
with natural language models.
Specifically, you'll be using a model
that uses the average word embedding implementation.
But what exactly is that, and why will you
use it over the many other models that are also available?
Let's dive into this deeper to learn more.
So there are many pre-made models available natively
in TensorFlow GS that are easy to use
in just a few lines of code as you've seen.
But today, you'll be considering models
that are available from TensorFlow's Model Maker
to learn how to take exported safe models from this system
and use them in TensorFlow GS, writing all the code
to interface with the model directly
ourselves at the Tensor level.
Let's take a look at the models available for text
classification.
Looking at the Model Maker documentation,
you can find three text-based models.
The key thing to note here is that all
of the model architectures shown essentially do the same job.
It's just that some are more accurate or versatile
than others.
In fact, all of these can be used
for comments spam detection.
As web engineers, size is a big concern to us.
For that reason, you can disregard the BERT base model,
as it's simply too large for us to deploy to the client side.
Though, it should be noted if you're
running in Node.js on the server side,
then this could be a very suitable candidate
for that environment.
So that leaves us with two options.
One is under 1 megabyte in size, which
would be very acceptable for both mobile and desktop.
And the other is 25 to 100 megabytes in size,
depending on this thing called quantization.
Now, if you go back to your high school days,
you may remember that computers have different ways
to represent numbers depending on the range of numbers
that you want to store, as shown in this table.
Clearly, the more memory you use to represent
a number, the larger range of numbers
you can potentially store, but it comes with a memory cost.
When building and training a model,
it's very common that the 32-bit float type
is used to store the model weights that allow it
to classify the data correctly.
However, it turns out that even if you reduce the precision
to an 8-bit integer for the weights,
the model does not lose too much accuracy.
Quantization simply means reducing the precision
of a number stored in the model to make the resulting file
size of the model smaller and therefore, download faster,
which is great for the web world.
In the chart, you can see how an unsigned 32-bit floating point
number is transformed to a single 8-bit integer.
One thing to note here is that very often only a small part
of the floating point space is actually used,
as shown in the diagram at the bottom.
So you can simply find the bounds of that range
and transform it to the full 8-bit space
to retain as much detail as possible.
Now, the number that previously took up 32 bits of memory
takes up just 8 bits of memory, which uses
four times less storage space.
Now, for the web page you're creating today,
you want it to run as fast and as many devices as possible.
So the 1 megabyte average word embedding model
is the one that you shall use.
You can always switch to the larger, quantized model later
if you wanted to.
Now that you've selected a model to use,
the next step is to figure out how to pass data into it.
As you know, machine-learning models essentially work
by processing numbers--
numbers in, numbers out, and typically, floating
point numbers, to be precise.
But if you want to learn information
about words, which are a string of characters,
how can these words be represented as numbers
such that words that are alike for a given measure
have a similar numerical representation?
Let's think about a few ways to do
this to learn how our premade model is
working behind the scenes.
OK, so let's start with the word "car."
You can simply assign it a number
based on its literal order in the English dictionary.
However, whilst this will show about the word's characters are
similar to another word-- for example, cat,
which would be numerically close--
that's not useful here as you want the meaning to be similar,
not the character order.
So how can you encode words to give a numerical way
to separate them for a given classification problem?
Well, another option would be to have
our own dictionary of, say, 1,000 words that you
want to be able to compare.
And then give them each a score for a dimension of interest.
For example, if you had a measure
of how medical a word was, "doctor"
would score very high--
maybe 0.95-- versus something like "pilot,"
which would score lower on this particular dimension--
maybe minus 0.4--
using the scale on this slide.
Now, one dimension is probably not
enough to be useful for classification.
So you may need to have several dimensions to allow
you to organize the words in a way that
might be more suitable to do what you need to do.
As a simple illustration of this,
let's say you discovered two meaningful dimensions that
measured how medical a word was and also how
biological the word was too.
You could now draw a 2D graph to represent this
and any word would lie somewhere on this chart.
So for each word in your custom dictionary,
you'd have a two-dimensional vector
to indicate where it lies on each of these dimensions, which
you can then use to represent words to train your model with.
In the case of the word "tumor," for example,
it may have a vector of 3, 3, which
you can plot on a 2D graph as shown to see
where the point would lie.
Likewise, the word "X-ray," whilst also quite often related
to medical things, is not biological.
So it may get a score of 2, minus 3.
And here are some other word vectors for comparison.
You can think of a vector as an array of numbers
where the array length indicates how many dimensions you're
working with, and the value of a number
indicates where it is on the scale of that dimension.
It turns out that computers can work
with any number of dimensions, like 200 for example.
Whilst this is hard for humans to visualize,
as you live in a 3D world, the principle
is the same as the 2D visualization on this slide.
In fact, when training a machine-learning model
like this, you can specify how many dimensions
you want to try and utilize for separating
the words in your dictionary.
And it will then go ahead and find the most meaningful ways
to separate the training data, leading
to the best classification results for the task desired.
And it's actually a rule of thumb determined from research
that the fourth root of the number of words that you have
here will work pretty well.
So if I'm using, say, 2,000 words in my dictionary,
a good starting point for this would be seven dimensions
to give us enough ways to separate
the words in a meaningful way for future classification.
So if you have a sentence like "free software"
and compared the resulting word vectors
with a new sentence like "cheap download,"
here, "free" and "cheap" have a similar vector encoding.
And "software" may have a similar encoding to "download."
If you add the vectors for each sentence,
the overall vector representing each sentence
ends up being roughly in the same location if you plotted it
on a seven dimensional graph as shown by my totals
on the right.
Now you've got a way to represent similar sentences
with similar vectors.
And you can use those vector representations of sentences
to train a custom machine-learning model to then
classify a sentence to be spam or not spam.
And here's the end result of visualizing the pre-made spam
classifier that you're going to be using today.
It also uses seven dimensions to function.
But here, those seven dimensions have
been mapped to a 3D visualization
to help explore the words' relations when
interpreted for spam.
You can see from this visualization
how clusters of words that have high correlations of spam
versus not spam end up living in different physical spaces
as shown.
And with that, it's time to get coding yourself
and put this all into action.
Head on to the next section to create
a website that can classify spam comments entirely
in the web browser.
See you there.
[UPBEAT MUSIC]