[MUSIC PLAYING]
JASON MAYES: In this section, you'll
learn how to load models from a site known as TF Hub, which
many folk from the wider TensorFlow community
have deployed their pre-trained models to.
For this exercise, you'll be working
with a model that expects an image as an input instead
of a simple number, so let's get started.
Now, first, what exactly is TensorFlow Hub?
Well, it should be noted that the ML community like
to create model gardens, which is basically
a website where people upload their raw, saved models.
One example from the TensorFlow ecosystem is TensorFlow Hub--
that's shown here--
from which you can then download the raw models, as you need.
However, it should be noted that the documentation for these
might be minimal, and the work could
be of varying quality or production readiness.
And the developer may not have spent time
writing an easy-to-use interface to the model itself.
In this case, just like before, it's
typically left to you to write the code that
can transform your data into a form that
can be input to the model, which is known as pre-processing,
and, of course, to write the logic to take the output
numbers from the model and do something useful with it,
known as post-processing.
Let's head over to TF Hub to select
a model for this exercise.
First, open a new browser window and head on over to tfhub.dev,
as shown.
You should see a landing page that looks
something similar to this.
The first thing you want to do is filter the model format
to only show models that are compatible with TensorFlow.js
using the Filters menu on the left, as highlighted.
Great.
You now see a whole bunch of models
of potentially very scary-looking names
appear on the right, as shown.
Remember, most of these names are just
explaining the architecture of the model
to help ML folk choose the right one for their needs.
Next, expand the Problem domain filter
along the top of the filter section
so we can refine this huge list further.
You can now see a whole bunch of potentially useful model types
that you might want to use.
Scroll down and select Pose detection for this exercise.
Once you click out of the filters area,
the result should update, as you see here on the right.
All of these models have some connection to pose detection.
Find the model called movenet/singlepose/lightning,
which is a super fast model for pose detection
that was created by a team at Google Research,
and click on it to be taken to its documentation page.
Now, this page has a whole bunch of useful information
in order to use the model.
The first thing you need to do is
switch to the documentation for JavaScript.
By default, the selected tab is for Python users,
which is not what you want.
In the image above, you need to set the second tab
for the JavaScript documentation, as shown.
Great.
Now, as machine learning engineers,
there is three key things we're looking for in order
to use the model.
The first is what inputs does the model expect?
The second, what outputs does it produce?
And the third, is there any example usage code
that we can use to get started faster?
Now, if you scroll down the page,
you'll find text around each one of these areas, so let's dive
deeper.
Now, first, we see a section explaining the inputs.
From this, you can quickly determine
that the model expects a tensor of shape 192
by 192 by 3, which essentially represents an image of 192
by 192 pixels who have red, green, and blue channels-- very
similar to what you saw in the prior sections
when learning about tensors that could store image data,
just with a different pixel size.
Now, more importantly, the numbers
contained in this tensor should be of type int32,
and all values should be in the range of 0 to 255.
It should also be noted that other models may
expect different values as inputs,
so always check for documentation.
Now, moving on to the outputs, you
can see it produces a tensor of shape 1, 1, 17, 3,
and the type of those outputs will be a 32-bit floating point
number, meaning there'll be some number with a decimal point
in them and not a whole number, which
would be called an integer.
Now, this is nice to know, but on its own
doesn't help us understand what these numbers represent.
Reading further, you can discover
what's actually contained in this tensor 1D output object.
The final axis of the output tensor has a length of 3.
This means, after all is said and done,
there are three numbers you can access
for each item in the array of outputs.
The first two of these numbers represent the y-,
x-coordinates of the found pose points in the input image.
And note the order here.
Instead of x, y image coordinates,
the developers of this model for some reason chose y, x.
So the first number would be the y value and the second number
the x value.
Furthermore, these values are normalized between 0 and 1.
And let's take a moment to understand normalization.
Let's say you've got a bunch of numbers
stored an array that can take the range from minus 100
to 100.
In another array, you've got the numbers
that range from 0 to 1,000.
If you wanted to compare a value from the first array
with a value from the second array,
it would be kind of tricky, as the number 50,
for example, in each case would be
at very different ends of the scale
for what it represents, as shown on the slide.
In the first array, 50 is quite a high value,
and in the second array, 50 is overall
a low value for whatever was being recorded.
It's sometimes more convenient to convert the numbers to be
in a form that's easy to compare,
so they're always in the same range of values.
This is essentially known as normalization.
It's quite common in machine learning to normalize
numbers between 0 and 1.
In order to do this, you first find the difference
between the maximum number that can occur
and the minimum number that can occur for each set of numbers.
In the first example here, you can see the range is 200,
and in the second the range is 1,000.
Next, for each value in each set of numbers,
you'd subtract the minimum value that
exists in that particular set.
Taking the example number of 50 like before,
in the first example, 50 minus negative 100 is 150.
And for a value of 50 in the second example, then 50 minus 0
is still 50.
Now, the final step is to divide the number you just
calculated by the range that you previously found.
So again, for the first example, 150 divided by 200 is 0.75.
And in the second example, 50 divided by 1,000 is 0.05.
Now, when you compare 0.75 and 0.05,
it's much clearer to see that 0.05 is much lower overall.
Now, returning to our documentation,
you can now see that the output numbers for y and x
are normalized to the input image size.
What this means is that instead of giving you an image pixel
value as an output for a single pose point that was found,
you'll get a number between 0 and 1.
So to convert that back to real pixel values
on the original input image tensor that
had a size of 192 by 192 pixels in y and x,
you can simply multiply these normalized numbers
by 192 to find its true pixel position in that input image.
So if the return values were 0.5 and 0.25,
that would be y, x position of 9648 in the original image
for that found point.
Now, what's particularly useful about
this normalized representation is
that often your original image may
be a different size altogether.
Even though the input to the model was 192 by 192 pixels,
if the original image was of the same aspect ratio--
so in this case, also a square, maybe 1,024 by 1,024 pixels--
as you have the normalized coordinates being returned,
you can simply multiply the normalized y value
by 1,024 and the x value also by 1,024, in this case,
to find the true position in that larger image, too.
And that's very useful and convenient when
you want to draw something on the original image
that you had in the correct position.
Now, do take note, however, if the original image
was a different aspect ratio, then
you would need to scale values accordingly in each dimension
to account for that.
Now, moving on through the documentation,
you can see that the third point in the numbers returned
is the confidence of the pose point
that it thinks it's found.
So here, a value of 0.9 would represent 90% confidence
that the point was correctly labeled.
You can also see in the output section
that row 17 of these key points got returned each time.
Each key point represents a different part of the body,
and the order of the points in the array
is shown in the documentation.
For example, the first point in the array of 17
will be the nose, and the second point
will be the left eye, and so on.
Heading onto usage, the developers
have kindly provided a TensorFlow.js example
of how to load the model.
As this MoveNet model is highly optimized,
it's in the form of a graph model instead of the layers
model you previously saw.
In this case, you use the tf.loadGraphModel function
that you saw in the previous section to load it.
The only difference when loading directly from TensorFlow Hub
is that there's a second parameter that you pass
of an object whose property from TF Hub is set to true.
This is a special use case just for when
you're loading the model directly
from the TF Hub hosting only.
If you had downloaded the model files and hosted them
on your own website, you would load those files,
as you learned in the prior section,
without needing to use this second parameter.
Now, one thing to point out at this stage
is that sometimes the documentation is stale
or out of date after model creators have pushed an update
to the model on TF Hub.
For this reason, when specifying the model
URL to load from TF Hub, always copy for current live version,
which can be found at the top of the documentation
and clicking the Copy URL button, as shown.
If you prefer to download the model,
you can do that, too, using the button right below it.
OK.
You now know everything you need to start using
this model in a basic manner.
Now, there are many other things you
would need to read up from their GitHub documentation
to use this model as efficiently and as fast as possible,
but that's outside the scope of this tutorial.
In my next section, it's time to head over to Glitch
to try this for yourself and follow along.
See you there.
[MUSIC PLAYING]