[MUSIC PLAYING]
JASON MAYES: OK, it's time to start coding
to use this raw MoveNet model.
So let's get right to it.
As before, head back to the TensorFlow.js Glitch page
and find the TensorFlow.js-boilerplate
project.
Remix a new copy of this project that you'll
use to write your future code.
In the first example, you're going
to be starting by editing script.js.
So go ahead and open that file on Glitch ready to use.
I'll wait for you to do that.
And then click Play when you're ready.
So this is the minimal code you would
need to load the MoveNet model and send a tensor through it
to get a prediction.
Let's walk through it.
And feel free to follow along on Glitch as you go through.
Now, just like in the prior section of the course,
you start by defining the location
of the model you wish to use.
In this instance, you use the URL copied from TF Hub.
You can also define a variable called
MoveNet, which you'll use to hold the loaded model once it's
ready.
Again, as TensorFlow.js uses a lot of asynchronous calls,
it's wise to create an async function to do all the work.
Here, you create a function named loadAndRunModel, and call
it right away at the end of the code on the last line.
Diving into the function, you can
call await tf.loadGraphModel and pass to it
the model path of the model desired,
along with the special TF Hub configuration object.
This lets TensorFlow.js know that you're
trying to load a model from TF Hub and only
needs to be specified when using a model
hosted on the TF Hub platform itself.
On the next line, you can use a neat trick
to create an example tensor filled
with dummy values of a shape and data type required.
And we can do this before hooking up the real images that
will need processing.
And this can allow you to quickly check everything
is working correctly.
Here, you can call tf.zeroes, which
will create a tensor just filled with zeros.
You specify the shape, which is the same shape
as defined in the MoveNet documentation
for what it expects as an input, which was 1, 192, 192, 3.
Finally, you can also specify the dtype,
which MoveNet's documentation specified should be int32.
This essentially represents a pure black image that
is 192 x 192 pixels in size.
Next, you can call the predict method of the model
as you're used to and pass it the input tensor
you wish the model to process.
Once a prediction comes back, you
can get the raw array data that was returned
in the correct output shape by calling the array
function on the output tensor that was created.
Note the act of fetching data from the tensor's memory
itself is an asynchronous call, so you
must use the await keyword here as shown.
Finally, you can console.log the array
to inspect what came back from the prediction.
And if you run the code in its current form,
you'll see the following log to the output.
Here you can see there's an array containing
an array, which itself contains an array of 17 elements, which
contains an array of three elements, which, as you know,
represent the y, x, and confidence values respectively.
The other thing to note at this point
is that you just sent the model a completely black image,
and it still somehow predicted 17 key points of where
it thinks it sees a human pose.
It's important to note that the machine learning model will
always give you an output, even if the input doesn't
contain the thing you want.
And in this case, if you look at the confidence scores
for all of the key points, you can see
that they're very, very low.
Here, all of them are under a score of 0.07,
which represents a confidence of 7% or less,
as the maximum value It could be is 1.
It's up to you, as a developer, to decide what is too low
and to ignore such results if they don't seem appropriate.
OK, so you've got results coming back,
and you know from the documentation
how to interpret them.
But so far, all you've done is sent a bunch of zeros
through the model.
How can you send a real image?
Well, as always, before you look elsewhere,
it's wise to check the TensorFlow.js API,
as there's likely a function that can help you.
Here, you can see under the Browser section,
there's a tf.browser.fromPixels function
that you can call to create a tensor 3D from a given image.
Even better, the image can come from a variety
of image-like elements, such as a HTML image, Canvas,
or even a video element.
So first, you need an image to use.
I've gone ahead and uploaded an image
that you can use for this tutorial
that's also hosted correctly so it will work across the mains.
And on that note, any image you want to use
needs to have the CORS header set correctly
on your web server, so that it can be used on the front end
without issue.
If you host a file without setting these headers
correctly, you might see a message
stating that your canvas has been tainted,
like the one shown here.
If your image is served on the same domain
as your front-end code, then this will not be an issue.
But this is something to watch out
for if you use images hosted from other domains,
as you'll be doing in this example.
OK.
So next, you'll modify the HTML code to add an image to the web
page.
So first, edit the body of your index.html page
to add the example image as shown.
Here, you'll give an ID of exampleImg,
force a width and height of 640 and 360 pixels
respectively, and specify the cross-origin attribute
to avoid CORS issues.
And finally, set the source attribute to point to the JPEG
image as shown.
If you updated the HTML correctly and view
your live preview, you should now
see a web page that looks something like this
with the image showing.
Now, it's time to head back and continue editing the JavaScript
within script.js.
Now that you've got an image available on the web page
to use, update your code to grab a reference to that image.
Here, you can call document.getElementById
and pass to it the ID of the HTML element you
want to grab a reference to, which in this case
is exampleImg that you just added to the HTML page.
This reference will be stored as a constant, named EXAMPLE_IMG,
as shown.
Next you can use tf.browser.fromPixels
and pass to it the EXAMPLE_IMG constant you just
created to create a new tensor, which you'll call imageTensor.
Finally, you can console.log the resulting tensor shape
to check what it looks like.
Now, if you check the console after running the updated code,
the first item printed will now be the shape of the image
tensor you just created.
Here, you can see that it returns a shape of 360 x 640 x
3 for the image that you added to the HTML, which
seems correct, as you specified the width
to be 640 pixels and the height to be 360 pixels in the HTML
markup.
Now, one thing to note here is that the convention
for loading images in TensorFlow is
to list the height first and then the width second, which
is why the shape of the tensor returned by fromPixels
has the axis sizes shown in this order.
Now, as a web engineer, I would have expected the first two
sizes to be the other way around, as you typically
deal with width and then height when using something
like the HTML Canvas.
So be careful this does not cause shape mismatch
issues for you in the future.
And if unsure, always print the shape
to check what you're actually dealing
with at any point in running the code in the program.
Now on the subject of shape mismatches,
the movement model requires an input of 1, 192, 192, 3.
And currently, you've got a shape of 360, 640 x 3.
The time has come where you need to crop and resize
the image accordingly to match the shape required.
Thankfully, the TensorFlow.js library has functions for that.
And because you're dealing with tensors,
manipulating all of those numbers
will be really, really fast.
Now, as with any challenge, there's
always many ways to solve a problem like this.
And if the image is large enough,
you could choose to simply crop the image.
But that would cause the loss of some pixel data, which
might be acceptable if you don't need to use it.
Alternatively, you can resize the image.
And there are two key ways to resize an image using
the TensorFlow.js APIs.
The first way is to use resize bilinear, which is the most
common way to resize an image.
To put it in terms of graphics editing,
if you were to take a small image
and resize it to be larger using this method,
the resulting image would be smooth, but blurry,
as you can see from the above example.
The second way is to resize nearest neighbor.
This method is useful for when the specific pixel values
of an image should not be interpolated,
creating new values that never existed before.
This instead creates more pixelated images,
as you see here, but the colors are preserved.
As the aspect ratios of the original image and the input
tensor image are different, a simple resize
would cause the stretching of the image.
Here, your original input image is a rectangle
and the input to the model is a square.
You must now address if you want to stretch the image or not.
So here's a rectangular image on the left-hand side,
and you need to turn it into a square image.
You've got three options.
First, you could stretch it in both dimensions
to become the new size and aspect ratio.
Second, you can maintain the aspect ratio
and fill any leftover pixels with, say,
a different color, maybe black.
And finally, you could crop part of the image of interest
to be of the correct aspect ratio
and then simply resize that.
Here, you can visually see the differences between these three
approaches.
So what should you choose?
Well, if the aspect ratio's of the final image
and the original image are fairly similar,
you might be able to get away with some small distortion
without affecting the model's ability to classify objects
in the image.
If there's a huge difference, like you
have in this face example, then the image
may be so distorted that the model will find it unusable,
as it wasn't trained on images that look like that.
Now for this tutorial, you will learn
how to crop an image tensor and then resize it.
So let's get to it.
Looking at the original image, there
are many ways you could crop an image
to get the resulting square image as shown here.
Now clearly, if you used the crop on the right,
the human's not even contained in the image,
which would be really bad.
Using the crop on the left would be preferred,
as it's just big enough to cover the human in the image, which
is what you want to perform the pose recognition on.
Now, do you remember a premade model
that you used before that could detect
the bounding box for common objects in an image,
like a human?
Yep.
That SSD model could actually be used here
to allow you to crop the image to provide a more optimal input
image for MoveNet to take as an input.
And as you can see, pre-processing
can actually itself involve other ML models
to get the best inputs to more complex models,
like this MoveNet model.
In the name of keeping the example code shorter,
let's pretend you've already used the Coco SSD
model to find where the person is in the current image shown.
Now, this is great.
But the bounding box is a rectangle,
and you need a square.
In this case, you can simply expand the width
of the found bounding box from the center outwards
to match the height size.
And now, you've got yourself a square.
Now, this square bounding box will
have a top left position-- in this case,
an x-position of, say, 170 and a y-position of 15.
And the bounding box also has a size,
which is its width and height.
As this is a square, both are going
to be 345 pixels in size in this case.
You now have the data required to crop
the image to a square form.
OK.
So let's go ahead and code up everything you just learned.
Start by creating an array to define the crop starting point
in the image tensor you previously created
on the prior lines of code.
Now remember that an RGB image tensor is rank 3,
so your starting point will actually
be a coordinate in 3D space.
The first value in this array will
be the starting y-coordinate.
And the second value will be a starting x-coordinate.
And then finally, the third represents the starting color
channel--
in this case red, as you want to retain all color data
for the cropped pixels.
Next, you can define the crop size.
As you saw from the [? crop defined, ?]
the square you want would be 345 pixels in size.
As such, the first element of this array
will be 345 to represent the height.
And then the second will be also 345 to represent the width.
Finally, you want to crop across all of the color channels
so that you don't lose any information, of which there
are three in total--
one for red, one for green, and one for blue.
So the final value in this array will be 3.
Next, you can perform the crop.
Here, you can simply use tf.slice and pass to it
the image tensor that you want to perform the crop on,
the crop starting point, and then
the crop size, which you just defined.
This will return a tensor of the smaller crop size.
And you can store this in a variable, called croppedTensor.
So now, you've got a square tensor that is of size 345,
345, by 3.
All that's left to do now is to resize
it to be the correct input size that the MoveNet model expects.
Here, you can use the tf.image.resizeBilinear
function that you learned about in the previous section.
It simply takes three parameters-- first,
the image tensor that you're trying to resize--
in this case, the newly cropped tensor that you just created--
second, the new shape of the image.
In this case, you want it to be 192 x 192 pixels in size, which
is represented by a one-dimensional array
with those values contained inside it.
Finally, the third parameter specifies
something called align_corners.
This is an optional property that
defines how the corner pixels are treated when scaling them.
And it's recommended to set this to be true.
Once that resize is performed, you
can cast all the numbers contained in the tensor
to be an integer, using the toInt method.
You need to do this, as MoveNet expects inputs to be integers,
not floating point.
So next, you can print the shape of the resulting tensor
to ensure everything worked as expected.
It should print 192, 192, 3 to the console at this point.
Finally, you can now pass the resizedTensor to the MoveNet
predict function.
However, there's one last thing you must do.
MoveNet expects a batch of 1 to be
passed, which would have the shape 1, 192, 192, by 3.
Thankfully, there's a handy TensorFlow.js function
to expand your TensorFlow's dimensions into this form.
Here, you can call tf.expandDims on your resizedTensor.
And the result of this would be the correct size
to be passed as an input.
If you run the code in its current state,
you'll get some output that looks like the one shown
in the console.
Taking the first element in the 17-item array,
you already know that this represents the nose
on the human in the image.
Here, it estimates that the y- and x-position of the nose
to be about 0.08 and 0.49, which represents
8% from the top in the y-axis and 49%
from the left on the x-axis in the original image.
Now, multiplying each of these by 192,
which is the square size of the input tensor to the MoveNet
model, and then swapping them to be in the form of x, y,
instead of y, x, you get the coordinate of 94, 15.
Plotting this point on the cropped input image
gives the location shown on the right, which
seems pretty correct to me, given
that this point represents the nose.
You can also see it's about 58% sure
that it plotted that point correctly,
which is useful to know.
Finally, plotting the rest of the 17 points
gives you the following image.
All that's left to do now is to take these found locations
on the square image and transform them
back to the location in the rectangular image you started
with.
As you know, the starting crop point
in the original image of 170 x 15 and that
original width and height of the crop
was 345 pixels before you resized it,
you can multiply the found normalized x- and y-coordinates
that were predicted by the MoveNet model by 345.
And then add those resulting x, y coordinates to the crop start
position in the original image, as shown here,
to overlay the points perfectly.
Of course, don't forget to dispose
all of the tensors you created once you're done using them.
As you've seen, more advanced models
need more preprocessing to take some input data
and get it into the form that the model can digest.
This work is not trivial.
And it can take time to do efficiently,
which is why the TensorFlow.js team provides
easy-to-use classes that do all of this work for you.
Now, if you plan on using the MoveNet
model in the real world, I highly
recommend you use the optimized TensorFlow.js
class that allows you to use it in just a few lines of code.
And there's a lot more behind the scenes
to make run very efficiently too.
But now, you've got the knowledge
to make machine learning models more accessible to others.
And I hope that with this knowledge,
you'll be able to take other models out there
as they come out and make them easier
for your peers to use and access,
who might not have taken this course before.
And with that, I'll see you in the next section.
[MUSIC PLAYING]