[MUSIC PLAYING]
JASON MAYES: OK, so in step 8, things get really exciting,
as you'll finally get your hands on using the machine learning
model.
In this step, you'll be adding more code to script.js.
So this is a file you'll be editing again and adding
further code to.
Let's go over the code to make sure you understand what it all
does.
As before, your model variable is first
set to undefined, to indicate that the model has not yet
loaded.
This globally accessible variable
is where you'll store the loaded model
once it's ready so that it's usable by other functions that
call it.
On the next line, you can now load the cocoSsd model
by calling cocoSsd.load.
Remember, you imported the cocoSsd model variable
in your HTML page just before you
imported the script.js code, which is
where this variable comes from.
Now, this load is asynchronous.
So as before, you use the then keyword
to wait for it to complete, upon which it will call
a function of your choosing.
Here, you'll use an inline anonymous function
for simplicity that takes one parameter, which
is simply the model class that's loaded, once it's ready.
Once the model has loaded, you can assign the loaded model
to your global model variable so that it's no longer undefined.
And then finally, you can remove the invisible class
from the demo section to automatically ungray it
once the model has loaded to visually show the user
the demo is ready to be used.
Moving on through the code.
Next, you'll define an array, named children.
This array will be used to store all of the HTML elements
that you will draw to the screen that
will be used to highlight the objects found
in a given frame of video.
Pushing all of those objects to an array
makes it easy to keep track of what
you need to delete before rendering the next frame
of data in an efficient manner.
Next, you will create a function called predictWebcam.
This function will act as the animation loop, if you will.
What this means is that once it's performed a prediction
once, it will call itself again to try and predict
the next frame of video when ready, and so on and so forth,
until you close the window.
This is known as a recursive function,
as it calls itself at the end, as you'll see later.
Here, you can see the first thing it calls is model.detect.
This single line of code is where the magic happens
for actually doing the machine learning model inference, which
is the act of actually using the model to do some work.
Now, do take note that all the code prior to this step
was pretty much regular web engineering-- setting up
the user interface, dealing with interaction,
or loading things in.
The actual act of using the machine learning model
was just one line of code, as you just saw.
OK, so model.detect takes a single parameter,
in this case, an image-like object.
And what do I mean by image-like?
Well, essentially, if you look at the documentation,
the model is able to digest any one of the following HTML
elements, including video, image, or canvas.
So here, you can pass it the HTML video element
that you had a reference to, from which
it will sample a frame in that moment of time
and try and detect what objects it
can find in that image frame.
Now, this act of prediction is asynchronous.
So it might take a few milliseconds to execute,
which, for a computer is a very long time.
Therefore, you must again use the then keyword
to wait for the predictions to come back.
Once the predictions are ready, they'll
be sent to the function specified.
In this case, you will use an inline anonymous function
once again, which simply takes one parameter that
will be the object containing the predictions that
was returned by the model.
Diving into the code for this anonymous function--
the next task is to clear the page of any previous elements
you used to draw found objects, before you draw any new ones.
Remember that children you already
defined a few slides back?
Well, it's time to loop through this array
and remove any children contained within the liveView
element, which is the element you will use to render bounding
boxes for objects found.
Finally, you can use a neat trick
by calling splice(0) on the children array
to delete all the erased contents efficiently
in one line of code.
With this done, you're now ready to draw
the bounding boxes of the new objects that have been found.
But what exactly are bounding boxes?
And how are they defined?
Essentially, in this use case, they're
just rectangles that are drawn around an object of interest
in an image.
In order to draw a rectangle, you
need four pieces of information--
the top left x,y coordinate where the rectangle will start,
along with the width and the height of the box itself.
With that, you'd then be able to draw
any rectangle to the screen.
Let's go ahead and see what comes back
from your model predictions.
You can now loop through the array of predictions
that returned by the model predict function call
in a regular for loop as shown.
Remember, there might be more than one prediction,
as there could be, say, several cats
in the image at the same time, for example, which is why
you need to use a loop here.
So while n is less than predictions length,
you're going to inspect each nth prediction further.
Now, for every prediction found, you
can check the score of the prediction.
In machine learning, when you classify something,
many models produce a number that's
often linked to how confident the model was
that the object was indeed the thing it thinks it is.
Reading the cocoSsd documentation,
you learn the score represents a percentage confidence,
but the numbers will range from 0 to 1.
So a value of 0.66 would mean it was 66% sure.
You just need to multiply by 100 to get the percentage
value in this case.
On this line, you can set a minimum threshold of confidence
to only draw bounding boxes for items that the model is really
sure about to reduce the chance of it drawing something
incorrectly, also known as a false positive.
As a developer, it's up to you to decide
what's an acceptable score to use in order to draw items
that your end users will then see.
Do you want more or less false positives?
If you set the value too high, you may never see anything.
And it should be noted that different models will
need different thresholds to work well.
Experiment with this value to see
how it changes what's drawn, once you've
completed the codelab.
Continuing to the next line of code--
if the prediction exceeds your minimum threshold specified,
you can render some details about what
was found to the web page.
Here, you can create a new paragraph element in memory,
using document.createElement(p), so that you can then render
some text to the page.
On this next line, you can set the paragraphs
in a text to be the predictions class that
represents the name of the object, like cat, dog, chair,
and so on.
You can also print out the prediction score too.
Now remember that the score will be a floating point number,
with a value between 0 and 1.
So here, you can use the parseFloat
to ensure the number is interpreted as floating point,
and then multiply it by 100 to get the score as a percentage.
And finally, you can use Math.round to convert it
to an integer to make it look a little neater
to look at on the eyes.
Next, you can add some CSS styles inline
on the creative paragraph element,
so that it's positioned in the same location as where
the object was found.
Notice here that the current prediction object
has a property named bbox.
Inspecting this on the console, you
can discover that it's actually an array with four elements.
As you may have guessed, those four values
correspond to the top left x- and y-coordinates
for the bounding box rectangle, along with the width
and height.
So bbox 0 would be the top left x-coordinate.
Bbox 1 would be the top left y-coordinate.
Bbox 2 would be the width, and bbox 3 would be the height.
You can use these values to position the paragraph text
in a sensible location, like the top left of the bounding box,
as shown using the regular CSS styling.
Now, the keen eyed among you may have
noticed I subtracted 10 pixels from the y-position
and also from the width.
This is because, in the CSS styles you copied earlier,
I have a 5-pixel padding on these elements.
So to compensate for that, you must
subtract 10 pixels in each dimension
so it renders in the correct position on the screen.
The only thing left to do now is to draw the bounding box
and add all the elements to the page.
Here, you'll use a development by creating
document.createElement(div), set with a class of highlighter
so that it gets a nice semi-transparent background
and a bright border from the style.css code copied earlier.
Then, just like before, you can set the CSS style properties
manually for the left, top, width, and height
of the element, using the values in the bbox array
to position it so that it overlays
the correct part of the webcam video stream
to highlight the object found.
Now that all the modifications of the new elements
are complete, you can append the newly created div and paragraph
to the liveView container to put it onto the web page itself.
Now, by using a pen child, you're
actually adding it to the HTML page, making it visible.
And the reason you do this now, rather than before,
is for efficiency.
If you had added the elements first and then
started changing best styles and values,
every time you made a change, the page
would need to re-render and reflow
all the other visible elements too.
By keeping them in memory until everything is pre-calculated
and finalized, you can create a highly efficient web page
as the reflow will only occur once.
Next, you can add both of the newly created elements
to the children array, which will
be used as a quick reference to delete them of a next run
through the function so that they're
disposed of correctly before rendering the new results.
Note that the order in which you add the elements
matters here, as you want the paragraph to be rendered
on top of the bounding box.
For this reason, it must be added after,
as the last element added will have a higher Z
index than the one before, by default.
The only thing left to do now is to recursively
call this function again, once the browser is ready.
You can use window.requestAnimationFrame
to do this, which would pass control
back to the browser to finish up anything it needs to do.
And when the browser is ready, it will call the function pass
to it, which in this case, is the same function you're
currently writing, predict webcam.
This will then start the loop all over again,
except next time the new video information will be available,
allowing you to update the position of the bounding boxes.
You should always use request animation frame
to avoid junkiness.
And never use settimeout for render loops
like this, to ensure the best performance across devices.
So with that, it's time to copy the code you just learned over
into the project for the grand finale.
Feel free to pause the video whilst you do that,
and then continue when you're ready.
[CLOCK TICKING]
All right, so if you copied everything over correctly,
you should now see a live preview
like the one shown here after you click the Enable Webcam
button on the app.
Bounding boxes will be drawn around anything
it recognizes, so try different objects
like bottles, chairs, beds, and see what it can detect.
Or check the documentation for more details.
The other thing to note at this point
is that the core code related to the cocoSsd model
itself was only three or so lines of code.
The rest was either setting up the user interface,
just like you would do for any other web app,
or writing code for the post-processing of the results
in order to draw the outputs to the screen,
like rendering the bounding boxes.
And TensorFlow.js ore-made models like this
are very easy to use.
And remember, the others detailed
before can be used in exactly the same frictionless
manner, just with different inputs and outputs.
So congratulations.
You've just created your very first machine
learning-powered web application with TensorFlow.js.
Now, step nine recaps what you've learned,
so let's take a look at that.
In this exercise, you've learned the benefits
of using TensorFlow.js pre-made models
and created a fully working web app that utilizes one of them
to classify common objects in real time using your webcam.
This involves you creating a HTML skeleton for the content,
defining styles for the HTML elements,
detecting and enabling a webcam stream,
loading a pre-trained TensorFlow.js model,
and then using that model to make continuous classifications
on the video data from the webcam,
and finally drawing bounding boxes around the found
objects in the image.
As for the next steps, I highly recommend
you play an experiment with the code
you've just written before continuing.
For example, try changing the confidence threshold
for detection.
And see what renders when you make it really low,
say 0.1, or really high, like 0.95.
What do you notice?
Also, think about how you can take what you've just made
and extend it to solve a real-world problem of your own,
like I did with the pet cam example.
Maybe even try to recreate something similar if you want.
Now if you decide to try that, consider the following
questions-- what extra code can you
write to detect if two objects intersect, for example, when
a person sits on a couch?
This extra code and logic needed is just regular JavaScript
at this point.
All of the machine learning is already done.
You then may want to think about how you might alert yourself
to the fact, this has happened.
Maybe you can use WebSockets or a third-party API
to send yourself an email or a text message.
It's completely up to you.
And finally, if you send an alert
every time an event happens, you might
want to consider the edge cases where multiple objects come
into and out of the view.
How can you deal with a group of people
at a party sitting on a couch?
Maybe you want to have some time delay
before you send a notification.
Or maybe you want to send a notification only if
the number of people increases, rather than decreases.
There are many edge cases to consider,
but all of these things are regular web engineering
problems and are not machine learning problems.
So feel free to experiment with this and see what you can make.
Now if you'd like to see how I solved
some of these problems in my pet cam solution
that builds upon the code you wrote today,
feel free to check out this video that
goes into much more detail.
And check out the code directly via the link shown.
Now, if you make something useful,
consider sharing it with the community using
the MadeWithTFJS hashtag on social media,
like Twitter and LinkedIn, so we can see what you've created.
[MUSIC PLAYING]