JASON MAYES: Now let's dive deeper
into our pre-trained models.
These models have already been trained by someone else,
so you don't need to gather your own data
or spend time and resources training them yourself.
Instead, you can load the model and use it directly
for the task it was trained for within your own production
system.
Now pre-trained models within the TensorFlow.js
can be exposed in two forms.
Some, like the ones for TensorFlow.js team
here at Google have produced, are
wrapped in easy-to-use JavaScript classes
that you can use in just a few lines of code
and are available for many common use cases.
These are great for people new to machine learning
and can be used in minutes, and you'll learn more
about these in this chapter.
Others require more knowledge of machine learning to use,
as they come in their raw form with no easy-to-use helper
functions wrapped around them, and you'll
be learning how to use these, too.
So here, you've got an example of a pre-trained model known
as BERT Q&A that can perform advanced text search in the web
browser.
Using this model, you can find an answer to a question
within any piece of text you present to it.
Notice here how in the demo, the question
uses words that are not in the answer.
If you ask it, what are the best stargazing days,
it finds the answer referring to the nights
during certain moon cycles, even though the days were not
mentioned explicitly.
This model can be used with any text and any question.
And here, it's shown running in a Chrome extension,
so you can also use it on any web page.
Now this pre-trained model is actually one of many
that the TensorFlow.js team have created and have
made freely available to use.
You may be wondering how hard it is to use something like this.
Well, using that set of official TensorFlow.js models
is actually really simple.
In fact, for [? core ?] code for this one
fits on a single slide, so let's walk through it.
So first, you import for TensorFlow.js library
and then the pre-made model that you want to use.
In this case, the Q&A model.
Next, you can define the text you wish to search.
This could be just some text on a website,
but here, I just use a simple string.
You can then define the question the user wants to ask,
which, of course, could come in some form of input box
in a real application.
Now you load the question and answer model itself.
As this takes time to load, it's performed
as an asynchronous operation.
So you use the then keyword to wait for it to be ready.
And once the model is available, a function
will be called, which is passed for loaded model
as a parameter.
Finally, you can then call model.findAnswers.
You pass to this function the question
you want to answer along with the text you
want to find the answer from.
Again, this is an asynchronous operation,
as it might take a few milliseconds to execute.
But once ready, this promise will
resolve to return an answers object, which you can then
iterate through to find the most likely answer from the given
passage of text.
In this case, it would predict cats as the answer
to the question proposed, which is correct
given the text you had to search on this slide.
And that's all there is to it.
It's no different to writing regular web apps.
Now since launch, the TensorFlow.js team
have released many easy-to-use pre-made models,
and we're continually expanding our selection, which
you'll hear more about shortly.
Models exist across many categories,
such as vision, body, text, and sound
that you can use in just a few lines of code,
like you just saw.
You can check out tensorflow.org/js/models to see
them all and to find the code snippets that show you how
to use each of them.
Even better, you do not need a background
in machine learning to use these.
Just a working knowledge of JavaScript is required,
but they are still very powerful.
So let's take a look at some of these in action.
And as I show you each one, try to think
about how you could use it to solve problems
that you or someone else might actually have.
First up you have object recognition.
Here you're able to run the popular COCO-SSD model live
in the browser to provide bounding boxes
for 80 common objects the model has been trained on.
What this means is that a rectangle or square
can be drawn that shows exactly where in the image
the object is located.
Now before I continue, you may have
noticed that some of the names of models
are not particularly friendly sounding
if you're new to the field.
This is something you'll get used to
and it should be noted that in many cases,
the name often originates from some combination of the data
it's trained on, the machine learning architecture
it uses behind the scenes, or the utility that it provides.
As you get more familiar with these things,
these names become less mysterious.
COCO-SSD, for example, was trained
on Microsoft's COCO dataset, which stands
for Common Objects in Context.
This is a famous dataset that contains hundreds of thousands
of images that were annotated by humans for typical things
you might see in your daily lives.
Furthermore, this model uses an SSD architecture,
which stands for Single Shot Detector,
and the scope of which is beyond this introductory course.
But know that this is just describing
some of the inner workings of the model itself.
And as you can see from the image on the right,
this COCO-SSD model allows us to not only understand
where in the image the object is located,
but also, how many exist, which is much more
powerful than image recognition that would tell us
that something exists in a given image,
but not where or how many.
And that's the key difference between object recognition
and image recognition.
So let's see this in action.
OK.
So here, you can see COCO-SSD running live
in a web browser on a real web page.
If I click on any one of these images at the top,
you can see the classification is coming back in real time.
Now here's just a few examples of the objects
it can recognize, and you can see
how you might use it for something useful, even right
now.
On the image on the left, you can
see that this dog is very close to this bowl of treats.
And you can imagine that you could detect this quite easily
and send yourself an alert when this occurs.
But of course, we can do better than that.
We can enable our webcam and now live
as I'm talking to you here today, if I scroll down,
you can see it classifying me in real time, as well.
And as I move my hands around here,
you can see the bounding box expand and contract all
in real time at a high frames per second.
You can see here it's recognizing me
as a person with about 86% confidence,
which is pretty good.
Now what's really cool about this is that all of this
is running live in my web browser
on the client side in JavaScript,
meaning none of these images are being
sent to the server for classification.
And that protects my privacy as an end user, which
is very important these days.
OK, let's head on to the next model.
Now you're not just limited to using images.
Here you can use our sound recognition model
to recognize short sounds.
You can even retrain the model to recognize custom sounds,
if you wish.
We even got models for understanding language.
Here, you can use our text toxicity model
to automatically discover if some text is
potentially insulting, threatening, or toxic.
Maybe you could hide potentially offensive things
as a page is rendered for a more pleasant user experience.
What would you make?
Next is our Face Mesh model, which
provides high resolution face tracking that's
just three megabytes in size and can recognize
468 points on the human face across multiple faces
all in real time.
A number of companies are using this with existing web
technologies, and a great example of this
is by Modi Face, who's part of a L'Oreal group, that combines
face mesh with WebGL shaders for augmented reality makeup
try-on.
On the image on the right it should
be noted that the lady is not wearing any lipstick.
This is being augmented in real time in the browser.
And then the user can select different shades at will
to see what's best for them without needing
to install an app or even walk into a store.
Let's see Face Mesh in action.
OK, so here you can see Face Mesh running live
in the web browser.
On the left hand side, you can see the machine learning
in action, rendering this nice mesh-like object
on my face in real time.
And you can even see where it thinks my irises are,
shown by these red circles.
And if I just scrunch my face a little bit,
you can see how well it updates.
So ah, and then I squeeze my eyes,
you can see that updating all in real time very nicely.
Now then, not only am I able to do the machine learning
on the left hand side here, I can also
render this 3D point cloud on the right using three.js.
And this is one of the beautiful things about JavaScript
is that not only am I able to do the machine learning,
but there's also plenty of other very powerful libraries
out there for data visualization or 3D graphics,
as you see here, that you can use in a matter of hours
and make something very, very quickly.
Now, the keen eyed among you will
have noticed that my performance right now
is around 20 to 25 frames per second.
That's because I'm running on my graphics card via WebGL here,
and my graphics card is actually pretty old.
If I change this to WebAssembly, you
can see it's now going to execute on my CPU,
and that shoots up to 30 frames per second instead.
So you can change at will what hardware
you want to execute on, and that's very powerful.
All right.
So with that, let's head on to the next demo.
We also recently released two new pose estimation models
in collaboration with research teams at Google.
The first, MoveNet, is an ultra fast and accurate model
that tracks 17 key points optimized for diverse poses
and actions and can run at over 120 frames per second
on an NVIDIA 1070 GPU client side in the browser.
The second, MediaPipe BlazePose, gives us 33 key points
and is also tailored for a diverse set of poses.
This extra granularity, such as tracking both hands,
could enable gesture-based applications that might
be useful for certain projects.
There is also now a 3D version of this model available too.
Now both models have higher accuracy and performance
over our original PoseNet implementation that some of you
may have seen before.
So we recommend you upgrade and try them both out
to see what works best for your intended use case
if you're looking to use pose estimation in a future project.
If you'd like to instead focus on the hands,
you can do that using our hand pose tracking model.
As you can see, it can track up to 21 points in three
dimensions for each hand.
And with some extra logic, you can
use this data to detect gestures, sign language,
or even control user interfaces in a touchless way,
opening up a whole new world for human computer interaction use
cases live in the browser.
Next, you've got body segmentation.
This model enables segmentation of multiple human bodies,
as you can see in the image on the right.
Even better, some segmentation models
also bring back the pose, which you
can see by the light blue lines inside the bodies
on this slide.
This, particularly named Bodypix,
can distinguish between 24 different body
parts, represented by the different colored
pixels shown in the image.
Now the premade models you just saw
allow you to create pretty much anything you might dream up.
So let's take a look at some real examples.
Here inSpace use real time toxicity filters
in their web conferencing app.
You can see that when a user types something bad,
it's flagged before it's even sent
to the server for processing.
And it alerts the user if they might
want to reconsider what they're about to send,
creating a more pleasant conversational experience
on the platform.
This is powered by our text toxicity
model that was pre-trained on a dataset of over 2
million examples of comments.
Well, how about this IncludeHealth system that uses
pose estimation models to enable physiotherapy at scale.
With many folk unable to leave their homes
or travel remotely these days, this technology
allows for a remote diagnosis from the comfort
of their own home using off-the-shelf technology,
such as a standard webcam, that many people will have access
to.
Well, how about enhancing the capabilities
of a typical fashion website.
Here, I use a body segmentation model with some custom logic
to estimate my body measurements,
allowing the website to automatically select
the correct sized T-shirt at checkout.
Even better, this was made in just two days
using our pre-made body segmentation
model that you just saw on the previous slides.
And with a bit of creativity, you can take a model,
add some custom code, and quite literally give yourself
superpowers.
First up, invisibility.
This is more advanced than simply replacing the background
with a static image.
For that, you wouldn't even need machine learning, of course.
But notice here how when I go in the bed,
the bed still deforms in the image on the right
as I move around to give you this ghostly effect,
or how the laptop screen still plays.
This prototype uses Bodypix that you
saw to calculate where the body is not so it can eventually
learn all the background and then keep updating parts
where it's safe to do so.
And even better, this was made in under one day
and runs entirely in the browser, meaning many people
could try it out globally, even without having
a machine learning background.
You simply click a link and it just works.
No images are even sent to the server for classification,
leading to real time results.
And next, how about lasers?
Another member of the community combined his love
for WebGL shaders with a TensorFlow.js model
to enable him to shoot lasers from his eyes and mouth.
This actually uses the Face Mesh model
you previously saw to run in real time in the browser
without any issues.
Now whilst this is a fun demo, you
can imagine using this for a movie launch
to amplify the reach with a creative experience for fans
or more.
Well, how about teleportation?
By combining TensorFlow.js models with other emerging web
technologies, like WebRTC for real time communication,
or AFrame for mixed reality in the browser,
or even Three.js for 3D, you can now
create a digital teleportation of yourself
anywhere in the world in real time.
Here, I can segment myself in the bedroom,
transmit my segmentation to save bandwidth,
and then recreate myself in the real world
environment using WebXR.
Remember, all of this is running in a web browser.
No app install is required, leading
to a frictionless experience for the end user.
Having tried this myself, it really
feels more personal than a regular video call,
as you can walk up to the person and hear the audio
from the correct direction.
Maybe next time I'm presenting to you,
I'll be able to do so in your own room like this,
as if I was standing right in front of you.
And you saw it here first, of course.
Now everything you just saw was created
using a pre-made off-the-shelf model that
typically can be used in just a few lines of code.
My point for showing you all of these examples
is that with a little bit of creativity,
and by leveraging your existing web engineering skills,
you can use many of the pre-trained models,
like the ones you just saw, for pretty much any industry
out there, providing your customers with new features
that were previously impossible to achieve within the same time
and budget constraints.
So keep this in mind as you learn more in this course.
Think about how you can relate what
you learn so that it can be combined with your existing web
engineering skills to produce something new.
And with that, it's time to try some of these out for yourself.
Choose three of the pre-trained TensorFlow.js models
from the ones currently shown on this slide,
read the documentation, and try the live demo of each
to get yourself a feel for the inputs the model expects,
such as image, text, or sound, along with the outputs
it provides you.
Now, some parts of the documentation
might seem overwhelming at this stage, but fear not.
You will learn how to integrate a model into a real web
application later on in the chapter step by step
from a blank canvas.
So no coding is required right now.
I just want you to familiarize yourself
with the models that are available.
And then, of course, you can answer the questions
in the form as follows.
What inputs does the model need and what outputs
does it provide you?
What problems in your or someone else's life
can it solve if you were to use it in a real application?
And finally, did the model demo perform well for you?
Share some examples of when it did
or when it did not work well, along with how
you might be able to overcome those limitations
once you've found them.
For example, maybe you find that the estimated pose
points move around slightly between webcam frames.
You might choose to average the found coordinates over time
to get smoother results.
Or maybe you're using an older device
and the model runs slower than expected.
Remember, as you're running on your machine,
everyone will have a slightly different experience
based on the hardware that you've got available to you.
Maybe you can change the user experience to account for this.
Or if a demo supports it, try a different backend
to execute the model on different hardware,
such as the CPU or graphics card.
So head on to the next section and share your findings
with your classmates.
[MUSIC PLAYING]