[MUSIC PLAYING]
JASON MAYES: Now you know what pre-trained models
are, how do you select one to use in a real-world project?
In this section, you'll walk through
a hypothetical, real-world scenario, breaking down
the problem into the following steps.
First, what exactly is the problem you're trying to solve?
Secondly, once you know the problem,
what models do you know of that could assist you
in solving that problem?
And then finally, considering all identified usable models,
how do you select one to deliver the best solution?
So let's get to it.
So first up, imagine a real customer has contacted you
and states, "I'd like to create a web app that
can detect and alert me when an intruder is in my garden,
but does not send any of my images to the Cloud."
Let's break this down.
One thing right away that you can take note of
is that the customer wants a web app.
This means TensorFlow.js could be a great fit,
as you know it can run both client side in the browser,
and also server side via Node.js in the Cloud.
However, the privacy was also of key importance.
And in this case, TensorFlow.js client side in a browser
is the only option.
Diving deeper into the requirements of the app,
it will need to detect when an intruder is
in the user's garden.
Upon clarifying with the customer what that really
means, what is an intruder, it seems
you just need to detect humans and not cats, dogs,
or other animals that may also enter.
This means you'll want to make a system that
knows when a human is in a given frame of the video stream.
All right, so you've already seen a few models
that could potentially do this.
In fact, can you guess which ones could detect
the presence of a person?
Feel free to pause the video here and note down
which ones you might consider before you continue to see
if you got them all right.
I'll wait for you to do that.
[CLOCK TICKING]
All right, let's check your options
from these known pre-trained models.
Technically speaking, all of the listed vision and human body
models could be used to detect the presence
of a person in an image.
And yes, that includes the face and hand models too.
It's also worth noting that, depending on the requirements,
sound could also be used to detect the presence of talking,
even if the person's not visible.
It's often quite surprising how many ways
you can solve a given problem.
Now let's assume, after checking with the customer,
that sound is not suitable, as they
have very talkative neighbors.
And they don't want to accidentally detect that.
In that case, you've got a choice of the following--
image classification, object detection, body segmentation,
pose estimation, face-landmark detection, or even hand
pose estimation, if the hands are in the shot.
Let's break down these to see the pros and cons of each.
In these situations, it's helpful to fill out
a table of features that matter to the task
that you're trying to solve.
Some common key things to always check
include the inference speed, which
is the time it takes from sending
new data as input to the model, to getting an output
prediction.
Lower times, typically measured a milliseconds,
mean faster performance that can also sometimes be
measured in frames per second.
For example, 10 frames per second,
which means it takes 100 milliseconds to run,
as you could run it 10 times before a second would have
elapsed, which is important when working
with real-time applications, such as video, as often video
will need to run at 24 frames per second or greater.
If the model runs slower than this,
some video frames will have to be skipped for classification,
which might be acceptable, depending on your use case.
But typically, if the classification time
drops below 10 to 15 frames per second,
the application may appear to feel laggy,
especially if you're trying to deal
with fast-moving objects, which would
lead to a less than beneficial user experience.
You might also want to check the amount of memory
the model uses, both in terms of raw file size,
typically measured in megabytes, where less is better
if you want the page to load faster,
but also in terms of runtime memory used to execute it,
which is the amount of RAM it would need
to run on the machine, which again, is typically measured
in megabytes.
With modern machines having a lot of RAM and fast internet
connections, which is becoming less of an issue,
but you should still be mindful of these things
as your end user might not have the same luxuries.
And on that point, knowing your users'
expected working environment will
help decide which models are not suitable faster, and of course,
avoid disappointment later.
As web engineers and designers, you
may already be used to this.
More than websites need to be designed responsibly,
based on the device it's being run on.
And the same rules apply here for machine learning models
you might choose to load.
Back to our example scenario.
After checking with the customer,
you learn they want to run the system on a spare smartphone
that they have lying around.
But fast internet is available at all times,
as it will be connected to the house Wi-Fi.
So it's now time to fill out a table
to help us reduce the options.
But how?
Well, if you're lucky, the documentation for a model
may include a performance section that details
some of these metrics already.
Check this first to save time.
In fact, a screenshot of the pose estimation model
documentation is shown here that shows
the expected frames per second you can
achieve on different devices.
However, no details for memory usage and file size
are provided.
For undocumented items, you will need to benchmark the model
yourself by creating a simple website that loads the model
and uses it, recording these values yourself
on the device of interest.
Let's walk through how to record these values,
if they're not already provided for you.
First, frames per second, you can simply
record a timestamp just before you execute the model,
and then record the timestamp once you get a result.
Subtract the two, and you'll have the number of milliseconds
it took to execute.
To convert this into frames per second,
you can simply divide this number into 1,000,
as 1,000 milliseconds represents one second.
So if it took 50 milliseconds to run, 1,000 divided by 50
would give you 20 frames per second.
Now, for file size, you can check this pretty easily
in the Chrome Developer Tools.
You can press F12 on your demo page to open it.
Or simply right-click anywhere and choose Inspect.
Once loaded, switch to the Network tab,
and ensure Disable Cache is selected.
Then refresh the web page, using Control+F5 or by manually
refreshing the web page in the browser window.
And as the page loads, you'll see all the web page
assets being loaded too.
Note that it might take some time
to complete loading all of them.
Now, TensorFlow.js models consist of a model.json file,
along with the associated binary files that use for .bin
extension, for which there could be multiple binary files
loaded.
Once the web page has stopped loading
and is working as intended, voiceover
recorded results for the model.json and the bin files,
from which you can add up the file sizes of each
to get a total model file size.
For this example model, it's around 4.8 megabytes
of total data transferred to the browser, as you can see.
Similarly for RAM usage, again, open the Chrome Developer Tools
once the application is running, and ensure that the model
is being used as intended.
You can now switch to the Memory tab and take a snapshot.
And then switch to the Statistics view
to get an overview of how much RAM is being used.
Note how, in this case, this is a larger size than the file
size of the model itself, totaling around 56.6 megabytes
of RAM usage at runtime.
OK, so let's assume these are the benchmarks you
get for the models of interest.
You know that for real-time applications,
you probably want at least 10 frames per second.
Looking at the table, you can see that the body segmentation
model only runs at 2 frames per second on the target device.
For this reason alone, it's probably not
suitable for this mobile use case.
So you can safely drop it from the contenders.
However, it should be noted that there
might be other body segmentation models out there that
run much faster.
But that would require further research and investigation.
Let's just stick to the models you know for now.
So now let's look at the file sizes.
Typically, a smaller file size may
be preferable for a web page that's reloaded all the time,
especially if it's loaded over a 3G connection
on a mobile device.
However, given your use case where you load the page once
and then leave the app running, this
is less of an issue for our customer.
The overhead of waiting, say, 2 more
seconds, to potentially use a better model
is acceptable here.
And remember, the customer for this use case
also specified that the device will have access to fast Wi-Fi.
So download time is not really an issue.
You can therefore, still consider
all of the models shown for this example scenario.
Now, looking towards the RAM usage,
you can see that face landmarks and hand
pose are really high, with 70 and 125 megabytes respectively.
Given that the intention is to run on a mobile device, which
might also be an older generation model, maybe
with just one gigabyte of RAM, it
would be wise to be mindful of this
in case the user decides to run other apps simultaneously.
Furthermore, not all humans have hands,
which makes the hand pose model less reliable for those use
cases.
And also, detecting the face would not
work well if the human was wearing a mask.
So this is also potentially a poor solution.
Let's discard both of these for now.
You are now left with three contenders-- image
classification, object detection, and pose estimation.
The next step would be to try all three of these models
to check if there are any obvious flaws in using them
for the intended use case.
And also, what unique benefits can each provide?
One thing you might consider next
is the number of humans in the image.
While this was not an original request from the user,
two of the remaining models support
detection of multiple people--
object detection and pose estimation.
As a developer, it makes sense to offer such functionality,
as this information could be beneficial, given the use case.
Remember, image classification will simply
be a binary yes or no, if a human is somewhere in the image
or not.
But not where or how many, which is less useful to you.
So given that object detection offers more information
with faster performance and almost half the RAM usage,
it makes sense to drop image classification
from consideration at this point too.
So now, you have two models left to choose from.
Diving deeper into the documentation for both,
you can discover that object detection, by default,
can detect up to 20 objects simultaneously,
and can even go higher if needed at the sacrifice
of inference speed.
The pose estimation model, by contrast,
can detect a maximum of six people at one time,
and uses over twice the amount of RAM to do so.
For this use case, the pose model
may not scale well if a larger group of people
came and you wanted to know how many people there were.
That being said, it is over two times faster
than the object detection.
So that could be better for fast-moving people.
Both of these models are suitable contenders.
You might, at this point, want to ask the customer
if detecting the presence of six people is enough.
And if they agree, then it makes sense
to use the pose estimation model, as it's faster.
If they need the extra granularity
to count a larger crowd, then maybe
the object detection model will be the way to go.
As you've seen, the model you ultimately choose
will depend on your customers' needs and the environment
on which it needs to run.
It's not always immediately obvious
which model will perform the best,
without further investigation and research.
And as a developer, it's up to you
to decide what provides the optimal solution
to bring the idea to life in the best way possible.
Now, in the next section, you will
create your first full end-to-end project
from a completely blank canvas where
I'll show you, step by step, how to make
your very own smart camera, just like the one
you explored in this scenario, using the object detection
model that you just selected.
So get your coding hat on, and I'll
see you in the next section to put what
you've learned into action.
[MUSIC PLAYING]