JASON MAYES: So before you move on
to using the second type of pre-made model,
you need to understand a very important concept known
as a tensor.
TensorFlow is, after all, named after it.
So as you may have guessed, it's pretty key to a subject.
So let's find out some more.
Now, the very first thing you'll find,
if you look at the TensorFlow.js API as shown on this slide,
is this thing called a tensor.
And tensors are the primary data structure
in TensorFlow programs.
Machine-learning models literally
take tensors as inputs, manipulate them in some way,
and then spit out tensors as outputs.
You can think of them as being similar in structure
to an array.
And just like arrays, they can have multiple dimensions.
But tensors almost always only contain numerical data,
unlike regular JavaScript arrays,
that could contain a whole mix of objects
that you might use or invent.
Furthermore, the tensor class, itself,
has a whole bunch of useful functions
that can assist in transforming them into different sizes
or dimensions, or doing math upon the numbers contained
inside them, and much more.
Basically, an array extended with super powers.
Now, this flow of tensors for a machine-learning model is where
TensorFlow gets its name from--
essentially, the flow of tensors through the model.
Now, I mentioned tensors can have multiple dimensions
just like arrays.
Let's quickly recap what this looks like,
and the slightly different terms you might hear
when speaking about tensors.
First, the most simple is some data
that has no dimensions, a single value, if you will,
also known as a scalar value in mathematics-- like the number
six.
Here, you can see a little bucket
of memory shown in orange containing the single number
all by itself.
Now, machine-learning folk like to describe dimensionality
in terms of a word called rank.
So here, the tensor has a rank of zero--
which simply means it has zero dimensions, or axes,
if you prefer.
In JavaScript code, you've used scalar values
like this over time.
For example, when you create a variable,
and assign it a single number as a value--
like let value equals 6, as shown on this slide.
Using the TensorFlow.js API, you can instead
call tf.scalar, and pass it the number directly
to return a tensor holding that number.
Next, you can have a one-dimensional tensor.
You can think of this as a list of numbers,
just like a one-dimensional array.
And mathematicians would call this collection
of numbers a vector.
And when constructed as a tensor,
TensorFlow folk would call this a rank one tensor,
as it uses a one-dimensional array
to store those three numbers.
It should be noted, however, that in this case,
the vector data stored is three dimensional-- it
has three numbers in the array.
Those numbers could represent a single coordinate
in 3D space-- in x, y, and z, respectively.
So in that sense, the data, itself, is three dimensional.
Now, do not confuse a 3D vector with a 3D tensor.
In the prior example, the vector is 3D
as it contained three numbers.
But for tensor holding that data was one-dimensional, or 1D.
If you had four numbers in the array, the vector would be 4D,
but the tensor would still be one dimensional with rank one.
So for term of dimensionality can represent
one of two things, as you just saw.
It can either denote the number of elements
along a specific axes of a tensor--
like the 3D vector that you just saw--
or the number of axes in a tensor.
In this case, it was one, as the container is just
a standard one-dimensional array.
Here, you can visualize the array
as a list of numbers in memory, all
going in one dimension, as shown on the right.
In JavaScript code, you probably used a one-dimensional array
over time to store values as shown.
Using the TensorFlow.js API, you can instead call tf.tensor1d,
and pass it to one-dimensional array of numbers
instead, to return a tensor of rank one holding those numbers.
Up next is two dimensional tensors.
Imagine you had a grayscale image.
Every pixel in the image would have a value from zero to 255,
representing different shades of gray the computer could draw.
Now, each pixel has an x and y location
somewhere on that 2D image.
So to store those values whilst retaining
the correct positions, you might choose
to store this data as a two-dimensional array
in your code.
Now, mathematicians call this a matrix.
And TensorFlow folk would call this a rank 2 tensor.
Here, you can visualize the array
as a rectangle or square of numbers as shown on the right.
In JavaScript code, you may have already used 2D arrays already,
like the code shown, to store some data for an image,
or maybe even a board game state,
for example, to keep track of what pieces is
in what location of the board.
Now, using the TensorFlow.js API,
you can call tf.tensor2d and pass it
with two-dimensional array of numbers
to return a tensor of rank 2, holding those numbers.
And in a similar manner, three-dimensional tensors
also exist.
A great example of this is a regular, full-color RGB image.
You may think images are two-dimensional,
which, for the purposes of grayscale images,
that's pretty true.
However, a color image is made of red, green, and blue
channels.
Each pixel, therefore, needs three numbers
to produce the color that you see on your screen
with the correct mixture of red, green, and blue.
So to store the data for a color image,
you actually need a rank 3 tensor, as shown.
This example essentially shows a 3 by 3 pixel RGB image
stored in memory, where one layer could be the red values,
layer two of green, and layer three the blue.
To visualize this, you can think of many 2D layers stacked
together to create a 3D shape, like this cube
on the right-hand side, with each sub-cube containing
a number in this case.
Hopefully by now, you can see the pattern
in the code, where as you increase
the number of dimensions, you're essentially nesting
more arrays inside each other.
And the TensorFlow.js code just calls tf.tensorxd,
where x is the number of dimensions
that you have, up to a maximum of six.
You then pass through that function
the multidimensional array of numbers,
which will return a tensor whose rank is
the same as the dimensions required.
Visualizing beyond 3D is sort of tricky.
So I've tried my best here to draw some visualizations
to help you see how this works in terms
of the nesting of arrays in a visual way.
A rank 4 tensor structure is simply
an array of three-dimensional arrays that you just saw.
Here, you can see a light blue container representing
the fourth dimension, which itself
contains a collection of 3D arrays like the ones
you saw on the previous slide.
Now I've not written the code for this,
as it's hard to fit on a single slide,
but the principle is the same as you previously saw.
Hopefully, this visualization helps
illustrate what's going on.
And a common use case for this sort of data
would be video data that contains a time element.
Here, you can see that a video after all,
is just an ordered set of RGB images over time.
And in the example shown here, you
can see that you have three, 3 by 3 pixel RGB
images stored over time.
Each with a red, green, and blue channel, one after each other,
and all contained within the new array.
And yes, you can go further.
A rank 5 tensor is shown here, building
upon the previous example, which is
like having a batch of videos.
Here, you can see how each video from the prior slide
is now stacked inside a new array containing them all.
And it should be noted that there are also
some real-world scenarios that might need a rank 5
tensor to store their data from sensors
that are more advanced than regular webcams,
or when using very specific data types, say, in the 3D industry.
One potential example would be storing
voxels, which simply is a fancy word for saying
three-dimensional pixels.
Lets look into this deeper.
In the image on the right, you can
see a screenshot of the popular computer game called Minecraft.
This digital world is represented
by a whole bunch of cubes that are essentially the smallest
subdivision of the 3D world space that you can have,
which is known as a voxel.
Just like the smallest subdivision of a 2D space
is known as a pixel for regular images.
Now, each one of these voxels has a specific color value,
representing different objects in the game.
But the users can manipulate to create and build
whatever they want.
Now, if you were to store a portion of this Minecraft world
in the form of a tensor, each voxel
may need to store the RGB color value it represents.
As free values would need to be stored
in a one-dimensional array, this would be the first dimension--
the color of the voxel.
This voxel now needs to be assigned a location in the game
world, for example in x, y, and z.
Which means you need a second, third, and fourth dimension,
respectively, so you can track the coordinates
of where that voxel is located in three dimensions.
And finally, if you were coding these values over time,
you would even need the fifth dimension
to store frames of this data.
You can think of this as an animation
that records changes to the voxels,
but you can then play back over some time interval.
And last but not least, you might summon a rank 6 tensor--
this is the highest rank tensor that TensorFlow supports.
Realistically, you probably won't ever
need to use it for most situations,
but know that it exists.
Continuing the previous rank 5 example
for recording voxel states over time,
you might find you need to record
a batch of voxel animations to send to your model in parallel.
In this case, you'd need that sixth dimension here
to do that.
All right, that's quite a bit to digest,
especially as you typically only work up
to three dimensions in your mind.
So feel free to rewind and understand the examples
provided before continuing.
Now, when working with tensors, there's
some common vocabulary you'll hear folk often referring to.
Most importantly, all tensors have
two fundamental properties-- data type and shape.
So let's dive into the details.
Data type is the type of data that the tensor will store.
For example, integers which are just
whole numbers, or floating point numbers, which
are just fractional numbers with a decimal point, like 0.2.
The amount of memory used to store these numbers
defines the range of values that they can store.
So as you can see on this slide, int-8 stores
a smaller range of numbers than int-16.
Now, as a JavaScript programmer in the web.dev world,
you may not have had to deal with specifying numbers
to this level of detail before.
Typically, you would just say, let x equal 2,
and be done with it.
But in the machine-learning world,
it's important to do so, as using the wrong type
could eat a ton more memory, as often larger networks are
dealing with millions of numbers in the model.
For this reason, you'll see typed arrays
being used a lot in TensorFlow.js that JavaScript
also supports, even though few people need
to use them normally.
This not only helps with memory efficiency,
but also for performance.
If an array only stores one type,
then it can be accessed more efficiently, too,
making a program as fast as possible.
Next, shape is the length of a number
of elements, if you will, of each of the axes of the tensor.
In this example, you have three-dimensional tensors,
as you saw before.
The shape for the tense on the left would be 3 by 3 by 3.
And on the one on the right, which is also
a three-dimensional tensor, its shape might be described as 3
by 3 by 6, using the same convention.
Now, rank, as you saw before, is simply
the number of axes that the tensor has,
or the dimensions, if you prefer to think of it like that.
This tensor has three axes, so it has a rank of three.
Axes or dimensions can be individually called up to.
For example, axes two in this visualization
runs along the cube from the left to the right, as shown.
Finally, size is just the total number of items in the tensor.
If you know the shape of a tensor,
you can multiply the numbers to get its size.
This 3 by 3 by 3 tensor has a size of 27 on the left there,
and the 3 by 3 by 6 tensor has a size of 54.
Now, once you've got your data in tensor form,
you can now make use of all the powerful functions provided
by the TensorFlow.js library.
A few simple examples are shown here.
In the first line of code, you can
see it just creates a new 2D tensor
and assigns it to a variable, called tensor.
In a similar manner, the second line of code
creates a scalar tensor with the value of 2,
and assigns it to a variable called scalar.
But next line shows you how you can multiply
all values in the first tensor with the scalar value you just
created by calling tensor.mul, and passing it the scalar
tensor you want to multiply by.
Next, if you put the contents of a new tensor just created
by doing that, you can see all of the values
are now double what they were in the first tensor
that you created at the start.
Now, you might wonder, why not just use regular JavaScript
to multiply all the numbers by 2?
Well, simply put, using tensors means
you can take advantage of the faster
execution on the graphics card, or other back ends,
that TensorFlow.js supports, which
can do many of these operations in parallel or much faster
than vanilla JavaScript.
This means your program is going to run super
fast when doing these sorts of operations
on huge arrays of numbers.
Which when you're dealing with things
like images that contain millions of numbers,
that performance difference can make a huge difference
in execution time.
Finally, on the last line, you can
see how you can even change the shape of tensors.
In this case, it takes the 2D tensor from the first line
and reshapes it into a one-dimensional tensor
with six elements.
Pretty neat.
Now, these examples may not mean much on their own right now,
but later in this course, you'll see more useful use cases,
for which these sorts of transformations-- for example,
when working with image data--
and converting back to being tensor form,
will be input into a model more efficiently.
Now, the beauty of having these standard ways to represent data
is that the library knows how it can use such data
on different hardware types.
As it now can rely upon the fact that data
will always be consistent in this well-defined tensor form.
But TensorFlow.js library works on several backends,
as you saw before, like WebGL for the graphics card,
or WebAssembly for the CPU.
And this means that the library can accelerate the tensor
operations that you perform on the desired hardware
for you, without you having to worry about what the tensor is
actually executing on yourself, saving you
a lot of time and complexity of writing all that code,
yourself.
Now, one final point I'd like to share for consideration,
is that tensors need to be manually destroyed,
and are not disposed of like most things in JavaScript.
In regular JavaScript, if you declare a variable-- let's say,
x equal 1--
and then stop using the variable, x,
the browser will automatically clean up
the memory it used with its garbage collector,
freeing up the RAM and resources for the computer.
For a tensor, you must dispose of memory
yourself manually using a special dispose function
or the convenient tf.tidy, which will automatically
dispose of any tensors created within a given function.
Now, this might come as a bit of a shock to you,
if you're not used to writing in other languages, like C,
you have to remember to dispose of variables that you
create but you no longer need as standard.
You'll learn more about putting this
into practice later in the course.
So for now, just realize this is something you'll need to do.
Otherwise, you could cause a memory leak
if you keep creating tensors, say, in a loop,
and never dispose of the old ones.
Great, so with that, you now have the 101 of what tensors
are, it's time to head back to learning
how to use slightly more advanced premade models well
you'll need to work at the tensor level
to pass data in and retrieve outputs from such models.
See you in the next section.