I'm gonna hit this button here and then
I think I don't know what this is not
something I've done before so let's see
in a second we should see this this
appear here yeah that looks promising
okay all right all right let me just ask
now if people in the live chat can hear
me and then I'm going to then I'm gonna
close this yeah this is gonna go on
forever all right all right hi hi hi hi
hi hi hi hi hi hi I just want someone to
say the sounds works okay okay so I'm
gonna close this so so from this moment
on my focus is now on you the the the
the people who are physically here in
this room
I just wanted to turn that on and so I
don't know if anybody if there's if
anybody is has that live stream like in
the background on their machine and
there's some like really important
question or well tons of people saying
like it stopped you could let me know
but I'm gonna not pay attention because
this is a rare and unique opportunity to
be physically with people in real life
so so hello welcome I guess it's a few
minutes early but now I might as well
start my name is Dan Shipman thank you
so much to grow Paris and litang for
having me here this is like a real honor
to get to be able to be here in Paris
and do a workshop and meet people who I
have correspondent with online or maybe
met before
and yes I'm thrilled to be here
so I most people are probably familiar
with me someone because that's probably
how you found out about this but in case
we've just just as a matter of quick
introduction what I do actually
full-time is teach at a program in New
York called ITP I'll just pull up the
website real quick this is a two-year
graduate program at Tisch School of the
Arts and actually that's why I'm here in
France as NYU has a study abroad program
at NYU Paris and I'm visiting that for
most of this week and so in addition to
this I have this YouTube channel where I
make coding tutorials that's where most
people find me yeah oh they can't see my
face oh because I fight go like this
alright so sorry to the live viewers I'm
this is this is just the webcam I'm
gonna be moving around a lot but I'll
give you a moment of looking at you but
otherwise it's gonna be it's gonna be
hit or miss so but and so the other
things that I work on I said I think are
important to mention which are related
to what I'm going to show today is I
helped to I don't know what the right
term is administer run manage
participate in something called the
processing foundation and the processing
foundation and say it's a United States
charity a non-profit company that
maintains a bunch of open source
software toolkits for the Arts
processing p5.js if you don't have a p5
sticker I didn't bring any processing
stickers I didn't have any important bit
but but if you want to see one Simon
here has one so so so so one of the
things that so so pressing foundation
has does a lot of in community and
education initiatives one project in
particular that I want to mention is
that I just did called p5 something that
I started working on a little over a
year ago is a project called ml 5 and
this is an open source library and the 5
is an homage did I use a
maybe slightly I won't I did I do have
like high school American high school
level French which is very bad
we'll see if I can see how much courage
I have any of my French but so the five
is an homage to p5 but this project is
funded actually through a grant from
Google which comes from the team that
created something called tensorflow Jas
so I'm just giving you that as
background for what I want to do in the
workshop today what I want to attempt to
do in the workshop today is build a
simple image classification engine in
javascript in the browser using the p5
web editor so you know so so let's let
me think here for a second so if you
Eddie all of you are welcome to be here
no matter what your background and skill
level is I imagine that some of you this
might be kind of beginner 4 and some of
you might maybe have just started
programming there might be some stuff
that's very new and confusing so I'll
try to manage that as best as I can but
you could certainly ask questions and
maybe also as a small community here we
can help each other for those of you who
have a lot experience with this so if
you're but if you're new to programming
or to JavaScript then I would suggest
using the p5 web editor I'll mention a
link you'll need to go to in a second
but if you're experienced with web
development in JavaScript you don't you
could be following along and doing the
coding through whatever editor system
build environment that you so desire but
but you'll need to have the things that
you'll need if you're in the p5 web
editor will all get packaged there for
you if you're not you're going to need
to have the p5 libraries and you can get
those through a CDN link which which is
which is here so you just go to the p5
web editor and open up the index.html
file you'll find these so it's some
point if anybody's like really stuck and
can't find p5 let me know and I'll try
to oh I'm definitely gonna like pause
and take breaks so my goal is to spend
about an hour it's one o'clock now kind
of talking about
what how image classification works in
ml 5 and a particular algorithm called
KN n which stands for K nearest neighbor
and try to build up something and then
hopefully if that goes well then we can
just hang out here for the second hour
and people can try to make something
with it maybe towards the end you could
share a couple things that are or just
you know so this can this will start
very presentational but I'm happy for
this to move towards being more just
hanging out and working on stuff all
right ok so and let me just let me just
show an important link here so if right
here github.com slash Schiffman slash
the tank workshop this is where i at
least so far put all the links of the
things that are relevant and I will I
will go over this in more detail in a
second but this this is probably the
most important link for you web editor
template because the thing that I want
to mention is that Here I am in the web
editor over on the top left up there if
I click this little arrow shape you'll
see in here it says ml5 Minjae s so I
have built a version of the ml5 library
just for this workshop with some new
features that don't exist in the
published version of ml5 so if you were
to be using the published version of ml5
the stuff that i've got to try to code
today won't work so on so that's
important and if you aren't used so
you'll need you'll start here and then
the first thing you can do is just to
save or duplicate I think so if you if
you want to use the P 5 web editor and
you have it before I would take the time
right now just to go to editor type
p5.js Dorgan sign up for an account
because and then you can go to this link
and click on web editor template but I'm
gonna I'm gonna be showing you a whole
bunch of demos and things before I
actually start writing code but I just
wanted in case people are trying to
figure out to get set up yeah
yes so I usually do this is a good good
question on that Simon has asked so here
in settings there are there's a there's
a few different views and usually I use
this high contrast mode I didn't use it
because it just wasn't set to it
actually on my laptop but also I think
I'm gonna have a bunch of text that
would this gray background might end up
looking a little weird so let's put it
back here to this okay all right anybody
have any like just sort of general
questions or logistical questions in
terms of like how to be set up before we
start oh and then there's the Wi-Fi over
here which is I won't broadcast the
secret Wi-Fi what could possibly go
wrong with me saying but the Wi-Fi
password is but that seems like it seems
like I shouldn't okay alright so I think
what I'm gonna start with actually is
this website called teachable machine
how many of you have seen this a couple
of people so so teachable machine is a
project that was made by Google Creative
Lab and a bunch in collaboration with a
bunch of different research teams at
Google and it was this came out like
quite a while ago before the official
JavaScript tensorflow was launched and
it demonstrates something to me like
this is a really amazing demonstration
of what is very recent
only recently possible to do in a web
browser so I'm gonna skip the tutorial
and just run this to show this to you so
the idea of a teachable machine so just
you know spoiler alert my goal is to
actually basically build a version of
this in the P 5 web editor it will have
none of the bells and whistles that
you're seeing here in terms of design
and you know thoughtful interaction but
but it will have the functionality it
will allow you to create an interactive
system in the browser that might be
different than a way you've thought
about it previously so just to show you
the way this works I
I think I practiced this with my code
example earlier so let's see so what I'm
gonna do here is I'm gonna basically
teach this machine to learn three
different what are often referred to as
classes or you can call them labels
three kinds of images so one image is
just gonna be me
that'll be green so I'm holding this
down so it's just learning like if I
just move around here this is I'm giving
it lots of examples of me in the web
browser okay
sorry not in the web browser me just me
and the camera now I'm gonna step away
and I'm gonna do I'm gonna train it just
on with nothing yes yes so if you yeah
you're right if you watch the tutorial
will walk you through this and so the
last thing I'm gonna train it with is
with me holding this book that I found
here called blog story so okay so now
that I've done this I've now trained
this system and you can see what I'm
sitting here it's confidence and
actually only when I'm looking straight
ahead its confidence is 99 percent that
it that's me the green category the
green label if I walk away we should see
that it became confident in purple if I
sit back down we're back to green and
then now if I hold the book it's orange
and what time is pointing out is right
so this is also showing you like it's
gonna display a different gift based on
which category you can also do sound so
it's playing birds
combo spreads okay so um so this is the
idea this is the idea of on top of how
the sort of like process of image
classification works that you're
teaching the computer you're saying
here's a lot of examples of images of
cats here's a lot of examples an image
of the dogs the computer in theory is
going to learn from that and then you
can present it with new images and it
will make an appropriate guess so but
there's something that's that I'm not
telling you which is super important so
this is um this is working only because
it is building on top of something
that's already already been trained on
millions and millions of images so this
is a process known as transfer learning
so there any of any questions about like
just I'm gonna switch over to some
slides that are gonna kind of like pull
go underneath the hood of this a little
bit more but before I move away from
here okay so let me just close this
window and I'm going to sew the this by
the way so one thing that I want to
mention is you name she who is a
colleague at NYU at ITP she taught a
course this past semester called machine
learning for the web and it's basically
it's a seven week course that goes
through some the basics of machine
learning and different kinds of
algorithms and models so here's a bunch
of and so basically what I'm attempting
to do in this workshop is what's here in
week two but there's lots of other
features of the ml5 library that
tensorflow library that might be things
you would want to look into later and i
forgot to include i really didn't want
to have to pull
my need to have a link to this make sure
my volume is off oops
okay so let me just also wanted to
mention so this particular playlist also
goes through and I realized people who
are maybe watching this live I'm
stepping away so I guess I could swivel
this so this particular playlist which
is actually in some ways somewhat of a
prerequisite so I'm kind of gonna gloss
over some of the stuff that is in some
of these videos but certainly if you
want to after today go back and dive a
little more deeply into some this stuff
this is some material that you could
look at and the thing that's different
the thing I want to point out what I
what I'm doing in this workshop today is
almost exactly what's in video four and
five but I'm going to use a different
algorithm for number five called KNN
this is a recent feature of the ml5
library that you name she who taught
that class added to the library itself
so as we go through maybe I'll come back
and try to distinguish those a bit more
but that just wants it like sort of et
this context okay so I'm gonna go to
yuning's presentation and use some of
that to kind of talk through to talk
through how some how this actually works
so where's I present okay and thank you
to you name for allowing me to use a
basic presentation so ml5 library comes
with something called a pre trained
model a pre trained model is a machine
learning model that somebody has already
spent the time to train to recognize
images and the particular model that
that that ml5 comes with is is called
mobile net so mobile net of you can see
this is what mobile net does it
basically looks at an image and then it
says like oh it's a Robin
confident 99% so weirdly mobile net so
mobile dad is kind of amazing in that
it's this machine learning model that
you could just immediately have access
to and we could actually I'm gonna I'm
gonna go to ml5 jazz org I'm gonna go to
examples I'm gonna quickly click on
video classification so this is an
example now of mobile Matt trying to
classify the images in this video and
you can see let's see if we can get this
maybe sunscreened sun-blocker
almost fit book and then it bookstore at
one point let's see bottle of water
maybe your glass water bottle water
bottles so and we look at these
sunglasses bowtie usually it says yeah
so this is the thing michigan machine
learning models are only as good as the
data set with which they were trained
yeah thinks it's a meat cleaver so so so
one of the so let me go back to this
presentation here so let me find this so
mobile net was trained on a particular
image data set called
image net and and basically this is like
a image data set that was made just for
researchers to experiment with not
necessarily for practical real world use
and it has a incredible amount of plant
elements in it it has like a whole bunch
of different like sport sport things you
can see it's it's it has six 1603
pictures of the dog so what but what the
what the mobile net model which was
trained on this database and I'm gonna
go to here real quick to show you
something
it only knows
about these 1,000 things so you could
see like this is what's so crazy a lot
like machine learning off it feels like
magic oh it's like this really smart
amazing system that can recognize
anything and in fact it's really good
this particular model can recognize a
variety of species of lizards and
reptiles apparently but it can't
recognize a person so so that's one
thing that's important
while it's useful and fun to play with
and it's a nice demonstration of the
idea of image classification it's not
necessarily going to be really useful
for you to use with your own project
unless what you happen to do is you want
a project about rare bird species then
maybe the mobile net model will by
accident be very useful to you
but what it actually does one of the
things yes so this is the list of the
classes so there is something however
you probably don't realistically have
access to a database of 50 million
images so let's say what you want to do
is recognize that the example that
meaning has made and that I'm gonna do
something with this like
rock-paper-scissors the game which I
learned today is Pierre Foy sees oh I
would have said wash we're by the way
okay so yeah so rock paper scissors so
you would like if you wanted to train a
machine learning model to recognize you
making the rock gesture or the paper
gesture or the scissors gesture mobile
net is not going to do that because it's
not able it doesn't know anything about
images of people's hand gestures but
there is a process called transfer
learning which it's defined here is use
the knowledge gained while sovereign one
problem in applying it to a different
but related problem and what what what
mobile net actually does the math behind
mobile net is when it looks at a cat
image what it actually does is creates a
graph of probabilities and you can see
these are all one thousand things it
knows about and ended up here with this
very high probability that it is like
you know number 288 or 280 the class ID
is 285 the Egyptian cat so this is what
this is the end result of the machine
learning and this actually by the way
comes from this really nice observable
notebook that I like I'll click on in a
moment also by McKeel or at who's one of
the creators of on the development team
of 10 attention flow - s so this is the
mat this is what the machine learning
model actually outputs an array of
probabilities but if you were to on
again this isn't going to be this
workshop I'm not gonna I'm gonna kind of
like hand wave the aspects of how neural
networks work but if I were to a neural
network in one way to describe it is
right if if the image is the input and
the very last thing that comes out is a
list of probabilities which I'm showing
you right here there are actually a
whole bunch of steps in between and one
of those steps which is the last step
right before the probability is known as
the like logits activation so this is
basically the unnormalized predictions
vector which sounds like a very fancy
word but basically this is this is
basically looking at almost like a
rating of a numeric rating for how
likely a particular image is any one of
these categories and you can see that
you see these high spikes but what what
this gets turned into in the mobile net
model is something much simpler because
all you want is kind of like a medium
specification is the answer which one is
the most likely so soft my max is a
mathematical function that basically
just watches everything and leaves the
highest ones but this is actually a it's
like this numeric
image this is what's known as like a
feature vector in other words the image
which is maybe 250 I know what the
dimensions are let's say it's 256 by 256
pixels has that's how many numbers whoa
could go I think I should know this math
but I'm in front of a live audience so
that's 65,000 pixels and if they're RGB
suddenly we have like one hundred ninety
six thousand numbers so 196,000 numbers
okay what did I do wrong
by the way this is like actually what
happens in right there so there's
infinitely more possibilities that's
that's right so there's there's 256
Siamese right there's 256 to the third
power possible grayscale values I could
say but but yeah RGB well aren't you
being right
but oh yeah yeah there's 256 x 256 spots
pixels and each each spot has 256 to the
third power possible yes but the point
what I'm saying is whatever whatever the
actual map is which I could get exploit
lead from but thankfully Simon's 0
correct me this is a much this is a lot
less information this is like a thousand
numbers and so the what what the what
the neural network actually learns to do
even though it boils all these pixels
[Music]
yes so so this particular this
particular like this particular array is
known as the features so what we can
then do and there's here's some more
examples so let me let's look at some of
these
okay so what we can then do is we can
make use of the fact that mobile net is
very good at taking any image and making
it into an array of numbers and then
what can we do with that so the reason
why I'm also using this term called
vector I'll write this over here so I
usually when I think of a vector I think
of like an arrow
it's a and what that this is a vector in
two-dimensional space
yes and it has an x and y component but
as Simon is saying you can have a vector
in three dimensional space which should
I have XYZ component the truth of the
matter is mathematically speaking even
though our brains aren't really able to
visualize past three dimensions in an
easy way this list of numbers is
basically a vector in 1000 ameno space
yes exactly
a plain list of numbers so if I set you
on this on this white board over here
and I know I guess I will make some
small effort to turn this this way you
know this point here is it more similar
to this point or this point you might to
ative lee say this point because it's a
closer distance so we can see that and
we could also we could say this is often
I think that an example I gave in some
of the videos is if we thought of think
of RGB color if we filled this room with
red green and blue colors on when like
this accent was the amount of red this
axis was the amount of green and then
this axis was the amount of blue similar
colors would be near each other so
looking at how data appears near to
other data in space
is a measurement of
molarity so if I were to basically look
these two are probably pretty near each
other in a thousand dimensional space
right and this is what we can this is so
the process of transfer learning is
basically use mobile net to get and a
thousand a thousand numbers from an
image and then find your nearest K
nearest neighbor King nearest nearest
which basically is a way of saying okay
which category is something just
something a member of based on its
proximity to other entities in that same
space so we go all the way back to that
teachable machines just for a second now
if I go all the way back to here what
I'm basically saying is here's a whole
lot of examples of me in a thousand
dimensional space here's a whole bunch
of examples of nobody in a thousand
dimensional space and now here's a new
image look is it closer to images that
it's already knows about of me in the
picture or with me not in the picture
and this new image is closer to all of
those but I'm not if I were to like try
to compare every single pixel color I
might be able to sort of get something
like this to work but it would run super
slow and it wouldn't be very accurate
but the fact that mobile net already
knows so much about what the mean the
meaningful parts of images and turns it
into this thousand dimensional array
that we can basically take forget about
this probabilities we basically get rid
of that last layer about with the
natural labels and just use the features
of an image to try to match any new
image with previous images I don't know
if that made any sense at all
anybody have questions yeah right so
sorry
this technique yes so sorry so I kind of
I'm jumped I'm jumping around here quite
a bit certainly but this technique the
the mobile debt model is just a digital
file with a lot of like information
numbers in it and that's a thing that
you could use in Python in the browser
and then so the concept of transfer
learning can be applied in any
programming language with a lot of
different machine learning libraries or
from scratch if you have you know a few
years to like write every every bit of
the algorithm yourself but what I'm
gonna show you is the quotes and what's
what's the reason why this is exciting
at least to me is the fact that this is
a way to get a image classification
system working in real time in the
browser which prior to this prior to the
JavaScript tensorflow version and a lot
of these models like Bobo met be made
JavaScript compatible would be things
that you would need more powerful
computers or lots more training time to
be able to do okay so let's see what so
let me show you actually a really nice
example of how this is can be applied
and so this is one of the tensorflow j/s
examples and I'm just gonna click on it
here so let's see this is yes allow use
of camera okay so I'm gonna make this a
little bit bigger this is basically a
project that someone made apologies that
I don't know the name of it but this is
one of the official attention flow Janus
examples where what I could do now is I
can say okay let's say I need pac-man is
for controls so what I could do it's
basically create a physical controller
from myself to to control the game of
pac-man if I train a train a set of
images for both right left up and
so I'm gonna just say that me just
looking here regularly is is this so I'm
gonna click here this is this is for
moving up I'm moving to the right is
hold this book for moving to the left
that's gonna be impossible to remember
and and I'll just leave for moving down
so now in theory I'm gonna hit this now
something that's really important I
don't know you might not have noticed
this but when I was when I was showing
you teachable machine
there was no button that said train I
just added the images and then it
started guessing but here in this system
there's a button said that says train so
there's actually two different ways of
approaching this st. there's more than
two but in terms of the examples that
are in the ml5 library and that you'll
find the tension flow j/s there are two
distinct ways of approaching this
problem this is using a slightly
different one this is using the method
that's actually in the video tutorials
that I made and now I'm showing you this
other method called K&N look things up
like pause for a minute or two and try
to distinguish those more clearly but
for now I'm just gonna mention those are
two different things so I'm gonna hit
train so it's training lost by the way
is a keyword that you'll see in machine
learning quite a bit it's also called
cost and the fact that it's zero is
probably not actually true it's probably
point zero zero zero zero that's a bunch
more zeros before it gets to some actual
numbers but on it's the amount of error
so it's been training and how does it
know the amount of error well it's
trains itself with all these images
knowing which ones are right left up and
down and then it almost is like well
even though I know the answers I could
also try to guess the answers so then it
looks at those images again trying to
guess but the answers are and sees it it
guesses the correct answers if it if it
didn't get a single one wrong it's would
be 100% correct mean it's air
would be zero okay so now I'm gonna
click play and we're gonna see if I can
play this where would I want to go to
the left first maybe oh yeah wait hasn't
started yet
left okay up up okay wait I wanna go
right down there we go so you got the
idea I'm not sure this is a left oh so
okay so the point of this and another
project that I will mention which is in
this presentation towards the end is oh
this is the post meta stuff which I'm
going to talk about is this is by and ni
TP our researcher named alejandro called
pong ml I won't play this through right
now but this is a similar idea where he
trained a teachable machine for certain
gestures to move pong paddles up and
down so this is a pretty you know
there's hopefully your ID your brains
are filling with ideas beyond just like
oh I could train this to be a like a
weird gestural controller for a simple
2d game but this is a nice way of sort
of demonstrating what is the sort of
creative possibilities of what you can
do with this kind of transfer learning
technique and this so there's the other
thing I think is interesting to mention
here is that you'll notice that like I
just trained this right now so in theory
a couple things one is I could I could
save all of these training images and
the resulting model and then load it
again like if i refresh this page all
that all that work is gone but I'm what
I'm going to show you when I get to the
examples you can actually save that and
load it again later but even though you
can do that which is more typical of
machine learning systems what's exciting
about this is you could imagine building
like an interactive like museum exhibit
kind of interactive kiosk where it's
just continually being people are
continually training it all day or all
night or all
whatever however long ago usually okay
so let me see what did I what did I miss
here so let's see what else is in
innings presentation that I wanted to
yeah so I've got I bet I'm gonna talk
about Poe's net but I you know I want to
be conscientious about the time here so
let me look here and see if there's
anything important here that we missed
so this is more for the code stuff that
I'll come by these are some of those
examples okay
and actually let me just briefly show
you
oops this is Nikhil's observable
observable is a system for creating kind
of like interactive JavaScript notebooks
it's one way of describing it's built by
a bunch of people but I think started by
Mike Bostock who's the creator of a
JavaScript library called d3 and so
Nikhil this is how you would basically
create a teep the teachable machine demo
with tensorflow j/s and i'm hopefully
going to show you an even easier way of
doing it with ml5 but the reason why I
wanted to bring this up here is this is
a nice also demonstration of calcaneus
NeighborWorks again we're in two
dimensions here and you can see it's
trying to guess is my mouse mouse part
of the blue I guess that's kind of blue
purple who knows blue group or red group
and basically yeah the reason why the
algorithm is called K nearest neighbors
is it classifies it according to a
coding system that's okay I just stepped
on this everything's fine it classifies
it according to a voting system so in
other words for this new point it's
looking for its K nearest neighbor K
being 3 2 of its nearest neighbors are
read-only one of its nearest neighbor is
blue therefore it is more likely to be
of the category red it is more likely a
cat than a dog but you can see that this
is the kind of algorithm if I first I
could add this which is showing you like
actually where is that decision boundary
that's every any point in space that's
on this side is going to be classified
as
this red category and then you could
also see how this works if we change
like K so you can see here it's you can
see that here it's got one two three
four five six six times closest to that
are red and three so this is a
particular algorithm that's a classic
machine learning algorithm K nearest
neighbor that can be computed very very
quickly which which makes it powerful
and it actually the oh look at this I
didn't even notice this you could change
this to three classes so now we can see
how its category how its categorizing
based on three different clothes three
different classes okay so a couple
things so let me quickly let me close a
bunch of these windows that I don't
necessarily need open anymore
because we're gonna start writing the
code for it you can follow along but
before I do that here we go
let me whoa oh boy there's already lots
of links here Thank You Alka who is
watching the live stream and I gave
right permissions on this repo who added
it done some fantastic okay so what I
want to just show you is are these two
examples real quickly so this is if you
go to this one that says KNN image
example this is basically a fully
working
[Music]
KNN image classification example in the
p5 web editor I'm gonna go back to a
blank a blank sketch and put the pieces
together step by step because I think
that'll be more most helpful while
falling but if at a good point you just
wanna go and use this this is the sort
of working version of it and has a few
certainly it's not as sophisticated as
teachable machines but has more dollars
and whistles so I'm just gonna mention
that that's there but the other thing
that I was like what what's interesting
about the can and algorithm is even
though I'm talking about okay well we
get those photo logits
from mobile net that vector of a
thousand numbers
there's lots of other kinds of data that
comes as an array of numbers so this the
ML 5kn classifier object can take any
arbitrary input so for example another
example that I want to show you which I
don't think we'll have time to code the
whole thing but I will show it to you is
this pose net example so what this
example is doing this is using a machine
learning model called pose net which
basically makes a guess as to where and
this is really designed this is not
designed to this is designed to be used
with like more of a full-body view so
attempt to see if I can make that happen
but you can see what pose net does is it
can it looks at any 2d image with a
person if I'm yes right if I turn to the
side it tries to make its best guess but
it's most accurate with a well-lit full
body image so what it's doing is it's
basically making a big list of x and y
values here's the X Y value for the left
hand the right hand the right elbow the
right shoulder it kind of has this
skeleton and nodes about so what this is
doing go back to it's actually will work
even as I can basically say like oh let
me make two categories here let me say
that these are a bunch of examples of me
with my head to the right and here's a
bunch of examples of be to my head with
the left and now you should see 100% be
100% a oh and goodbye I don't know what
just happened there
I was weird you know why that just
decided to completely die but I'll do it
again left I think I took the other way
before and so you can see it's able so
so this this is not using even though
the image is there the can and
classifier is not trying to categorize
the image input at all it's trying to
categorize the results of the pose net
model which are XY positions of the body
and then so you could imagine using this
with an application to choreography or
and/or gesture in a more robust way than
the image classification example work I
don't know why this decides to just like
die after every few a little bit that's
the I think the live workshop disease
that like live demo itis I think it's
called I don't know why that decided to
stop working okay so that's the sort of
like story of all of the pieces yes so
I'm keeping an eye on the time thank you
Simon so what I want to do now is I want
to start a bit from scratch and so if
you want to follow along you would go
here to this link web editor templates
and you should see that it says the tank
template by coding train and then what
you're gonna want to do is you have to
be logged in I'm already logged in as
coding train but what you would want to
do is then save or I'm gonna do
duplicate and then what you would see is
the tank template copy by your username
so I'm gonna give everybody a minute to
see if they can get to the point where
they have a copy of this and if you want
to work offline you could also just do
file download in old
all the libraries and everything and can
use a different text editor so I'm just
going to give everybody a minute or two
to see if you can get set up with the
web editor at this particular normally I
would start a workshop of just like oh
just start a blank sketch but the reason
why I can't do that here is you need a
copy of this because it has this special
version of the ml5 library as part of it
so you can check in this left-hand
column to make sure you see Emma five
minutes
you've gotten working you can help your
diesel
[Music]
the live the live audience to Larry I'm
tempted to check I don't let's just I'm
just gonna check real quick on the live
audience do I have any I see people are
tweeting about this workshop that's
great let's go to live stream know
there's like a lot of people watching
this that's crazy okay okay okay so the
tank template the template copy okay
okay I'm going to move on in a minute
you know like 30 seconds to a minute
anybody stuck and won't want to ask okay
if you're too shy poke the person next
to you get them to help you
we're all we're all nice helpful people
here I can see smiling faces okay okay
so so I don't know how far like I mean
you certainly we have a fair amount of
time we want to leave some time for
people to just try to play around with
it a bit on their own so I'll try to get
as far but this as I can
in the next half an hour and then we can
always fall back on the sort of pre-made
example and really you can just kind of
give that a try but so so I'm gonna you
can follow along I am going to the first
thing that I want to add is just the
video live video so I'm gonna create a
variable called video and then I'm gonna
say video equals create capture did you
know I think that's what it is let me
run this so if you've never used the p5
web editor before it's really just this
is the text editor where you can write
JavaScript and you can stop and restart
your program with these stop and play
buttons you can also click this auto
refresh and it'll update
it'll rerun the sketch as you're typing
but I'm going to keep that offer right
now yes yes now you see me next to me
probably okay so now we have the video
now one thing I want to do is I actually
just want to I want to make this smaller
and I want to just I think it's gonna be
this is not really important for the
machine learning aspect but I just want
to take this video and draw it on the
canvas my drawing canvas which p5 is
through my default setup to work with
and so I am going to say image video 0 0
I'm gonna change the videos size to 320
240 so now you can see
the video is in two places so three
times so now I'm gonna say video dot
hide and so now I just have the video so
this is just a few bits of code to just
get the video drawing in the canvas okay
now can people see this font size okay
okay so I'm just gonna try to pause but
I'm gonna move fairly quickly through
the stuff you should all just wave your
arms flail your arms and me to slow down
if you need a general question okay so
now now the next piece that we need is
we need to have this thing called the
ml5 feature extractor so I'm gonna call
this feature I just call a feature
extractor to call it so I'm gonna say
feature extractor equals ml v dot
feature extractor then so the feature
extractor an image but he knows how to
do that based on a pre-existing model
that pre-existing model is mobile net so
I'm gonna and in theory we could apply
this same technique to non-mobile other
models but ml5 really only said at the
moment only supports mobile net maybe
one or two other models and then I'm
also gonna give it a callback model
ready so this is the kind of thing this
by the way requires an internet
connection I mean it already requires an
internet connection to be using the P 5
web admin record but even if you if you
weren't using the P 5 web editor
it's got to load the mobile net model
from the cloud it's possible to download
that and have that locally but that's
not a thing by default that it does so
I'm gonna say console dot log model
ready so now if I run this again you can
see but down here in the console if you
see that at the bottom it now says
already so I've created the video and I
have made a feature extractor so I have
video which is the source of my images
and by the way you don't have you can do
this with you know JPEG PNG image files
you don't have to do this with live
video that's just the way I'm
demonstrating it in sort of like live
interactive sense okay so now what I'm
gonna do is I'm gonna add a function
called mousepressed probably I'm gonna
need some buttons at some point but this
will be fine right now and I'm just
gonna show you something there is a
method feature which is feature
extractor dot infer I think this is all
I need to write so what this function
does I could I might have this wrong I'm
about to look this up what this function
does is it says give me that list of
thousand numbers from an image and that
image is the video and I'm gonna say I'm
gonna put that in an array called inputs
and then I'm going to just do something
called inputs I'm gonna say inputs print
and Alex want to make sure this works
and then I'll explain again what's going
on so this is me just testing the model
is loaded what I want to do is when i
press the mouse i should see here in the
console a list of a thousand numbers if
that happens then things are working
there we go I don't see all the numbers
but I see some numbers so this is very
important cannot move on if you don't
have this stuff working so you should be
able to see the video create the feature
extractor object and then infer that's
that I by the way I'm not a 1 percent
sure infer is the best name for this
function so if anybody has any ideas
about that you can join the ml 5
open-source project this ml 5 and
finally get up issues saying I have an
idea for the name of the infer function
might be good but the idea is to do
inference to infer the essence of
to take that energy give me my list of
1000 numbers so the weird thing is
though how come I didn't just say like
console dot log inputs so this is like
you know if this were like a longer
course about machine learning we'd have
done a lot of steps leading up to this
point but I'm just gonna kind of give
you the quick details here if I click
look at this whoa look at this is
disposed in internal shape d-type size
so what this thing actually is now I
meant to have both of these print out
even though I'm describing to you let me
just get this back what I'm describing
to you that the logit is just like yeah
how does it what's the first correct
pronunciation but I'm describing inputs
as well I meant to
I meant to I'm going to attribute that's
a jet lag so this array of numbers isn't
to plain JavaScript array of numbers
it's a special kind of object called a
tensor and by the way that's why
tensorflow
is called tensorflow the idea of a
tensor is a it's a fancy word for an
array of numbers but of any dimension so
it could be you know a multi-dimensional
array a two-dimensional matrix a
3-dimensional array it's tensors this is
a generic term and so the library tensor
flow is named for that because that's
the sort of core building block of
machine learning systems right even
though we think of like oh it's
generating code like while in creative
machine learning
this machine learning algorithms
generating poetry or it's classifying an
image these are just the sort of human
dressing we put all around the system
but the inputs and outputs of a machine
learning system are always numbers
tensors so what the feature extractor
does is it gives you this object called
the tensor now if I wanted to look at it
as a plain array I could say values
equals logits dot I'm gonna use a
function called data sync data sync is
like a function in that's part of
tensorflow j/s that will actually pull
the values out of the tensor and put
them in a regular array and now if I
click here we can see there we go we got
an array that has a thousand numbers in
it so I could look at it this way if I
want but I actually that's just for
demonstration purposes I want it as a
tensor because the next thing that I'm
going to do is I'm going to take that
tensor and add it to the K and a k-9
classifier right so what I need now is
I'm going to make one more variable
called KN n I'm going to say K and N
equals ml 5 I'm totally digging this up
classifier I should look at the
documentation or my other example let's
just look at this example yeah ml 5
cannon classifier I got that right ok
make a make a cannon classifier and then
what I could do is I could say KN n add
example these particular logits
are a cat
legit it's legit too legit to quit
okay no G wha sorry that's the fact
okay so let's I don't know I don't know
if I got this 100% right but let's see
if this works let's see if police I
don't get it here so now I'm saying like
I if I'm here here's some examples of a
cat I'm not getting any error so all I
can do is assume that it's working now
let's let's do this in a sort of awful
way I'm gonna change this to keep rest
I'm gonna say if key equals C I'm gonna
get the logits out otherwise if else if
whoops key equals this is where the
YouTube live chat will complain that I'm
not using a switch statement thank you
equals D then add an example of a dog
and let's books let me put some print
statements in here console.log cat
example added so again I'm doing it this
way to skip over the complexity of
making a nice interface just to
demonstrate the idea so now the idea
here is that I can say hey any time I
press a key on the keyboard I want you
to extract that list of a thousand
numbers associated with this image and
then I want to tell my KNN system that
this is a cat or it's a dog so if I and
by the way I might have to click over
here first to give this area focus to
register the key events so now I should
be able to say no so let's try this
console dot log key this was happening
the other day with the web editor and
then I assumed no this is not this
didn't just happen
whoa oh my goodness yeah this is the
demo itis again look at this what oh I
mean the whoa
the template Oh weird this is my code
but it's like oh yeah this is fine there
it is I don't know why it disappeared
weird but you should know that the p5
web editor is a you know a relatively
new project it's pretty amazing but it I
have note there have been some recent
bugs with it all right sorry I mean yeah
I don't know why that wasn't working
before and now it is hopefully you don't
you lose you're good by the way if you
if you're feeling nervous which I am in
a constant state of you can always just
do something like you know copy paste
your code every once awhile until like a
text file somewhere in case you're
worried that's the web the web editor is
gonna eat your homework okay so okay so
this is working I don't know why it
wasn't before oh okay let me start it
over so we can see that now it's adding
some examples of cats adding some
examples of dogs
[Music]
I can't alright there's a cat here this
is my favorite sticker by the way but I
can't firstly don't have any of those
okay so so sorry let me so this is where
we are so far so the idea the idea here
is that we have loaded the video we have
created and let me see if I can get more
of the code in case people are still
following along we lower this here so
this is almost all of the code I can't
get it all on to the screen but they
we're creating the video we're creating
a feature extractor let me move this
over then we are also creating a
classifier so the feature extractor gets
numbers from an image the classifier
associates those numbers with a
particular category and I just made up a
cat and dog as a kind of classic machine
learning example of distinguishing cats
and dogs okay so I'm gonna I'm gonna
just pause for another minute to see if
people get caught up here and then we're
almost ready for like the last piece of
this I really think this is actually
quickly there's not a lot to this if you
don't build a whole interface anybody
have any questions they want to ask
about what's I've coded so far any
questions
p5 JavaScript ml five machine learning
everything's fair game
yes so I will I will get to the Sagan so
that's the so but what's left for me to
do the next thing what the things that
are left for me to do is number one to
actually make it make a guess right I've
got to have a guess and then number two
I want to show you how to save your
training set and then I realize there's
still this there's still this question
of vs. but I I want to talk about the
difference between classifier and the
technique that I used in the video
series which are for the same result to
train your own image classifier with
built on top of mobile net but they're
both slightly different and I kind of
like the KNN technique better for a
couple reasons so I want to make sure I
talk about that at the end before I
finish okay
any other questions alright so now what
I'm going to do with my terrible no
interface system is I'm gonna add the
mouse pressed function back oh boy does
processing actually know what you know
what I'm gonna do ah oh jet lag jet lag
okay
actually what I'm gonna do I'm gonna
have it always guess that'll be let's
let me do mouse press first I think we I
want it to always guess but let me just
have it guess with a mouse press right
now so for it to guess the first thing
we need to do is take when you I was
about to say do the opposite of training
but it's not the opposite at all
it's
same first step this is by the way
training so we should in in machine
learning we talked about training and we
also talked about prediction aka like
inference maybe there's other words for
this guessing so for training we say
give me the numbers that are from that
image and then training I'm saying this
is called supervised learning I am the
supervisor this particular set number is
associated with this prediction is
saying well I don't know what it is what
I want you to do is now say KNN classify
those that array of numbers and then
this I need a callback and I'm gonna
call this function and again I'm using
kind of old-style es5 JavaScript which
hopefully doesn't mean anything to you
because who wants to fill our minds with
all this like es5 or es6 yes 8 well this
is there a 7 I don't know if maybe we
skip 7 okay yes Thank You Simon
not many new features right so it was ok
if we just talk about es6 in the essay
ok so but yeah I'm writing my JavaScript
to be sort of long-winded and
beginner-friendly
but just for those of you who might be
watching the live stream are here you
can use dot this will return a promise
you can say can classify then or you can
use the arrow syntax this JavaScript
there's always a thousand different ways
to do the same thing but in the simplest
sense what I'm saying is get that num
numbers from the video please classify
that for me and when you're done make
this event
function run this callback got results
you just print out the results to the
console so now in theory if I run this
again and here here but first I'm gonna
neurotically paste all the code into
here just so if it all goes away okay so
now I'm going to uh I'm a my a cat or a
dog I'll be a cat for today I am a cat
it's not working Cat Cat Cat Cat Cat Cat
Cat Cat Cat Cat Cat then I will show it
this book which is a dog we give it a
bunch of examples of dogs and now for
the grand a very anticlimactic if this
doesn't work I'm gonna just click the
mouse and it should say cats
alright let's so what did I get wrong so
this is what's when this has changed a
little bit let me look at the ya know so
there should have been an error I knew
there this error whatever so if I
haven't given it any examples when I
clicked a mouse
it's definitely gonna give me an error
because I haven't given anything to
train we could handle that error in a
more elegant way so let's let's uh let's
let's look at let's go look at the
working example oh boy this is has
changed again
alka is working working very hard here
okay let's go look at this code and find
classify feature extractor yeah that's
fine infer video Oh features that's a
nice way this is by the way I want to
rename my variable because that a I feel
awkward Lee can't pronounce logits or
load Jets and features is kind of a nice
word because those are that that's what
when you hear this oh what are the
features of the image and machine
learning speak that just means what's
the boil down numeric essence of that
image into that so let me change that so
this looks
let's let me go this is the working exam
this is this is the working example - if
anybody sees anything Oh interesting
well yeah so you know where it's okay
hold on infer classifier features got
results oh I know it's a problem is
classic error oh boy
I just err I make every single time
nobody's caught it yet in that I just
caught it I mean it I didn't show it to
you so I just realized what it is it is
because I come from a world of
programming JavaScript and weird
incorrect beginner-friendly ways that's
not very standard so this is how p5
would work like if you if you've looked
at in the p5 functions like load image
or load JSON there's typically a
callback and the arguments of that
callback is the stuff you want it's the
answer it's the result it's the JSON
file it's the image this is a most most
JavaScript libraries are written with
something called error first callbacks
meaning the callback always has more
than one argument and the error is
always first and the reason why people
who know more about actual JavaScript
programming than I do think this is
important is because this guarantees
that you handle your errors right if
because called if error came afterwards
this is just a design pattern it's not a
requirement of writing if this were the
way ml5 were implemented where the
results come in first but if there's an
error it comes in second that means I
could optionally ignore the error and
this is not a good practice in terms of
writing real software that needs to be
out in the world with millions of users
in the sort of like hey we're here at
the tank in Paris messing around with p5
you know we don't handle our errors so
well life will go on but so it actually
the error was undefined it was working
so so if I wanted to be a little bit
more thoughtful about this I would say
if error console dot error error and I
said console dot log but you can
actually say console dot error else
console dot log results so this this
should fix it and actually this should
give us on UPS and again the mysterious
that your code disappears for no reason
error okay so one thing I want to do is
I'm just gonna click oh whoa do we do we
maybe figure out what cat for you oh
thank you so you could use it so to be
interesting as an exercise would be
great to try to make this program but
use like images you take on your phone
as opposed to like from the webcam all
right why is that's really weird
does anybody yeah okay hold on let's
sanity check here stop play what's going
on there's no okay that's the this is
correct this is what I was expecting to
see there is no example in any class huh
okay let's let's this is this is loads
of fun this never happens when I
livestream ever I never have run into
errors or problems oh boy well where am
I
save open let me get my code back
I'm Savin in both I'm just gonna let it
okay let it rest
somebody needs to like let it rest for a
minute yeah let's comment out the mouse
press function for a second
okay click over here and I can add my
cat my dog examples okay let's put this
back save let it rest for a minute okay
click over here okay
whoa it's so weird is any other people
having this happen to them it's working
for for you let me go back here let me
okay so let me do something just out of
curiosity let's change this to like else
if key equals spacebar let me try doing
this oh yeah it's so great so what I'm
gonna do now yeah but that's gonna
happen up here in the got result
callback so what I'm going to do just
maybe I don't know something weird was
going on with the mouse press
interaction for so let me see if this
works here by just making it based on
the spacebar I mean a global
mousepressed event is never really a
good idea anyway so so let me add some
cat examples I'll just step out a frame
to add some dog examples and now I hit
the spacebar okay
it's a cat okay so I don't know what was
going on with that weird mouse press
thing let me step out of frame dog there
we go yay I don't know if you people can
see the bottom of the console here but
by the way what's useful about this I'm
going to click in here is well
by the way it also will even when it
says the label that it is cat that's
giving you the answer that you want but
it also is giving will give you its
confidence is bilingual in this case
there's only two categories but data is
so simple it's there's very few examples
so it really like this just means it
matched it matched the image the new
image 100% to only cat images right we
can see confidence is by label no so I
think it's just like the sort of like
smallness of the data set here okay now
okay so let's let's add a few things to
this to make this a little better what
I'm going to do here is I'm going to
create a paragraph element a called
label P label paragraph label paragraph
equals create P no training data yet so
now when I run this it's going to say
here let me make that larger just so
it's label P style I don't know who
knows CSS in this room font size don't
eat you let's see if that's right there
we go so I'm just putting a paragraph
element on the page that says no
training data yet because there's no
training do you would prefer Helvetica
okay I'm gonna go with excitement
suggestions here Helvetica
oh that's beautiful now thank you
okay okay so now what I want to do is
the first time the first time it has an
example we're trying to think about this
so one of the things you can get let me
just show you something here
this by the way unfortunately the
console in the p5 web editor isn't
interactive so I'm going to actually
open up the chrome console for a second
just to show you just to test out an
idea and I have to switch it to I have
to also go here to switch it to like
canvas frame so that I can use so if I
can look at the KNN variable and I could
say KN um classes it's not in Canon it's
in the feature extractor Nam classes
yeah I think we should change this
because the classes the categories
aren't really part of part of the
feature extractor oh no no no no KN hold
on class example count that's what I'm
looking for sorry I basically just want
to find out what variable class example
count where was that
you know that's weird
let's go along here but hold on let me
look at my let me look at my example
because this is the thing that's in the
example num get numb labels that's what
I'm looking for
there's a function yeah okay so what I'm
looking for here did I close that by
accident I certainly did and go back
what I want to do is what I want to say
is I'm gonna create a variable I'm gonna
create a variable star training start
training or like I've tried to think of
a name for this variable but it
basically what I need to do is I don't
want to start guessing until I've
trained it on the first class so I'm
going to basically say start it I'm just
gonna say started is false so here in
key press the first time I'm gonna say
if K and n get number get numb labels is
not equal to 0 and you haven't started
already then start it equals true and
you can start classifying so this is a
little awkward
I'm sure somebody here can think of a
better way a different way of doing this
but basically I want to start this
process of classifying and I want to
classify continuously so as long as
there's at least one label meaning I've
at least given it a cat or a dog
and I haven't already started then I
want to say that I've started this is
just the first time I classify and so
why am I getting why am I seeing errors
here unmatched oh right like I lost uh
end bracket by accident okay
so now let's make sure this works no
training data yet so I could press a lot
of keys but if I don't prep but until I
press the C I have to click over here
first compress a lot of keys until I
press the C then you can see it
classified it as a cap right so we added
the image and then it immediately
started classifying but just did it once
and what I want to do is when it gets
the result I want to say the result dot
label label P dot HTML results dot label
I want to put that I want to get the
actual label and put it right there
where it says no training data yet and
that HTML function will say take that
label and add it to the content of the
label paragraph so let's just do this
one more time no training day I'm gonna
press D click over here and press D and
it put dog there because it thinks it's
only got one training image no matter
what it looks at it's always gonna be a
dog right so now here's the magic thing
what I can then do is once it's done
that I can have it do it again
so this is like recursive in the sense
that it's gonna call classify which will
make this callback happen it'll get the
results and then immediately call
classify again and make the callback
happen again so this is just gonna and
now we might see something slow down or
happen you might see some problems with
this I don't know yet try to do it this
way I know what the example does
sometimes it's nice to put a little like
breathing room so I can't wait a couple
of milliseconds and then call it but
let's just see how this works so now no
training data yet and I'm gonna I'm
gonna start with C for a cat it's a cat
it's a cat it's a cat now I'm gonna give
it some more examples of cats I'm gonna
move out of frame give it some more
examples of dogs and now it should be
there we go
continuously continuously guessing this
worked all right there's a lot of code
here now I'm just gonna pause for a
second so you might have questions or
people might want to see certain parts
tell me to scroll to a certain spot
there's a certain spot need to see and
again this is not the though way of
doing this this is my like let me just
in the short workshop time that we have
today kind of like patch a bunch of
these like pieces of functionality
together the the actual example is a bit
more thoughtful has some buttons it has
some comments in the codes variable
naming but this is the idea right the
idea is I just need to give it and by
the way I could create like ten
different categories now these
categories could be things that the user
makes up like I could actually put a
text box here and with a button and have
the user enter in a label and hit submit
so if you can have an infinite amount of
labels I'm using by the way the word
label category and class interchangeably
the technical term is probably
class 4a classification but I like the
word label English is a little
friendlier I know what you would say
French just that's just the word
category bad French accent okay any
questions okay I definitely want to show
about saving see so let's a whoops let's
let me add stop here let me I'll add
another key press s because this is the
best way to interact with this and I'm
it's called kmn save cats and I'm just
gonna cats dogs dogs so nice thing is
saving is literally as simple as just
calling this function called fate
so let's we're gonna train our super
awesome model here Cat Cat Cat Cat Cat
Cat Cat oh no sorry to click over here
first cut cut cut cut cut cut got that
got that dog dog dog hug ok is it
working I'm a cat I'm the dog and the
cat now I'm gonna press s look at this
automatically to the downloads is a file
called cats dogs JSON so you'll notice
here that I didn't specify the dot JSON
extension but it is adding it to the
file automatically if you're not
familiar JSON JSON is a file format
JavaScript object notation is a way of
storing data with JavaScript syntax
spacer plate so I am now going to open
that file up let's open it in like
visual studio code so we can look at it
just
so we can see like look what is actually
in this photo it's all of those numbers
this is what a K&N model is it's just a
list of all of the examples all of the
examples and there are a thousand
numbers or associations if I save here
another another nice way of looking at
this actually is she just look at it in
the browser cuz my browser has should
have no it doesn't have a hold on this
is worth doing for sure even installed I
don't know I don't know why I don't have
a surprise that I don't already have
this extension
oh yeah maybe yeah anyway but you can
see we don't me I just was gonna try to
show you them but this this it's sort of
crazy you would think like this is what
it's actually doing it's just a file
with all the numbers of it that's
exactly what it is and but if you think
about it we deal with files with lots of
numbers we already talked about earlier
how many numbers are part of an image
right and so right and we can see here
that a by the way ignore this other file
that happens to be on my computer it's
it's not even a megabyte it's 969
kilobytes just this one file so this is
something that's quite reasonable for it
to do all right now what if what I
wanted to do is I wanted to load that
model when the program starts
so I'm good yes so images could easily
an image file could easily have
trillions and numbers in it because it's
that many pixels each with in RGB so
actually just having like 100 example
images each with a thousand numbers
right right alright so there's one last
step here which is to load the json file
i hear people like talking amongst
themselves which is awesome this means
like you've already moved on to maybe
trying something alright but but but
does anybody have any questions are
really stuck like i can't believe that
everybody has this working perfectly but
the nice thing is you can you know I
will put this URL or alko will do it
before I get to it you can actually just
go in addition to the actual example the
full example this URL is also one that
you go to just get this code immediately
okay so the next thing that I want to do
is I'm gonna say K and N Load so what I
could do is in the setup function right
what I could actually do is have it load
that file so but that file needs to be
part of my p5 sketch so I'm gonna go
here under add file then I'm gonna go
whoops then I'm going to grab cats dogs
JSON and upload it so you can see now my
project has sketched out JS which is my
JavaScript code in the excitation which
is the HTML code it has that version the
ml 5 library and then now it has cats
dogs JSON so what I could do now also
when the program starts is have it load
cats oh I don't know what the I think I
supposed to do it like this cats dogs
JSON I always forget I think I'm in
JavaScript the standard way to say
something is in the local directory
is this we're gonna find out if this
works then I could give that a call back
I'm just gonna say KNN loaded and I'm
gonna another function function cane and
loaded and then here also by the way I
could say now cuz I started is true and
I could start classifying immediately I
know I that's what I always thought too
and then there's some weird that's let's
see if it works this way I want it to
work this way let's try it this way so
I'm pretty sure oops
I know why it just went away cats dogs
json k and n loaded and then i'm gonna
have that again okay and then loaded i
this bothers me that it's capital so i'm
gonna make it lower case for no reason
other than my own neuroticism
and then i also could say here started
equals true and get the classification
process started by calling classify
alright let's see let's see if it loads
up Oh guess what
interesting okay first of all let's
double check to make sure that load is
actually the right it might be right
also I might have to wait for a mobile
net to load before I can load it but
let's let's see I shouldn't have to do
let's look at this example code just to
see because it's
load my K&N is the name of the function
in the example it does have it here
update yeah so let's try that so I think
this is a little bit of weirdness that I
think we have to iron out cats dogs JSON
let's try this oh whoops
why is this year that's bad this was
supposed to go here no wonder I I put
this in the wrong place okay okay ready
ah there's been a problem loading the
file that's good sign I mean it's not a
good sign but okay I wonder if oh whoops
no no no I need to get I need to call
infer forgot that I by the way usually
if I'm like live-streaming
once I get past the two-hour mark my
brain sort of shut down and just
continuously make mistakes and
apparently I think that I mean I
probably don't like a half an hour but
definitely hitting it now okay so I need
the features I need to classify
something so and I should probably put
that in its own function like start
classification but let's let's try this
here we go
there we go cat dodge okay so now I've
loaded the month this is by the way now
he's finished completely finished
version of what I wanted to show you
let's review
let's I would have just like you're
probably trying to catch up and get this
word you all by the way I actually
something's really important to show you
so okay so what are the steps okay hold
on one second sign okay the steps are
load mobile net that's the first step
the second step is create K&N classifier
okay
that's here now there is an optional
third step which is load previous
training data cuz I can continue to
train by the way is these stuff these
aren't mutually exclusive
I could start with no training images
and add training images or I could load
training data and add more to do but so
this is really like optional I put it in
parentheses which is load the previous
so that's what we're seeing here so let
me put comments in here right this is
really I mean I'm skipping create video
but load mobile net this is step 1 step
2 create K&N classifier and then this is
optional load pre load existing or like
existing training data optional
okay so then the other steps in no
particular order our ad example
this adds new training data so that's
what's happening down here add new
training data and then and that is so
this step for now if you stop and start
again unless you file and so by the way
this is the awkwardness of saving the
file having to go to your downloads
directory and then having a copy and
paste it back uploaded back ep5 web
editor is is a problem that exists only
because we are doing client-side
programming only but if we were writing
say like a node.js server or wrote our
own server we could have the file save
as the programs running and reload
automatically but with just clients I'm
not writing our own server code we can't
manage the file system for user
and where the sort of where the creator
of the coop I sketch and the user but so
that our only option is to is to to run
all the training downloaded to our
Downloads directory and then manually
copy the file into the sketch but that's
perfectly reasonable if what you want to
do is like calibrate a model for an
installation you just you might what I
might recommend is actually two separate
schedules one that's like doing all the
training and saving the file on another
sketch that just like runs the model or
unless you drink something that's but
right right oh you could think yes
there's no reason why what's in that
model that JSON file is just did I could
easily go to like a database as a
service like firebase and there's so
many what you can put it in a Google
sheet if you wanted to there's so many
ways you can save the K&M do okay so
what I want to do want to show you here
is let's see if this is actually still
working
cat dog oh let me finish shouting the
comments sorry
so forest training data and then five is
classified which is here and also here
and that's the last step now I want to
show you something
let's start oh let's run this and cat
dog it's still working I'm gonna swivel
this around maybe nobody wants to be on
camera
I'll swivel it this way okay I'll just
swivel it over here okay now if I go out
of the frame actually getting it it so
amazingly this is still working but what
I wanted to point out to you is that
interesting it's like one of the things
that's easy to what I'm attempting to
point out to you now is that I basically
could break it by turning moving what
the cameras seeing is the background
because what you have to remember is
it's not just it's not actually learning
what I look like in particular it's just
learning what I look like and what the
background looks like and then comparing
new images to either of those so if I
have a new background then it might
suddenly the results are going to be
different now that the thing is the
mission the thing is it actually is kind
of still working anyway because it's
always trying to make a best guess and
guess what me standing in front of this
background is more similar to just a
background that you know same thing with
this here so it's similar enough you
know was trained with about this behind
me
yeah and we could look at the the
confidence - yeah we could put that we
could output that now would be a useful
thing to output but this is something
that's important to remember that you
know if you start trying to train it on
objects like the what the background is
also plays a pretty significant role the
other thing is mobile net like if if I
were to let's let's actually let me not
load the I want to show you one more
thing I'm not going to load the existing
one and I'm going to do something yeah I
really do
okay so good erase this
[Music]
no you'll you'll see what I'm gonna do
here we go I'm gonna draw a cat okay
and now this is my cat I'm giving it a
bunch of yeah I should move it a little
bit I should move this a little bit this
is a very flawed demonstration now I'm
gonna erase this and I'm going to draw
oh boy yeah okay wait for assignment
this by the way I what I'm trying to
show you something that's not gonna work
at all
that's pretty good here Simon
I will finish your your dear dog for you
so just tell just to make it somewhat
different this is probably like looking
more like a monkey or something but this
is going to be the dog
so now all right look you can see it's
like
let's start with the cat right that the
point is this not you know that one of
these is only two it's got a 50% chance
of getting it right I think I gave it a
lot more cat images still a cat but the
point is that's like a weird frog the
point of what I'm saying is mobile net
because we're doing what I'm trying what
I'm the point of that took way too long
for the point I'm trying to demonstrate
but the point that I'm trying to
demonstrate is that mobile net was
trained on photographic imagery it
wasn't so it actually just thinks every
drawing is basically a drawing now it
probably thinks it's a webpage drawing I
don't know what what what the features
it's not gonna be able to extract unique
features for different line drawings
those all look like the same category to
it so that's not gonna work
but photographic imagery that's full of
color and shape and like things that we
see in the real world it's gonna be able
to somewhat distinguish that in a way
that's meaningful so if you wanted you
know there's a bunch of projects like
gene Kogan's doodle classifier is a you
know like this is not a magical solution
that works for everything and so this
particular project which recognizes
drawings is not using transfer learning
at Canon and mobile net it's actually
training an entire model of some
scratch-off of a massive database of
drawings I've used these a lot if you
have seen the videos the quick draw
dataset so that's what that's one thing
that I wanted to mention is that like we
are we are we have the our ability to do
this and let me let me load it back to
something that was kind of like working
our ability to have this work is limited
by the kind of data that I show it and
I'll show you another just to show you
the the let's just to go to this
particularly this the example that you
need me
let's try to actually I think
rock-paper-scissors won't actually work
super well so let's try this so I'm
gonna put my hand this is rock I'm gonna
give it a lot of rock examples I'm gonna
move my hand all the way around to still
like give it lots of examples now paper
try to make this fairly distinguished
paper and then scissors start predicting
okay Rock you can see that it's getting
it wrong a lot of times paper Wow really
got it it's really good at doing paper
scissors so maybe I yeah maybe I need
more training data but the thing is the
quality of these images is kind of the
same it's sort of like you know the
dural network ends up seeing like these
this blob of a person with their hands
somewhere kind of thick I mean it's not
it doesn't think in those terms it's all
just numbers but the difference what
those logits are what the features are
between this and this are probably not
super significant so it's good yeah
yeah yes yes it's fine - fine that's
actually that so that would be a fun
project to make would actually be like a
thing that just plays against people
always always wins and certainly this
would probably fail if I were to move
the fail it'd be much worse if I remove
the computer over to there and have
people with different hand sizes and
skin colors work with the system so
there's so many inherent limitations to
what we think of as like you know robots
are gonna take over and we'll all beat
them and we've got a ways to go because
it really is can't play it can't beat us
at rock-paper-scissors okay I'm gonna go
back to you I mean I'm gonna try to feel
somewhat successful by going back to
this example and stepping out see it's a
volume
it turns out
[Music]
all right cute yes 6 5 256 256 the 5th
power
that's all right
all right so that's actually right
because there's pixels and nurse how
many pixels are there 256 oh that's the
total number of
if this is another well then actually I
I just got a raise this to the power no
this is just times yeah right so
basically they're pixels there but then
if you that's multiplied by another 256
squared because right because I can't we
did that Simon will you make will you
make a video on this when you get back
home
that's the size it's actually really
phenomenal because we don't think about
the scale of the possibility of we you
know we work with images so often and if
this seems so sort of like crazy how
many possibilities are our boxes
the scale of these numbers are just
unimaginable I can't even like think in
those sizes like if we were actually
writing that number out yeah you would
need you need like a fancies calculator
right much smaller that's the world of
every possible image that you could ever
make in the entire universe it's just
one of those is a much smaller
the size of the image there's me about
this many bits it's actually much more
why because actually there is it's
something called a compression outward
yeah the idea is that yeah me being that
one pixels
or berries kind of similar to and so and
so you only really have some core color
right and so basically and so that
compresses the image right and in a way
the feature extractor is something like
a compressor because it's taking the
larger image and converting it down into
a thousand numbers and in fact you know
JPEG of all these JPEG compression other
compression on recension to like find
smaller amounts but then when you go to
redisplay the image
it's decompressing it so you can see all
those pixels you you wouldn't really be
able to can't the neural network doesn't
really work so nicely in Reverse but
when you see all of those kinds of
things that people are publishing about
like neural networks that are so
dreaming or generating images in a way
that is the backwards process so
thing that we started with if you take
you take the full image when you boil it
down to a thousand numbers
well could you run that algorithm in
Reverse start with some random amount
random set of numbers and get and get
the image out of that and that's the
best it's like a decompression and
that's what when you see a look of
knobbies fancy for fancy videos that
people are neural networks are dreaming
and walking through the Big Dance paid
space we've seen yourself a big Dan
model types of general networks I mean I
skipped up a hundred thousand pound
steps in between what we're doing today
and that
well we're seeing you know do I you know
I think we could we could probably go on
with this discussion for a very very
long time because it's an interesting
question like are we seeing because but
but in essence we have to view all of
them we are you're right that we are
seeing the compressed image but there's
still a full pixel being drawn for every
location but but but but it is a good
point and I think we could argue it for
a long time right because it can't find
[Music]
you know most machine learning about
something I was working with recently is
called the universal sentence encoder
when you hear that what is that what
could that possibly mean
well the universal sense encoder is a
particular machine learning model that
takes a string of text and converts that
into numbers so you could actually
figure out what keeps trying to boil
down the essence of what that text is
into a set of numbers and so then you
could do things like say like oh are
these two sentences similar because you
can always measure the similarity of
numbers by distance by distance in some
dimensional space so this is this
process is you're absolutely right music
sound images text anything that's data
that we can that we can turn into data
that we can turn into numbers we can
pass it through a machine learning
system to detect features this again
this isn't magic the universal sense
encoder only exists because you know
some researchers most likely at Google
happen to be able to train this model on
some massive amount of text
yes yes right
oh the summary exactly yeah
you mention things we passed you yes so
this is a hidden really now there are
there are countless variations on all of
these different techniques and
algorithms but at its core this is the
idea of a shape if you if you start to
read into it like more of a
documentation attention flow that day
asset tend to flower other machine
learning models you'll you'll see this
idea of a shape so for example image
classification we're taking this image
the webcam image is 320 by 240 and
sending it into mobile net well I don't
remember what it is exactly but mobile
net actually only accepts input data
that is I can't remember with its but
it's some weird dimension like this to
24 by 24 so I don't remember whether
it's p ml 5 or 10 to flow digest but
somewhere before it's actually passed in
there that images video size so the
other way a sort of standard typical
neural network system works and this is
different than a neural network that's
recurrent that's designed for sequential
data which is a different discussion is
the inputs are fixed so weird things
happen like with with that like a
sentence like a sentence is some
arbitrary number of words so how is that
boiled down into a fixed amount well
read the paper let me know how they did
that but it's typically done there's
like each word is probably looked up in
a dictionary and then assigned some
other number that's then repackaged in
another array there's you know elaborate
techniques for like fitting the data
into
particular sub dimensions but that is
different than if you've heard of
something this is also an ml five it's
called a recurrent neural network which
is a neural network that's trained on
sequences or can generate sequences
that's really useful for data that is a
of not fixed dimension what I mean by
that is just I mean this is you know
certainly merits more than two minutes
of discussion but if we have the image
classification example right where it
has some fixed number of inputs one two
three four five six seven eight so this
is like a four by two image you don't
eat pixel values that goes in and gives
us you know cat or dog so any any
arbitrary image would have to be resized
to this very tiny dimension before going
into this particular system the current
network a classic example is like you
know a lot of comma
my name is the way I would curl now
recurrent neural network works is it
feeds in one character at a time so H
the input is a single character and the
output the idea is trying to guess the
next element of the sequence and then
that is fed back in recursively so it's
able to do something one element at a
time so you can basically train it by
saying like hey I'm going to give you an
H you're supposed to get an e mixed it's
going to make a guess which is probably
to be incorrect it's gonna tweak its
dials and then it's gonna feed the e
back in and then it's gonna say okay
one minute you're supposed to get an L
next so it's gonna train itself based on
the sequence and then it can generate
things one at a time and so examples of
this that I've done have videos about a
site recently like a model sketch RNN
cuz a drawing is a sequence so it can
try to predict where you're gonna go
next based on the drawing path so an
infinite amount of possibilities here
yes
yes yes so this is a great question and
so you the short answer is no who get in
the sense that ml5 is dependent entirely
dependent on tensorflow j/s and the work
that the tent villages team did is very
to make it work is very specific to
javascript the browser and WebGL so
what's kind of missing from this
discussion is all of this can happen now
in the browser at very efficiently
because of WebGL which is using your
computer's graphics card to do all these
matrix math calculations that you know
graphics cards were designed to do all
those calculations to surrender graphics
but turns out those same kind of
calculations the field of linear algebra
matrix math that drives a lot of
computer graphics also drives these
machine learning algorithms so their
implementation is entirely an ml 5 is
entirely like focused on JavaScript and
WebGL that said it would be technically
possible to make a port of ml 5 to
processing by building on top of Java
bindings for reg regular tensorflow so
the original tensorflow library is is is
C++ as written in C++ most people
interact with it through a language
called Python because that's you know
the sort of language of the data science
world and so you're kind of you can
control tensor flow from Python
basically execute tensor flow machine
learning
operations from Python you can also do
those from Java and pretty sure it's
just this is like not a thing that a lot
of people know how to do Java bindings
tensorflow and look at this install
tensorflow for java so in theory you
could make you could make a you could
sort of wrap tensor flow and put some
niceties around it in processing the
same way that but no one has done that
anybody if anybody wants to do that
something a message on Twitter or
something I'll be there I'm just gonna
like cheer the whole time you know it's
it's there more you know it's it's a
weird sort of thing cuz I don't I don't
I don't have a good world I don't have I
think an accurate worldview in the sense
that if you I think people are using
Java and Java is used in industry and in
education a huge amount but in the world
that I kind of walked through this
creative coding education art school
there's so many more people now invested
in learning JavaScript than contributing
to JavaScript so I think it's it's
harder to find contributors to
processing Java than it is to JavaScript
libraries these days from but I don't
know if that's really true in a sort
global sense or just in the sort of
small little circles that I operated I'm
not reaching out to the right people or
communities other questions yes Simon
one more question from Simon or is this
more you are you're gonna talk to us
about how many pixels are any image
again because this is writer you really
should make a video about that so let me
pause you for a second because I want I
want to hear about this but I think also
we're reaching the time where maybe if
people want if we want to like break out
and like just hang out here for a little
bit trying to work with these examples
and we can also have like some side
discussions so hold that saw for like
one more minute can you do that Simon
okay so what what I want to do is let so
let me just stop here with this let me
let me stop here with this example so
I'm going to if it isn't already what
I'll do in a moment is I'll take this
URL and put it as a link right here
maybe it's here already working progress
sketch look it's there already and so
you know I'm certainly I'm gonna be
hanging out here for at least another
hour I don't I don't know
you know this was like a a free event I
don't I don't know what time's with the
tank closes down but you ought you're
all obviously welcome to lead held
captive here um but what I would be
interesting to me is if people can take
this example and play with it whether
that's just like trying to train their
own model with some objects you find
running around whether you want to hook
it into something else you've already
been working on people want to hang out
here for another hour or so and work on
stuff I can float around and answer some
questions if you didn't get a sticker
stickers I know there's some treats that
home-baked and also some that care about
from a vegan place nearby
so so we can just hang out here for
another hour and work on stuff okay
so I'm for that purpose I'm gonna shut
down the live stream and then Simon I'm
gonna start with you to hear about this
thing that you want to tell me but first
let me shut down the livestream and then
so and then what is the we'll make this
an official moment how do I do this okay
I'm gonna go here yeah we're like
they're there let's check in on the
livestream for a second okay uh Java
seems good okay okay stop streaming okay
this has been longer than an hour and a
half that time is not accurate okay all
right I'm feeling very nervous about
clicking this button okay goodbye
everybody watching this live if we make
maybe what we'll do also before I get
kick stop there and what we'll do is if
people make stuff from today maybe we
can come up with a way of aggregating
that and I know certainly that just
having people add links to things you
made from that github readme would be a
start but certainly if people have other
ideas hashtag the tank the coding tank
is that like a thing like a tank is like
a train how do you think how do you say
never mind okay I'll get my French
lessons later okay I mean it stop
streaming thank you