[Music]
good morning so first of all before I
start I really want to thank everybody
involved in making Jesco aphasia happen
for having me and for really organizing
a really cool event so far and thank all
of you for being here this morning it
might be a bit early for some of you so
thanks for coming here to listen to me
talk about machine learning for
front-end developers so before I stop
I've given this talk a couple of times
before and the feedback or comment I
usually get is that I speak very fast so
I'm gonna try to be mindful and slow
down a little bit on some slides but I
get really excited about sharing what I
know with you so my natural pace always
comes back and also I mean I'd like to
remind you that we're in a fast-paced
industry so I'm not you know I'm not
fast you're slow so I'm kidding I
already want to try but um so my name is
Charlie Jerrod I am a developer at a
product company called Atlassian in
Sydney and outside of that in the
community I'm also part of two tech
community programs called Google
developer experts and Mozilla Tech
speakers if you're interested in
learning more about what it is I know
that there's a few others here as well
so there's you know quite a few
community of people sharing knowledge
through conferences and blog posts and
workshops and things like that but I'm
not here to talk about that I'm here to
talk about machine learning so usually I
start because I don't know what you know
and what you don't know so I usually
start with an example to try to put
everybody on the same level about what
machine learning is but yesterday I
think Monica did a very nice
introduction
talking about how to recognize our
pictures of cats and dogs and I thought
that was really simple and well done so
I kind of removed some of my slides and
I'm just gonna briefly say if you were
not here or if you you know don't
remember that is basically giving the
ability to computers to find patterns in
in data without being explicitly
programmed so if I just take back the
example from Monica yesterday the
time that your computer sees a picture
of a cat it doesn't really know what it
is it's never seen one before but you
know by giving it millions of samples of
data of what a cat looks like in a dog
looks like using algorithms and machine
learning models the computer is able to
understand to find patterns in what
makes a cat a cat on a picture and what
makes a dog a dog so at a high level
this is basically that so at the core of
a is machine learning models and to
build a model you need algorithms so
let's dive a little bit into this so if
you start looking into machine learning
you might come across some of these
algorithms so I know that I when I
started I definitely heard a lot about
naive Bayes and K nearest neighbor and
there's really a lot of them and they're
good at solving different types of
problems and dealing with different
types of data so before even looking
into what algorithm you want to use you
have to understand what type of problem
that you want to solve so there's three
main types of problem problems in
machine learning you have supervised
learning unsupervised learning and
reinforcement learning for this talk and
knowing that I only have half an hour
I'm only gonna cover supervised and
unsupervised also because a lot of the
time when you get started the problems
that you'll want to try and solve can be
solved with supervised and unsupervised
so at first maybe don't bother with
understanding what reinforcement is so
let's start with supervised learning if
we start with a Britisher definition
it's basically creating a predictive
model based on a set of features and
labels and there's three main terms in
this definition that you need to
understand it's you know what is the
predictive model
what's features and what's a label so I
don't know about you but I learned more
when I can kind of like relate to
examples so a classic example of a
supervised learning problem is
predicting the price of a house so let's
imagine you know we're in Singapore we
have a house if we're lucky enough that
we want to be able to sell and but we
don't know how much we should sell it
for for it to you know for us to make
profit and you know sell it fast and
things like that but what we have is a
lot of data of all the houses in
Singapore and that's what we have so we
start with that so
how can we predict the price of our
house based on the price of houses in
the market so the labels in that case is
the price it's what you're actually
trying to predict so in your data set
that you're going to put together it's
how you would classify each entry so as
our problem is trying to predict the
price we would have if you you know
imagine an Excel spreadsheet we would
have a column with all the prices of all
the houses in Singapore and then we
would have our features and features are
the characteristics of the entries in
your data set so for houses it would be
the number of floors the number of rooms
bathrooms that you'd have a garden or
know the neighborhood and things like
that and using these labels and features
we're using an algorithm you would try
to understand the correlation between
the features and the price so the
characteristic and the price of a house
to generate a model that can then and
being a mathematical representation of
the outcome of your training
so using everything that you know about
the house is you use algorithms that end
up being like a function that takes new
parameters of a new house it's never
seen before and it's able to generate a
prediction of a price that the house
should go for based on all the houses
that it's seen on the market so it's
important to know that supervised
learning needs labels and features to
generate a predictive model and then if
we look at a type another type of
learning the unsupervised learning it's
the ability to create predictions based
only on a set of features and the end of
the definition is really important
because it's the main difference between
unsupervised and supervised so if we
think about our problem of predicting
the price of a house it wouldn't really
work as an it's not an unsupervised
problem how can you understand how can
you predict the price of a house if you
have never even seen what a price is you
only know the characteristics of a house
so in that particular case a type of
problem that unsupervised learning
solves is more around clustering so an
example of that would be predicting
customer behavior so when you go to the
supermarket they might not know who you
are at all but based on what you buy how
often how expensive it is where they can
actually really cluster people together
into different categories and base
on your buying behavior they wouldn't be
able to know what advertising to send to
you because you're more likely to buy
the same thing that people like you have
bought as well so so unsupervised
learning is more around clustering
rather than getting a particular answer
out of a question so in this - so now
that we talked about the types of
learning I talked about you know
problems that you're predicting the
price of a house or customer behavior
but it might not be something that you
want to do and yesterday in Monica's
talk we saw that machine learning can be
used in a lot of more creative ways so
let's talk briefly about different
applications I just put like a short
list together and the first one I'm
going to talk about is this example here
where you upload a picture of a cat and
then an algorithm generates for you a
piece of text that you can use as alt
text for your image tags in your HTML so
this experiment was built by I'm sorry
Dresner I put the link there if you want
to try it a bit later but so maybe for
one image my Billa oh I can do it myself
but when you like you can see that using
machine learning can maybe eventually
automate some of the things that we do
as developers so we can actually spend
time solving some of the logic that we
want to solve so that's just like a
quick example another one that would
have been really useful for me a few
years ago was it's this framework called
not-safe-for-work j/s where again you
upload a picture in in this little
experiment here and it gives you the
probability of it being an image that is
not safe for work and I used to work in
advertising and when you work for brands
sometimes they want to put competitions
together where they want to engage with
customers so people have to upload a
selfie or whatever to win a contest so
people who produce a lot of images and
most of the time they the brain wants to
see the images live in a gallery or
whatever and the thing is that the check
that you have to do to make sure that
these images are okay they usually done
manually either by like a product
manager or or by the developers so using
machine learning in the front end you
could actually do a pre-check of what
people are uploading before saving into
the database and before displaying it
live
on a page so that's definitely something
I could have used a few years ago so I
thought it was very interesting and then
like another example that is really
really like totally like awesome
I got really loved it because it mixes a
lot of different technologies and it can
actually help people in terms of
accessibility so I don't know if you've
seen this one but this developer called
Abhishek Singh created a prototype
around the concept of kind of like
future interactions so he was thinking
about if no voice UI when you talk to
Amazon Alexa what about people who can't
talk are they are we just gonna
literally cut them out of you know
innovation and things like that
so what he decided to do is to use
machine learning in JavaScript to look
at the to capture data from the webcam
feed train and algorithm to recognize
and gestures map them to words so he
could actually communicate with Alexa
just via via gestures like by training
and algorithm so I thought it was really
a creative way and a really interesting
way of using machine learning in
JavaScript so let's dive a little bit
into instead of just talking about what
you can do let's have a look a little
bit about first of all why because when
you think about machine learning
you probably think about Python before
you think about JavaScript but I think
it's really interesting to now be able
to do that in JavaScript because in
terms of learning curve it's a lot
easier for developers to know that they
only have to understand a few concepts
of machine learning and and understand
the syntax of a certain framework rather
than having to learn an entirely new
language we learn new syntax like every
day when we move from jQuery to angular
to view to react so we kind of used to
doing that and now it's just
transferring that skill of being you
know adaptable to another domain so I
really love the fact that is you don't
have to be a data scientist you don't
have to only know Python
it's actually out there and you can
start doing it now and now the point
that I think is important is around
rapid prototyping so I really love doing
that it's you know like this what I'm
talking about is has nothing to do with
what I do for work so all of my
experiments are things that I do outside
you know after work you know yeah
or on the weekend so sometimes when I
want to validate an idea I need it to be
fast because I don't want to spend like
I'm not that patient so I don't want to
spend six months on a project I usually
want span like two hours so being able
to to prototype something fast in
JavaScript is really cool because if you
want to just you know if you have an
idea in mind and often if you want to be
given the time or the money at work to
explore it more you have to convince
people that it's worth it and to be able
to convince them usually you have to
show them something some kind of MVP or
approve of concept and and your being
able to do machine learning in
JavaScript means that you could validate
an idea and then eventually move to
Python if you want to push it further
but I think that's one of the really
awesome thing that I like about
JavaScript and then finally there is a
bigger and bigger ecosystem of tools
that you can use in JavaScript and
there's more and more documentation
there's a lot of tutorials and courses
so I think it's a good time to get
started if you want to get into that
space so what can you do so you can do
three things first of all you can import
an existing pre-trained model so it's a
model that has already been trained with
a lot of data for a specific purpose and
you can just import it in your app and
use it as is you can also do you can
also retrain and earn important model
that we call transfer learning so you
kind of do the same thing from the step
one you can you can use a model that is
open source but you can as you you can
add your own samples on top of it and
kind of retrain that ready fast and then
finally you can do all of it in
javascript in the browser you can define
train and run your model entirely in the
browser so in terms of tools I think the
most popular is tensorflow j/s but
sometimes I feel like some of the terms
that tensorflow gest uses might be a bit
confusing for people who is just getting
started so I really like the second one
on the right ml 5gs the it's a framework
that tries to abstract some of the terms
like you know it's trust to make it
really really easy for you to get
started so it's a lot simpler and
there's a few examples I feel like it's
not it's still kind of like in
construction there's some examples and
some algorithms that work but sometimes
when you look at the docs you can see
that it's stealing
yes but definitely have a look and then
if you don't really want to write any
code but you just wanna pings on api's
you can leverage the tools from Amazon
Google and Microsoft and then the em in
the middle is for yes so now if you were
here yesterday you know what that is but
yes so it's like some some frameworks
are actually more specialized about
certain type of data and to do certain
things okay so we are at this part where
I'm going to do some demos and code
because I feel like to get people on
board sometimes you know you can't just
say oh it's easy you kind of have to
hopefully show that it's easy so people
are a bit more comfortable so to get
started I'm gonna talk about the quick
demo that I built around using a pre
trained model so what I did so I I want
to get better at recycling but I always
forget what should go in
what bin so that is not really helpful
when you want to recycle so I wanted to
see if I could use machine learning to
help me do that you know kind of do it
automatically so I'm gonna demo out what
I did in an hour show the code so this
demo is supposed to I kind of built it
usually to be used on a phone so I could
just like move around the desktop like
it's working because it's a website but
let me just all right so I'm gonna start
it's gonna need the camera feed and I
have a rest sticker hiding my camera I
need to buy myself for real okay so if
you've ever alright so I have the camera
feed so you have to imagine that you
have it on the phone and it's not
pointing at you but so I just have like
a button that here that says you know is
this recyclable so it works well with
bottles of water so so I'm going to put
it my microphone down because I need so
it is
[Laughter]
so then on my phone I would click yes it
is a bottle and then it says oh it is
recyclable 3 in the yellow bin so in
Australia it's yellow actually didn't
check here if it was yellow oh well ok
so you can you can start again with
different objects and I'm gonna exit
this because I don't need to see myself
so and and and yeah and so it does the
whole image recognition using machine
learning and then you could imagine that
you could build an API that you know you
can just ping to say hey this object
which bin should it go in and things
like that so in terms of code I used the
kuku SSD open source model that's an
object detection model and then I import
tensorflow J's as well then you load the
model to be able to use it then you get
the camera element from the HTML and you
use some of the tensorflow built-in
methods to look at the pixels of that
camera feed you transform the data so
that it can be in the format that
tensorflow and the model is expecting
and then finally you kind of give that
processed image to the model and you ask
it for predictions it's kind of like
here's the de Graaff from the webcam
feed have you seen this before do you
know what this is and it recognized that
you know my bottle so of course you know
there's a bit more code around like the
UI and buttons and things but just to be
able to recognize images using that
model that's basically just it you load
the model you get the data from the
camera feed and then you transform it a
little bit and you ask the model you
know can you detect this so at the first
thing of importing a pre-training model
that's it so now let's talk about
transfer learning so transfer learning
as I said before is using a pre trained
model by adding your own samples so for
this one I so I like to try and play
around with new technologies and mixing
them with accessibility so I'm just
gonna launch the oh I'm still here okay
so this one is gonna use the camera as
well so say the camera feed is gonna be
really small but there's
for that like you're not supposed to see
it I just showed it so so you would see
the what's going on but okay so I'm
gonna go to the right I'm adding samples
I'm gonna go to the left I'm gonna go
down then I am testing okay down lift
right that's pretty good and then I'm
able to actually unless like you know
what 30 seconds I trained a model to
recognize my hand movement my head
gestures and I can write like why
all right so I really like this is like
my favorite feature of using javascript
for machine learning and in terms of
code so there's a little bit more code
that I can't go too much into it because
I'm looking at the time and I think I'm
gonna be over time but I'm loading
another kind of like image recognition
model called mobile net I'm again
loading tensorflow GS but I'm also
adding a classifier because I need to
kind of reclassify my my personal
samples with the model then you just
declare some variables I have four
classes that I'm trying to classify
because I had four gestures but it could
be you know the number that you want the
image size of 227 that is hard-coded I
started that from an example I think
it's because the bigger the image the
longer it's going to take to train so I
think if you want it to be fast then 27
is fine the top K is the K value for K
nearest neighbor that is kind of
arbitrary as well you can change it I
can't go too much into what the
algorithm actually does but this value
is needed but it doesn't have to be ten
but then so the more interesting part is
we create the classifier you load your
module then you do the same thing as
before you look at the data coming from
the camera feed but instead of giving it
straight to the model you kind of have
to like I did with what's going on with
the model that you loaded so I'm not
exactly sure exactly what the infamous
that does before what I understand it
kind of merges your image data from the
browser
to the mobile net module and then it
adds all of this to the K nearest
neighbor classifier that you are created
before and then you're able to predict
the class of your new samples based on
this merge of the model and your
personal samples so then you kind of get
a prediction of your current position in
my personal sample like type and yeah
that's it so that was a very poor
explanation but in terms of steps you
create a classifier you load the module
you look again at the data coming from
the webcam you kind of do the merge and
you ask your model to predict and then
finally you can dispose of the images
that you got from your webcam because
you kind of already retrained the
classifier so you don't need to keep it
so that was for transfer learning and
then finally doing everything in
transcript so for this one this demo I'm
always a little bit nervous because I
don't know how people are gonna take it
but yesterday somebody mentioned pornhub
on stage I think I'm fine
what I built is a wheelie detector so
we're gonna stay on the same level if
you don't know what a wheelie is a
wheelie is a penis but it's not pictures
I'm not crazy
so and also I'm really not interested in
pictures of this so I'm just gonna
launch it quickly I'm just I mean if you
just go stop this it was still going
well there's a bit small but that should
be alright so basically I build
something from scratch that recognized
I'm gonna start with something you know
like safe OOP so it's a Kindle just
think if that's the Kindle and if I
predict it tells me it's not a really is
that yes it's not a wheelie and then I
can clear a and do I need no I don't
need two hands for this so
and Willie okay and what I forgot to say
I forgot to talk about why I built this
because I don't just spend my spare time
drawing release so I built this because
I wanted to build something else where I
wanted to let people collaborate on the
browser you know using WebSockets from
like anywhere in the world but I you
when you let people draw stuff on the
Internet you can be sure that they're
gonna be trying Willy's and I didn't
want to spend my time checking you know
what are people drawing and then
deleting it and stuff like that so
that's my way of using machine learning
to do that for me so at the moment you
know like I have to click to be able to
predict if it's really or not but you
can actually change the code to detect
as somebody is drawing so I mean you
would you wouldn't even have time to
finish the woolie would be gone so I
still haven't been on that project but
I'm still happy with that so now let's
talk about how I put it together okay so
first of all and because when you do
yeah I know and that's the only thing
people remember from the talk so so
because okay so when you build when you
do everything from scratch you usually
have to kind of you have you need data
right so on the right you have the quick
road that I said that Google put
together where they there was a game
online where people could were asked to
draw I don't know a strawberry and a
door and glasses and actually all these
samples were downloaded so they were
actually saved in a database and Google
put this data set open source so you can
use it the thing is obviously Google did
not ask people to draw Willie's but I
needed them for my you know for my
prototype because if it doesn't know
what it really is it can't tell me if
it's one enough so I drew some of them
definitely know as many as the data set
because that's like millions but so I
needed that so I needed to have data
together the thing is that when I drew
my own
but so when I drew it on the canvas on
the in the browser it was 200 by 200
pixels but the images in the data set
were 28 by 28 so for me to be able to
work with them usually have to do some
kind of data processing where you have
you resize so that the whole data set is
the same size of pixels and then finally
finally next you do you split between
the training set and the test set where
it's usually 80% and 20% and the reason
why you do that is because you uses 80%
of your data set to train your models so
it kind of understands is it a wheelie
or is it candle and then your test data
that is already labeled you already know
what each image is you use that to
against your model that you created to
see if the accuracy is right and if it's
not right you know that you have to
retrain but then after you split your
data then you can choose your algorithm
I don't really think there's a purpose
to choose it before but I in this
particular case I used what we call the
CNN convolutional neural network because
from what I read it was with images so
you know I have images then you can do
some parameter tuning you train you get
your prediction and depending on you
know the level of accuracy if you're
happy or not with it you kind of repeat
these steps over and over again you will
rarely get a very good accuracy the
first time parameter tuning is kind of
like weird as well like you yeah you
don't really understand sometimes why
changing a parameter makes the accuracy
better but you kind of have to play
around with it so in terms of code I'm
gonna show like a really small code
sample I try to get like the core bits
of it but what you have to do intensive
flow you have to create tenses which is
basically like the way a way to shape
the data so that tensorflow will
understand how to deal with it and as I
said you have to create a training set
and a test set and then you create your
model so I have a sequential model and I
add layers of a convolutional 2d layer
and then you add different layers as
well I don't have time to really go into
this in depth but also the amount of
layer is it a bit arbitrary some time we
can remove some add some see how it
performs the parameters meaning that I
was talking about is what you see like
kernel size and filters that's a little
bit more of a like an art than a science
you kind of see what works then you get
your training data you do what we call
fittings so you give that training data
in labels to the model you pass it a few
parameters and
and what you do is that you get your
taste data and you test it against the
model so using them the testing data you
give it to the model you get the
prediction and if you get the right
label that you're expecting then it
means that it's fine if you want to look
into this the documentation for
tensorflow J's is actually really good
a few limits very quickly so you need a
large amount of data if you're not using
a pre trained model you know as we
talked about before for a computer to
understand what something is usually
needs millions of samples it can take a
lot of time to train your own model
depending on what you're trying to do
especially if you're I think dealing
with images it can take a lot of time I
remember to think about the mobile
experience because when I was playing
with loading like importing a pre train
model some models can be a few megabytes
and that kind of hurts so if you're on
like a low network it's gonna be really
impossible and even in terms of using
the battery of a mobile device it can be
quite expensive so just you know think
about that if you try to develop
something liability sometimes when you
use machine learning models you actually
don't understand what's going on in the
background so if you decide to use
machine learning to build an application
who decides who gets alone or who gets
into what expensive school or whatever
you have to be sure that if you tell
somebody that they can't get a loan or
that they can't get accepted and they
sue you you can't just say oh we use
machine learning so I don't know you
have to just be able to explain what
happened and most of the time people
don't know well I definitely don't know
and then finally and really opponent Lee
even if you're not interested in coding
machine learning the topic of bias and
ethics in AI is like super interesting
and I just like to mention often the
algorithms are not biased we are I feel
like we love to push the responsibility
on a piece of code and not just like
look at what we do wrong so remember
algorithms is just code and the reason
why we say it's biased is because of the
data that we give it and the data that
we give it is something that we produce
so just a quick example here this was
already a few years ago so I hope they
fixed it now though if you if you use
before Google translates with some
languages are gender neutral and
if you translate some languages it to
English then all of a sudden it becomes
gendered where you have things like he
is hardworking and she's lazy like all
of a sudden his gendered I'm not lazy I
mean you know she's a cook he's an
engineer that's the worst so it's that
kind of stuff it's like and then you had
articles around oh you know a Google
Translate is sexist it's like well what
do you think it's the data from so I
think it's just remember I mean I do
think that machine learning can help us
recognize some of our biases and then
help us you know be better but the
algorithm itself is not the problem okay
I'm getting to the end of my talk so the
story about that one is that so I gave
this talk I'm just looked at the unicorn
so I gave this took a few weeks ago in
Lithuania and a few like just a day
before or a few days before I had
finished the prototype that I was
building that I was like super excited
about that involved machine learning and
JavaScript and I wanted to show the
world but as I showed it well now for
some people it's not a surprise anymore
but um I thought that if I didn't show
it here it would be unfair so I'm gonna
show it as well and and yeah it's a toy
it's supposed to be a topic for a talk
later this year but it is related
because it uses machine learning so I
hope it's gonna work so the inspiration
for this is so this is a prototype built
by a developer called Minko Jewish who
used the webcam feed to train their
algorithm to recognize some gestures to
then apply it to a game - so to be able
to have this personalized experience of
of playing a game the thing I was
thinking though is that well you have to
be in front of the webcam so what if all
of sudden it's dark you can't play
anymore so there was a bit of a shame as
I'm home I think I can do something
about it so I'm gonna try I need to
remember what I have to do I think I
have it ready let me think
I'm gonna have some water if you don't
mind
right hopefully is going to work so
preaching said that we could use the
unicorn so I'm gonna use the
it's me and you now unicorn they can
only be one a rainbow on this stage so
lemme try something I'm the one on the
round in red okay it didn't work okay
wait
[Music]
oh I just like to do that mister
[Music]
[Applause]
[Music]
I love how the music you play okay
[Music]
No okay I'll give it one last try this
is what happens when you get shitty for
the guys in two hours
[Music]
Apple pencil who is using an Apple
pencil
[Music]
you ruined my demo one last time
no I will not get off this stage until
it works I'm sorry for the speaker after
me all right I swear
okay is somebody using bluetooth and
they're like that was working this
[Music]
don't let me down
[Music]
alright let me just
it's not the way to end
[Music]
yes
[Music]
[Applause]
[Music]
this was like one of my worst life ever
all right so just very very briefly I
wish I wish we just like yeah well magic
not so much I I am you know in the
process of writing like a post about how
that works and stuff like that I don't
know if you want to know that because
yes so just quickly a few resources I
can share the slides online later but
here's a few tools and useful links and
I'm Way over time that I said and well
thank you so much for being with me