[Music]
my name is Nikhil so we're here to
talked about machine learning and
JavaScript with tensorflow J's so if
you're familiar at all with machine
learning and data science you'll know
that most of it happens in Python so
this is a poll that was given by a
popular data science blog called Katy
nuggets and you can see that Python
absolutely dominates and this is for a
pretty good reason you know there has
been many many years of tooling built in
Python for data science from numpy to
pandas to scikit-learn to tensorflow
and this field is gonna continue to
evolve in python and i don't have to
convince the folks in this room
javascript is a very very popular
language this is obviously the October's
survey by github and you can see that
JavaScript absolutely dominates so we
think that there's actually a lot of the
JavaScript community and the folks in
this room can bring to machine learning
so who here has seen this this is
tensorflow playground okay a few people
but not a lot of people so I definitely
recommend checking this link out this is
an in-browser visualization of a neural
network it was built by one of our
colleagues at Google and the idea here
is you can immediately change the number
of layers neurons learning rate and so
forth and immediately see how the neural
network generalizes over a data set now
this thing was a huge educational
success you know it's being used in
universities across the world now and in
Google's own machine learning crash
course so again I definitely recommend
checking this out so we step back and we
kind of asked ourselves you know why was
this such a success like why do people
care about this in browser machine
learning thing at all and we kind of
distilled it down into a few points so
the obvious thing is that you just click
a link and you get going if you've done
any Python ml it's like really pain in
the butt to get your drivers installed
Python libraries installing that kind of
thing it's super interactive you know
you have buttons you have hyper
parameters you can play you can
immediately see how the changing the
knobs affect the the the learning
configuration of the model we didn't
take advantage of this in the playground
but you know in the JavaScript world in
the browser you have
cameras and microphones and standardized
access to these things which you don't
really get as much in the Python lab and
super important to us is that you can
actually make predictions locally and
data can stay on the client so this is
privacy-preserving so we took this and
we launched a library called tensorflow
JS we released it last year in March its
GPU accelerated so we use WebGL to make
everything fast so we actually do all
the the linear algebra in fragment
shaders and I'll talk a little bit about
that in a second but one of the things
that we do is we let you make
predictions through machine learning
models but we also let you train them
directly in the browser or even in no
Jess and we'll talk about about that as
well so when designing the library you
know we had a couple goals in mind one
is that we wanted to empower a diverse
group of Java developers like the folks
in this room you know there's a lot of
really awesome people in this community
and we want to sort of marry these two
worlds at the same time we wanted the
folks who are experienced in the machine
learning to be able to port their work
to the web now these goals are kind of
sometimes at conflict so we'll talk a
little bit about how we resolve those
okay so one of the principles that we
had was we wanted the library to be
super easy to use and we kind of leaned
towards that over performance at the
same time we didn't want to sacrifice
any functionality for simplicity so just
jumping into what that means we decided
to go with this eager only approach I'm
not going to go into what that means but
it's a much simpler way of programming
and actually most of the machine
learning world is moving towards this
eager approach versus a graph based
approach where you actually stitch
together a computation graph and execute
it later and we've really wanted to be
easy so we move towards eager we also
provide a high level layers API which is
a set of best practices in the machine
learning community so you don't have to
think about all the details of your
linear algebra when you're constructing
a model and we also provide a whole
repository of pre trained models that
require zero understanding of machine
learning to get started and I'm going to
show you a couple examples of those in a
second and I want to highlight this we
focus on performance when and where it
matters obviously we want matrix
multiplies to be fast what we did was we
took individual models and we figured
out
how to make those individual models
faster on a use case basis so as I said
we don't want to actually make the
library less functional so we support
gradients this is fancy talk for
sensitivity of each of the weights which
means we can train through any operation
that you use in the tensor flow chest
library support a lot of the tensor flow
ops 130 of them and for any of these
models that we're about to show you you
can actually dig down and get some of
the machine learning constructs out of
them if you want to okay so quickly
jumping into what the technical stack
looks like at the very top of the
abstraction api's we have our models
repo so this is set of pre trained
models I'll show you a couple of those
in a second require very little
understanding of machine learning below
that this is our layers API this is
where you can construct a model you can
train the model you can serialize the
model for later and we'll show you some
of that soon too and we also have below
that our core API which is or just
linear algebra kernels so these are
matrix multiplies convolutions and their
gradients which are derivatives so all
of these api is you can sort of poke in
at any of these abstraction layers all
of these sit on top of WebGL in the
browser we use fragment shaders to run
all of our math in parallel and in node
we actually bind with the N API to
tensorflow C++ and what that means is if
you use that same API for any of these
things you immediately get the hardware
acceleration that tensorflow has been
working hard on for the CPU and for GPUs
with CUDA and eventually we're gonna
have TPU support it was also very
important for us not to silo ourselves
in the JavaScript world there is a whole
wealth of models that are trained in the
Python ecosystem that we want to take
advantage of so we have converter tools
that let you take a Karass model or a
tensorflow Saves model and bring them
back into the JavaScript world
ok so let's quickly take a look at what
the models repo looks like so if you
check out this link these are all our
pre trained models these are hosted on
github and on NPM so we host all the
weights and all the JavaScript for you
and we have a wealth of models from
object recognition to
human pose detection - localization -
segmentation - text classification and
the list goes on just go check this link
out but I want to show you one of the
demos because it's fun
so okay so this model is called pose net
it's running completely in the browser
nothing is being sent back to a server
and the idea here is we take RGB images
from webcam we pass it through this pose
detection model that generates key
points for each of you know sort of my
body parts and then it returns an object
that we can just render on the screen
and obviously it works with two people
so this is a lot of fun and we'll show
you how to use one of these models in a
minute the second model is very similar
to pose net it's doing person
segmentation so it's you know this
backgrounds a little funny but basically
what it does is it draws a mask of a one
where it thinks is a human pose and a
zero where it does where it thinks
there's not so this is what this one's a
lot of fun and I don't know if this is
gonna show well here but one of the
effects I really like is the portrait
mode you can see this thing blurring so
we have you know a software-based
portrait mode that's running directly in
the browser pretty fast okay so let's go
back to the slides and I'm going to show
you how that actually works and what the
code looks like so that model is called
body pics it's a pre trained person
segmentation model that we've done a lot
of work to make super fast pretty
straightforward you import tensorflow
jeaious and body pics to libraries we
have them on NPM we host them on CDN for
you we you know have a regular image tag
that's it and this image is Frank now
Frank is Knicks baby and he is doing a
yoga pose for us so we're gonna try to
figure out where Frank is in this image
so first we just load the model we call
a weight body picks out load and this is
gonna download all of our weights these
weights we host on our GCP buckets for
you so you don't have to pay for any of
that and then you just call one line of
code SMA person segmentation on the
image and you get a JSON object out and
inside of that JSON object is a binary
mask of where it thinks the kid is that
is that simple you don't really have to
understand the end all bits of this one
of the other things is modeled
you is parts as well so it'll tell you
which pixels our face which pixels our
arms and legs and so forth and we
provide some fun utilities for drawing
masks on top of those so you can imagine
this being used for like a video game
sprite you just jump around on screen
and you immediately have a fun video
game sprite okay so I don't have to
explain this to people in the room but
JavaScript runs in a ton of tunnel
places and we're working hard to get
tensorflow J's working in those places
so we have the browser a node obviously
but working on electron and react native
and WeChat so we'll talk about those in
a second but I want to show you some of
the cool examples that we we like in
these worlds so in the browser side
hopefully the links are here we have a
project called create ability this is
one that's done by Google and it's a set
of experiments around can we make
interacting with music and art more
accessible so we're using that pose net
model that I showed you and we're
actually able to play a synth with just
our face this runs completely in the
browser the link is there go try this
after the talk please
cool so then we also have a project
called uber manifold not by us by uber
and this project is a way to debug and
understand machine learning models as
their training and I actually use
tensive ojs just for linear algebra so
fast matrix multiplications in the
browser Airbnb is also using tensorflow
J's they ship a little model to the to
the client so when you're about to
upload a profile picture if they see a
license or a government-issued passport
in that in that photo they'll yell at
you before they upload to their server
so they don't have to own that PII on
the backend on the on the desktop and
node there's a project called clinic
doctor and clinic doctor is a project
that monitors your node application for
CPU spikes and they use tensorflow dress
actually to disambiguate garbage
collection spikes from your CPU in your
in your actual program one of my
personal favorites is a project called
magenta studio magenta is a team at
Google that does generative music and
art and they actually have an electron
app that plugs directly into Ableton
Live and it can generate MIDI notes on a
track for you or it can generate a drum
beat alongside maybe a guitar groove
that you have so this is a ton of
on augments and existing work fro our
workflow and you know JavaScript is
awesome so of course we do it there okay
so this other platform called WeChat is
massive in China if people don't know
about it it's got a billion users lots
of mini programs lots of developers and
they all run a Java Script and we're
working hard to get GPU acceleration
stories working inside of that with that
I'm gonna hand it off to Nick to talk
about some other stuff
thanks aquel I speak kind of highlighted
JavaScript runs in a lot of places and
we're starting to think of areas where
we can keep expanding where you can run
tensorflow j/s i want to step back and
talk about our node bindings first
before we dive into the next topic we
launched these about a year ago and the
library is great because it's super fast
it uses that C library like Nickell
mentioned and it's great for deploying
on the servers or doing local workflows
on your desktop or workstation but there
are a few downsides this particular
library we have one of them is the GPU
acceleration requires nvidia cuda
library it's a really fast library but
it's very large and we at tensorflow
don't currently support mac OS so
there's no GPU acceleration on math and
the other thing is the node package
itself is a native module all built on
an API and it links to the C library of
tensorflow that can be really large
depending on which library you're using
CUDA can be around 250 megabytes or so
just on Linux so it's a very large
package to ship so we started to think
is there something in between we can do
on node and we started working really
hard and launched earlier this year a
new headless graphic stack for node and
we launched it it's called the node - -
yes package we worked hard with the
chrome team here to build a headless
graphic stack for that and we wanted to
take that and accelerate our existing
WebGL stack all headless and node and
this library runs by angle and angle is
the driver we ship in chrome today and
it translates WebGL calls to your native
system graphics stack so in Windows
that's direct3d
OpenGL on windows and in your native Mac
OS graphic stack implementation so we
think this is gonna be great for some
desktop apps like electron mobile and
embedded space and in devices plus this
is going to bring GPU acceleration to
Mac OS we're working hard to finish this
up a couple things so we're hoping the
launch here later in June or sometime
this summer and I wanted to show a demo
of this actually running we've built a
really quick electron app so if I go
ahead and just run my app this app uses
mobile net which is one of our
out-of-the-box models that does basic
image classification so you can see an
image and tell you what it is so as I
pull up in my app here lo not the most
exciting UI but we it shows the GL stack
that's running you can see it's running
angle with an open Geo for one core and
the latest OpenGL ES stack through angle
and when I click run demo what's
happening it's going out it's fetching
our model it's loading it and it
predicted that that's a Labrador
Retriever and we're running about 150
milliseconds or I'm sorry running 150
predictions on an image and our
averaging about 23 milliseconds so
that's that's very close to 30 frames a
second in real time so we think this is
going to be really great on the electron
side it doesn't block your UI thread for
doing all the displays your dispatching
all these Kreml calls through the node
process all with a headless GL and that
package is like five to ten megabytes
it's very small and I also wanted to
show one other thing this is the latest
type of IOT boards this is a Nvidia
Jetson Nano and basically just has a big
GPU stapled to the top of it and we were
able to last week at this writing with
this headless stack as well running that
same model a note console dump is it
that most exciting but we're doing
around 76 milliseconds of inference time
just with the very thin arm 64 build of
our our node back in
I also want to talk about another
library we've been working really hard
and it's in browser visualization for
our tensorflow GS library so ounce this
package it's called TF GS - biz and you
can think of it as like the chrome dev
tools for ML models we have this thing
called the visor and it slides out and
it's a canvas for painting a bunch of
elements that the library provides we
have a bunch of built-in charts such as
loss in accuracy for ML training we also
have what we call high level
visualization methods this basically
allows you to look at those complicated
hops like convolutions which do a bunch
of filters on your image while you're
training and see what's happening in
between each of those convolutions model
evaluation utilities is another set of
drawing libraries and that sort of shows
you where your model is might be over
biased in particular class and ways that
you can sort of see how you might alter
your dataset to make sure you have a
very nicely trained model all right
we've been talking about we've shown you
a lot of stuff but we want to show you a
lot of the things that Nick held myself
and the team have been thinking about
and where we're going forward with the
project
one thing is we're really excited about
the current future with all the new
specs that are coming down the browser
especially with JavaScript on the
website we have two new standards we've
been looking at really hard the last
couple months one of them is web GPU web
GPU is the next-generation graphic stack
that's coming to the browser we've been
working really hard with the chrome team
to try to get that implementation up and
rolling another one we've been looking
at is wasum now in the ml world we
really need to do Sindhi to make wasum a
really an effective accelerator for CPUs
so we've been working really hard on
that with again with the chrome team and
we're hoping to have something for
devices where the GPU isn't all that
great we can fall back to Lawson cindy
one of the great parts about the ml
space is just amount of research and
we're finding about every year arm all
the same models have get faster from
reductions in architecture or new
hardware acceleration stories so every
year of the models that we keep showing
continue to get faster especially on
edge and browser devices another great
product we have at Google is Auto ml an
animal solves the whole training part if
you want to do an image classification
problem you can give it a set of images
and it uploads to the cloud and it
automatically finds the right
architecture for your model and then
spits out the model that you can deploy
on your device we're looking at some
integration with that team as well to
make it just a really seamless
experience and the other thing that our
team has been focusing on is just
optimizing our existing backends
so our WebGL implementation for example
we worked on packing textures which is a
fancy term of using less memory as as
much as possible in our acceleration
library and we found that speed spent a
bunch of things including iOS up to 10
times faster than what we were seeing
before looking at the things were going
to launch this summer visualization has
already launched another package that we
didn't really highlight it's the data
library and the data library is there
real easy to use package for getting
stuff out of the browser microphone data
webcam data you don't have to worry
about converting to tensors so you can
just sort of streamline these things
into the braaap into your model on our
platform side expanding where we run
tensorflow GS as mentioned we chat the
headless WebGL stuff and then we're
really starting to dive into how we can
provide a nice react native experience
and then on our out of the box model
fronts we're going to continue focusing
on audio and text models as well as
improving the accuracy and performance
of our existing offerings what I don't
want to thank you for attending our talk
everything we've shown is we work purely
an open source and all of our stuff is
found on GS tensorflow org one of the
other things we wanted to acknowledge is
while Nicola and myself work at Google
and we get to work on this project we've
this project would not be where it's at
without the large number of open source
contributors we've had and we want to
extend a thank you to them for all the
hard work they've done and then one last
plug
we are actually hiring a developer
advocate for our team and if anyone is
interested please follow that link or
come see us at the booth here at GSK
that's all
[Applause]
[Music]