Okay, good, so welcome to this talk and it's about advance user experience with tensorflow J's and
I'm Oliver
And this talk really is it is about user experience, but it's much more about
Finding good applications for machine learning in the browser and I want to kick this off because I have zero slides
So I want to kick this off with a stone joke
Hopefully no one knows it already. So this is about user experience. And oh sorry, this is about user experience in the way
It actually works. So hope for the internet works here. So this is what you the user do
So it doesn't work quite
Obviously guys wrong
Fixing it. This is I mean
Because you're laughing. I know you have the same feeling as me. Yeah, if you have watch people using user
So same thing fixing it
Again, yes
So that's the joke that I stole but obviously it worked
Because I think it only works if you have some sort of experience like this you think so hard about you user
Interface and then that's what people end up doing
They will figure out how to how to deal with it, but maybe not them sense you expected it
So this talk is about the same thing
Don't blame me for the for the styling
This is my weak spot. Oh wait
so, um
In a typical user interface. So this regard those buttons those are real buttons the the buttons down here. I'm not real buttons
so the buttons are just here to
make a point so typically
In a very classic
user interface the buttons they will light up when you hover over them to give people the the
certainty when you click now, you will click this button, so
What I had in mind was would it be at all possible to highlight the buttons before you click them?
So would it be possible that the machine learns? Which button are you going to click now on next and
You may ask what could be the applications of this and I've talked to quite some user experience people that said
Nothing or less and then they thought for a little bit longer. So maybe I asked you so I get something back here
Why would even do that? I mean highlight the button before the user clicks on it
Predicting which button is it going to be any ideas any takers? Yes
Perfect so about flows so actually I haven't implemented this but with the buttons up there
that's a horrible user interface because you need to click those buttons in certain order and
Some orders are totally don't make any sense you get weird error messages
This would be my next step like learn how to actually use those buttons up there because they're they're complex
But I haven't implemented this yet. So what?
What would be another reason why would you why would you do that? Yes
Accessibility yes, so make it easier. So that was also my fault. Yes. You have another point. Yes in the back
Sorry
Yes preload resources that was also something that I came up with
someone said maybe there so I don't know a server less function behind that button and you want to start it up because you think
Oh, the user is going to click on that button soon. You better have this thing ready or
Preload the next page or someone
I've written down some stuff. Wait. Oh
yeah, someone came up with the idea when there's artificial reality and it's always
Pretty hard to decide which detail are you going to show and then if you can predict where the user is going to look next?
then you can preload that stuff and
Again I only implemented the easiest word because I just wanted to know it's this at all possible
Could you even do that? And the easiest version that I implemented is just what you said
I highlighted the button that I think the users going to click. So, um
There could be heuristics to do this you could just measure I don't know
It's the user close to a button like this
and if the if it's reasonably close then you would highlight that button which I don't do now and
You could also check its the mouse closer to this button
And so they and it might all work, but it might not match the individual style
That someone has that used this user interface
for example
my I always use the trackpad and it's
extremely sensitive because I don't move much and that might mean if I want to hit this button I might
Briefly go over it and then go back
That means depending on the user that you're having and depending on the machine that you're using and how is it set up?
The mouse movements might be completely different. So I'm pretty sure
In my I'm not sure that I'm pretty sure a machine might identify me just by my mouse movements
Out of all of the people in the on somebody's nodding. I haven't made the experience but my mouse
Mouse movements are so weird. Italy makes people really nervous
angry
so
Having said that so what I came up with is that I said, maybe it's possible to learn that
Those movements using machine gun and
If you want to do that, the the crucial part is you need data
So the typical machine learning that we do you need to have data and if you don't have data
Probably not going to work, but fortunately I can generate data on the fly and I'm doing it like this
So once I hover over button, I generate a data set and typically the user would also need to click that button
But it would take me forever to generate just a few data samples if I really clicked on that button
So for the sake of this demo, I'm just counting hovering over button is a click
this is this is just fake or hacked but that's the only hack part and I'm doing this to
quickly
Get some data. Yeah, and what kind of data would you I mean, I'm
Generating data. What kind of data would you would you collect what would you do?
I mean if you were to predict the button I'm going to click you solve those movements. What would you recall?
Yes
Proximity you could record that I did not do this. What else?
Yes
Yes, that's what I did so I recorded XY of the mouse I recorded the Delta between the previous
Movement because I just get this from the mouse movie event. I just get this for free and
delta-t exactly what you say so and
There's also something really special about this. I recalled a sequence of
So to say time stamps and I can show these
If I find them on local storage, there you go, so I stole them local storage and
They are they give me a second here
And that's how they look like so this why hopefully it's big enough is it big enough for the back
Yeah good. So the why it's just an idea of the button and two means this one I
Think so. This one should be
One and this might be zero or maybe this is one. This is two this is three, but I'm not quite sure but why represents?
The button and I
Made this already complicated so x and y don't mean positions on the screen, but they mean the input sequence
This is x and y is the button that I'm going to click so
this is the input sequence of
Mouth positions you see this is position X this is position Y
Delta X Delta Y that means difference from the previous mouse position and this is the time step or delta T
So how much time how many milliseconds past from this event to the other?
And sometimes it's very similar and sometimes it's different
so that's my input and
I
Did some pre-processing on that because I was considering
how many of them would I really need like maybe 10 maybe 100 because I didn't know I
Cut all these sequences into parts and trained all of this into the model
So but you could like for example
What would he suggest that you could have taken anything else and if it works you find out by experiment?
That's how it works in machinery. So you make an assumption you try the model and then you see if it works and
well, this might work at see so I have the data now and now I
have this
This collection of buttons and I'm the only person in the world right now who knows how to
Use this thing because it's a horrible user interface giving a talk but user interfaces and then they playing horribly. I
Apologize, but that's the way it is. Now I can train on that data
So I collected some data and I collected some data before you see I have
369 dataset that means um, I don't know my wave down here from over from outside to down here. I
Did some clicks to have some sample data and if you're at all into machine learning having?
370 samples for three categories is nothing
Tippity people won't even start they will say this is not going to work. Well
But maybe does so I'm clicking this train model now, this is all
I've no fall back. That's why I'm having sweaty palms here. So this looks good
reasonably good
this is using tensorflow, Jess and
Tensorflow jess comes with a very nice brutalization library. I did not write this code
So this stuff that comes with tensorflow Jess and it shows you something and hopefully something good
Oh and shows me something good, but I didn't know before I kicked off this thing. So it looks good
What's most interesting is these?
Two curves down here. That's why I'm doing it now
and
These curves tell you something about how well the training process
Proceeds I will show you some code later
But let's save on the conceptual level for a while
And we concentrate on that blue curve and the blue curve describes over time
How well is the machine learning a model able to reproduce the data that that I gave to it?
so I've had in the mouse movement sequences and
as a prediction the button that is going to be clicked and now the machine learning tries to
learn which sequence
Spits out which bar and
the higher the blue line gets the better it is able to do that and this is around 70 percent or
0.67 that means of the 70
Sorry off of all the mouse movements that I did 70 percent of them. The model is able to predict correctly
so I made some movements to the to the button and
That's pretty good 70 percent on that
pretty astonishing
But what's much more astonishing is the orange line because that is data points that the model has never seen that means
20% which I decided deliberately I've taken off from the
Start now it looks good. I've taken off from from the training data, and I'm just using it
To
Constantly check how well the model is doing on something it has never seen and is anyone at all familiar with machine learning anyone?
When we say it doesn't look I mean would it would you have expected that
Who would have expected that?
Maybe you didn't expect that because I did make myself quite clear, but this is a pretty outstanding
Result so it's not something you could expect from very small training data training on this machine is a very powerful
GPU but it's still a laptop
Something like this. Typically. I don't know. It wouldn't perform that well. I'm not quite sure
Why does this is so welcome to the world of machinery, you make an experiment. You can't explain it just works and
that's my world at least javascript is also my world but also that so um
before we try that and I'm hope you saw I just tried trade this online we can get some more information and
This is also built into the visualization tools of tensorflow
and I'm always so happy if you guys don't see any red stuff in the console because it's it seems to work quite well and
This is a while. I'm hiding this for a second because what you see now, it's a very powerful tool
It's called the confusion matrix. I don't think it's only used in machine learning but in
Machine learning it actually is so what can you see on?
The y-axis that is this axis here
you see the real buttons that the model should predict because it knows those are the real buttons that means
left button middle button right button and on
The x-axis you see the prediction of the model and this is useful
Because in the perfect world and in the model that works perfectly you would expect all the clicks to be on that diagonal here
But you don't see that you see something else. So what do you see? So this means the real
value that the model should predict is left and
The prediction actually also is left that's good. But then this here
The predictor should be left, but it was middle
22 quite a few and then the prediction should be left but it is right and
Then for all the three buttons so for the middle button
we see something that at least from my perspective was expected because the middle button is so close to the other buttons that
You get a lot of confusion
Even though it is the middle button you get a lot of confusion for either left or the right button
so that's also totally expected and
Finally for the right button that's also something that's slowly expected and also something that I would have expected for the left button
So it's actually the right button, but sometimes you confuse it with the middle button, but only rarely with the left
And this is because of the way I created those silly mouse movements
But it's actually pretty good even without trying the model. I now know that this model might be good. Um,
There are some problems with it first problem
This is only on the data the model had we still have no idea
is it going to perform well in real life I mean and
will I do the same mouse movements I make a little bit interesting because I'm still I still also don't know but it could it's
It looks good now
If you think you would never show this to the end user something like this and the end user would never use something like this
this is totally true and you can at any time heighten sting and
You can toggle it back in later
this is just tools for developers like me and
I also haven't shown this but while I was training it was totally usable
So the whole thing was was trained in real time, and the user would not have noticed that I'm training
Something I also did not show him. But maybe it's also too late now the hit on my GPU was immense and
Also the the fan of my laptop
Started to blow because this takes a lot of compute power on your GPU
Uses WebGL, so that means this this is your installation
You could load this in your browser right now
But you wouldn't know which buttons to click so don't do it because it has horrible user experience
Um, so that means the idea is the user clicks something you record at a certain point thing
this might be enough data you train the model you get all the
Evaluation results and then decide am I going to use it for a real prediction or not?
And this thing might be okay. We have one more metric and this is accuracy per button
That means this is this is just a number and this number there's an API for it
I'll just question that number and then I could write some code if the accuracy is above say 70% for all the buttons
I will enable that model and I can just do that
How shall I do that?
Yes, I'm gonna bit scared. I'm really so this is no show that I'm scared
so I can toggle the prediction and
I need to open the console because all the interesting stuff
Is still locked here. This is very early example, so now
Let's see. So what you see
I'm doing this thing where I'm pretty sure that that it will be. Okay. So just see oh it looks okay
So now it will predict I'm going to click the right button and there's no explanation for that
Yes, there might be an intuition. It changes a little bit still right button, but
This thing basically is a black box. So you have no idea. Why does it you just happy that it does
but you get some information you get some information down in the console and this is
probability per button
so these
So the microphone is here
I could have used the mouse, but then I would change this so
let's stick to this one, right so
That means the probability for the right button
75%
the probability for the middle button is 10 and the probability for the left button is 15% that
Means at a certain threshold that I decided to be 50%
I will highlight that button and say the user is going to click that button and let's see now so
Middle button, you see I'm not hovering over it. I'm them just close to it
Left button in works, and I had sweaty hands it sort of works
You wouldn't probably say it's perfect. And it also seems to work on proximity. Not not of mouse path, right?
Maybe limit on Mouse path, but mostly on proximity
At least that's my impression. Would you agree?
Is it Mouse path?
It's not perfect. You see that it's not perfect, but it might be good enough
For anything reasonable and we just did this on the fly right here. So it's it's not something that I prepared
Um, but you might ask it's this maybe you asked that I don't know if you do. I'm just saying you ask
Is this the right way to do or would you rather?
Not use proximity so much but do you really the movement of the mouse is that more interesting and
if you value proximity or the the past of the mouse movement over a proximity and
That is something that you model into your model and I will show the model now
Hopefully
Sorry just enough space
So, um, then it's the tensor flow test part and all the other part
I'm pretty sure all of you could also write this is just
clicking and then have a mouse missed 9 then have a quick list not all that stuff and prepare the data has had a little
Bit but this is the part where you only have very few lines of code
But you need to think about those few guys of course and have variations of this so
This tensorflow thing is it's the tensorflow namespace
I'm saying I want to a sequential model. That means we have layers they follow each other and
The first and this now it gets interesting. So this stuff before maybe not so much
This says it's a simple RNN and that is the sort of neural network that can learn from a sequence of things
And there's a lot of stuff that you don't need to know about
but this will say I
have
quite sure how many maybe it's 25 I cut the the
mouse movements into into chunks and then have five features that means how does the input to this thing look like and
Then this says how many neurons do I want? And this is tony hand-wavy you just try that out
and typically you don't want to use too many because it needs to be trainable on the browser and
You saw it is if you take less it goes much faster, if you take more it might not work. So well and and
It also takes a lot of time and you saw the predictions were actually real time online
so the interesting part is
The evaluation of this thing was is much worse
Than on the other things that I will show in a minute
But in real life it showed it showed the behavior that people actually expected that when I moved close to a button
It should be highlighted
That's the that's a result people expected. But actually maybe you rather want something where the whole path the whole movement
Falls much mentor into into account and there are other
styles of neural networks that are better at this so you decide how much of the history do you want to keep and how much
interesting how interesting is their history and I have
Wait I have this
Pre-recorded because I didn't know if I would be able to make this run in return. So this thing here
In the middle you see it reacts much faster or much
Very much earlier. It was it was an earlier version of this thing
But it only reacts not by proximity but of the path that you did that mouse describes
so it's
Much more expensive it took much more resources to train and it's still not
Intuitive because if you if you just move close to the button
It will do won't do anything because that's not how you click on a button
You don't slow you go to the button and then you eventually click. This is much more
Much closer to what you actually do and if you want to do that
you will use a different type of machine learning model and
this different type of machine learning model is
Commended out here
it's called an LS TM which means long short-term memory network and
You might see that apart from this thing here
Where's this? It's same thing and the name is different. But the name is just name of variable. Then there's a
cheaper version of this which is called GRU, but
You could just exchange that and do different experiments see and I made some comments
slow to train
Evaluation is bad, but it actually performed really well and in its it's the network that I used for the for the second example
Good five minutes left so I will show some more code unless
They're questions right now questions
Yes
Yes, so the question is
Have I made something like a baseline? Where you?
Exit and no but exactly what you propose also was in my head
I'm just using the vector and I'm seeing what is the the mouse curve pointing?
It might be it works better, but I haven't done this and also not proximity
I think it would be it would be pretty easy to implement, but I didn't want to spoil it for myself. I think
Maybe it works better. If you do that, please contribute. It's on github make a pull request
I would be maybe I would not be so happy, but please still do
It could be better. So but the point being here, maybe that's maybe the use case is silly
I'm not even sure myself, but the point here is so this data it never left the browser
So this stays in the browser, you can download it yourself
But it will never reach the server of someone you don't trust in terms of privacy
For example, you don't want that maybe and also the model that you're training is also completely it stays on your machine
It will not leave your machine and the prediction will also be on your machine and it will be in real time
That means you do something and and the model will predict you didn't see any at least I didn't see any delay
And you didn't see any installation so you can start this maybe not on your phone because you don't have the mouse
But you couldn't start this on any laptop and it would still work if you know, which part is a click
And that's the point. So if you if you have anything you think that might be a good idea
This might be the right thing. Otherwise, it's pretty hard to explain. Why would you even do machine learning in the browser? Because
for example
the browser doesn't
doesn't train at least in Chrome when when the
The window is not upfront when it's not high light or when you look at something else. The browser will restrict your resources
So it's not the ideal platform for training, but it might be if you after
previously or
high activity
so
just
Well, hopefully to show you that it's not very hard to get this done so this is something you could copy from pretty much any
Tutorial and that's the whole model so model definition
this part
And this part and the other part is commented out so it doesn't count I guess and
the most
Experienced that have put into it was to make it very general with very few data sets
Which is typically extremely hard and this involves
tweaking with strange parameters where there's barely any theory to it and one of the theory sorry one of the
parameters that was tweaking was this drop out thing and then this patch normalization and and
this is just
experimental you've
You have some intuition and then you just play with it and you see when it performs well and what you see as a result
Of this very strong. Well sort of strong
Regularization so it still works with not many data sets. Sorry
so
preparation boring
training
This is training, so you have the model you call fit that means you stick in
the input that is the mouse movements and you get back the
button, but in this case you just train the model tell it this is the true data that I have learned from it and then
There's still too many parameters that I can't explain now, but I would be able to do that in five minutes
Okay, then prediction so done after that
This was triggered by one of the buttons. So after that you can evaluate that
Somewhere and then eventually you call predict on that and this is again model stick in the X data set get back
the Y that means the bar or
In this case to be more accurate the percentages of probability for each bug being picked
so
You can save that to local
Storage you can download that
And then you have two
files that you can store you can use the data to train that on just normal tensorflow, which also did
Where I exported the data I did the training
and then you can also
Export the model that you trained on a real server and load it into the browser if you really want to and the idea
Behind that is maybe you want to share
Some training experience with other people who are willing to give the data to I don't know someone else
Uploaded it to it
To a server and then you could train the general model
Upload it to this thing here. I could do this. It's loading remote model now and you can see I
Don't know if it's performing well
Not so much, but also not so bad. So this model that I just loaded it has been trained on a server
Probably more data and still works
So with that, I'm
Closing it and what I would like you to give to me is is ideas
So I don't with our I don't on person
is there anything any inspiration you got from this what you could do in the browser didn't tell me and
If I mean if it's great that you keep it to yourself just implement and make a lot of money
But maybe maybe if you need I don't know
Some going back and forth talk to me. I would be really happy. So thank you and have a nice conference