♪ (bell ringing) ♪ Hello! Welcome to the third video in my
build a classifier with p5.js and TensorFlow.js and there is neural network
in there.
So... I am really exploring this machine learning library TensorFlow.js
and I want to come up with a creative example that shows the full classification
process from collecting data, training and deploying a machine learning model.
And so... the example that I'm working with is this idea of color classification.
So... I'm crowdsourcing this data from you, the viewing audience, and,
if you look at this... you might remember I built this little web app to...
in the previous video... I think that was yesterday... and then now it's today,
24 hours later. It's been improved, thank you to the internet, the wonderful
people who have pull requested various design fixes and updates. You can check
all that on github to see who the contributors were.
Now, let me add a few things here. Brown, kind of brown. Let's purplish.
Let's bluish. Now, one thing I will mention, thank you to Bruno who brought
this up in the patreon slack channel. I sort of said yesterday, you know...
I just wanna pick a trivial data set. I wanna make something that's not...
that has very little... sort of like meaningfulness to it just to sort of
demonstrate the whole process.
But, there is something kind of interesting going on here, in theory,
which is that we are looking at human perception. And I am not mathematically
calculating... like labeling a color according to the RGB values.
I am asking you, the viewers, to tell what you think a color is.
And so, there is a lot of interesting scientific research
in this area, and will reference this video that talks about Berkley researchers
and other research around the surprising pattern
behind color names around the world.
So... there is a lot there
that you can dig into.
So... maybe there is more here than I might originally have thought.
The problem with what I built over here...
is that...
you know you're wonderful, I love
all of you who watch these videos and leave nice feedback...
and leave critical feedback, and all that sort of stuff. But...
the database is a little bit off the rails because
I just left the rules wide open.
Anybody can write and anybody can read to the database.
And so...
thankfully...
panzer on github...
left a pull request analyzing the data
and looking at, kind of like: okay, well there is a lot of stuff here
that looks wrong. Maybe there are some bots that started classifying colors.
And so... wrote all these functions to analyze and filter the data.
So I encourage you to check out this wonderful pull request.
This pull request is now part of the repository, however, it took a slightly
different approach, which is, and thanks to meiamsome, who helped me with this,
which is that I changed the rules.
So the rules yesterday were just basically read: true, write: true.
These are firebase rules. And meiamsome helped me look into how you could
customize the rules. And the things that have been added to the rules now are:
we have some things to validate to make sure the RGB values being put in the
database are actually numbers. So you can see how this looks here, newData
is a number and it's between zero and 255.
We have something to validate at the label.
You know... one of the things that people put: other words that weren't
part of my set of classification labels into the database.
So I have to check that it is a string. And that the actual data's
value matches this regular expression. So... if you have never seen regular
expressions before, I do happen to have a video series about that
(bell's ring)
that you can go watch. But this, you can see that it matches any of these dash ish.
So that's protecting, and then... authentication was turned on.
So what you don't see is that it is anonymous authentication, but you can
only write if you've been authenticated.
This way, it's anonymous... I can track every person, or every entry.
Well, it's not necessarily a person, but every entry from a particular
IP address into the database with a unique id. So if I consider there is a bot
that's just flooding the database, I could either block it or just, like...
clean that data out of it.
So that's what I am going to do in this video.
I am going to use a similar approach to this pull request. I am going to actually
read the data from the database, and then, I am gonna... like... analyze it
and delete stuff if it seems like it's no good, and then download a
JSON file that I'll then use in the TensorFlow.js example that I am going
to build. Did I just spend the whole video
introducing this topic? I think I might have, but I am going to move on
and keep going anyway.
Before I dig into to code, let me just reference one more web page to you
I wanna show you. This is a project that, at the time of this recording it hasn't
technically been released yet, but you can find it at ml5js.org, and probably is
released now that you're watching this video... but ml5 is a machine learning
library built on top of TensorFlow.js that I and other
folks at ITP have been working on. And I just wanna reference on Hannah Davis
who's an artist and researcher, a contributor to ml5, who wrote this
wonderful tutorial about making your own datasets, and thinking about questions
to ask, in particular about responsible data collection, and tagging and
crowdsourcing. So hopefully I'll come back to this topic again and again in my video
tutorials. But I would encourage you to check this out and really think about it.
One thing we could think about it here is: number one: I am building an example
that requires people to see the colors. So what about people who are colorblind, low
vision or blind? That's something I really should be thoughtful about in this example
how can I approach that. And then, you know, who's really able to participate in
tagging and submitting data. Who's being left out.
So, I think the good news for me is this is meant to be somewhat of a
generic tutorial and the data, wow! Does it matter so much if it's perfect? because
I just wanna show that whole process, but you, then, actually being a person who
might work with machine learning out in the real world, you really wanna be
thoughtful about that data. And I hope that... I can link to more resources
about that and cover that more on this channel as well.
So... all that aside, now I am ready to dig in and look at the data and do the
thing that's probably gonna take me the next 24 hours, or three days,
or three weeks: try to clean the data and make it usable for me.
So... yeah, that's what I'm gonna do.
Okay... so... I have a... I mean I could do this... I could download the
data directly from firebase and just put it in like a google sheet to
look at it. That might be useful.
But what I'm gonna do is: I'm just going to actually write a p5 sketch or
just a javascript program to look at the data first. So... I have... I have this
sketch. All that's in it so far is just that connect to firebase and authenticate.
So... what I wanna do is to retrieve data...
I think I'd say something like database... once... value... and then I have
a callback like gotData. I don't know if this is right.
And, by the way, I've learned that the javascript, recently at the Javascript
Convention, which is not how p5 works, which is often the error is first as the
callback arguments, and then the results is second.
I don't know... I'm just speculating what the firebase API might be like.
Let's see what happens. Database once is not a function.
I probably need dot ref. And then a probably need like... colors or
something, right? Probably something like this.
I could just go and look on the documentation. I also have this
firebase tutorial. Oh yeah, I need the database reference. And then... on my
tutorial I say dot on but really... oh! gotOne and errData.
So maybe there is two callbacks. Who knows? Who knows!
Let's say... let's do this: let ref... equal database... ref colors.
And then let's say ref once, value got data.
And let's look and see what comes back.
Let's go back to here. Alright, something came back.
No nothing came back. Nineteen...
Alright, what's going on here? Maybe I should go back and look at my
actual example. Got one... error data... so let's...
Oh! that's a pointer to the data, right!
So actually the data... so... so... so... it actually is a separate callback for
error. It looks like.
So, I'm gonna like... not worry about the error callback right now.
I'm gonna use gotData. And then...
Let's look at the results.
This looks really like: how could I possibly use this? So...
what you're actually getting back is like this pointer to the data, you gotta call
functions on it to actually look at what's there.
So, presumably, something like results dot value is probably what the API is.
No. It's not a function.
So I have to go back and look at my tutorial... dot val!
Let's try that! Let's try dot val.
And there we go, look at this!
Oh! it's a lot of data. Boy, the console is not able to render this.
So, now, is this actually an array?
Oh! It's just actually an object with all the data in it. So I need to turn
that into an array. What's the...
Because I kinda wanna loop through it...
I wonder what's the... oh! You know what I'll do... this is what I'm gonna do:
I am going to... so now... now I'm gonna process the data.
So, first, let me just get all the keys.
So I can object do keys... so let me just say let data equal results dot val...
Let's not console log that.
Object dot keys, data.
And then console dot log keys dot length.
And, I don't need this page anymore!
I'm going back to here, clean data...
So there is five thousand, nine hundred and two entries into the database.
This is never gonna change because just while I'm recording this video I shut off
the ability to write to the database.
Okay, so... what we can actually start to do now is: I could say for let key of
keys. And, I could say let a record equals data key.
And I could say console log record.
So this is gonna log all five thousand of those one at a time, I think.
So I can see, these are all... it's just logging every single data point.
And we can see that they're... For every single one, there is an
R, a G and a B, the label and then this user id.
So now, I think there was... I have...
Watching the database this morning...
I think there was a bot, that was posting to it. So now, it's possible... it could
be that there is just one person who actually clicked a lot of times.
But, so let me... what I'm gonna do right now to just examine the data
a little bit is... I'm going to...
look at... the user, by user id and count up how many entries for each
user id. So I basically need to do something like a concordant.
So... if I say... UID...
if I look at that...
You can see there is all the user ids...
And what I wanna to now is just associate... so I'm gonna just say:
user id by count... and I'm gonna say... is an object...
Okay... so what I wanna do is I wanna say if user id by count of that...
okay... so I need the id, which is this.
If user id by count id... it does not exist. Then, I want to set it to one.
Otherwise, I want to increase it.
And then, I want to... console log that.
So let's look... this should give me all the user ids...
By how many people... by how many entries I have.
So we can look. 35, 33, 78, 147, 208... 189... what's the record here?
201... there is something I've already forgot, 236...
So what's this, is there anything suspicious here? That's the question...
Alright! It's probably worth me sorting this. This is why I should just put
everything into a spreadsheet. But I'm just gonna sort it myself.
So, in order to sort it, I want user's... I'm also gonna have an array that
I'm gonna sort... and so, if I find the new id, I'll put that in the array, and
then I wanna say... users dot sort... and now I need a comparing function...
to compare... two of them: a, b, and I'm just gonna say return...
user id by count a... minus... user id by count b.
So that'll sort the array.
Sorting probably makes a new array, I think... I can't remember, does it
change the array or make a new array?
So... right! I wanna sort the users array and then... I'm just gonna do...
Let... id of users... I'm just gonna iterate over the array and console
dot log... id plus user id by count for that one.
So I know I'm kinda like... ops...
This would be a good time for me to use those new string literals...
Someday I'll get to that...
So let's take a look at this. What did I get wrong?
Ah! It looks like I did! That worked!
Amazingly that worked.
So we can see somebody just did one, thank you!
Thank you! (indistinguishable sounds) who did one...
And then we can see here... 236 entries from this particular user.
Alright! these by the way are called... template literals is what I meant...
By the way, since I mentioned it, let's actually use it. This is a new feature
of the ES 6... I'm here... why not...
Where... if I use back tick...
I can create a string...
That's just with variables with this syntax, I believe...
Does this go out here? Yes that goes there.
So what this does is, in other words,
I could say:
user...
submitted...
I can just write a full string...
And then, basically, anything that's in between these dollar sign and curly
brackets is rendered as the variable value.
So now if I run this...
Again...
I don't know why that didn't... Ah yes! You can see... now it has
that full... and I kind of don't want all of this extra stuff, but I'll just show
you that you can put together a string...
And so... okay... so I happen to know, based on earlier research of the day...
and watching that this particular user is a bot... but I'm getting the suggestion
from the chat to just... discard anything that's over one hundred...
So it looks like this... so that... So those are some...
Those are ways that I can do this.
Another correction I just got is that I believe users dot sort actually
changes the array...
And, you know... since I've started using arrow syntax, I could write this way which
is perhaps a bit more readable... but... who knows...
Okay... alright! So the question here really is: what to do next?
I know that this... from my analysis earlier looking at... these...
things being added to the database that this appeared to be a bot...
I also could, other chat suggestion... I could... just remove everything that's
a hundred or more...
I'm actually gonna stop this tutorial. This was sort of like getting to actually
being able to look at the data and I'm gonna do a whole next video,
because what I think might be useful is actually just look at the data. Because
this is information I can visualize. I can say: show me everything that's pinkish;
show me everything that's bluish... and I could also say like... ignore this..
I could see by user... what they... I can actually look at what they...
try to see: is something really just way out there that maybe I shouldn't include?
Okay? So that I'm gonna do in the next video.
I'm gonna add some tools to visualize the data.