[Applause]
[Music]
hello hello
[Music]
singing will happen happening happen
will happen happening happen will happen
again and again because you and I will
always be back then you and I will
always be back then if there were some
amazing force out some enough time to
take us back to where we were and hang
each moment up like pictures on the
walls inside a billion tiny frames so
that we could see it all it would look
like will happen happening happen will
happen happening happen there we are
again and again cuz you and I will
always be back then you and I will
always be back then
that's why you and I will always be best
friends
[Music]
I'm just new I play the ukulele and I
also like games and so there's many
things you might know about me or want
to know about me these are the things
that are pertinent to this talk I like
games like lots of games board games
card games video games like a lot of you
I also program and then lastly I really
like mischief I like because I think of
it as a jump sounding board jumping
board whatever that term is into like
creative problem-solving creative
thinking and innovation I like mr. so
much that I named my son Loki this is my
like a Terry father shot of my my son
who I'm so proud of uh-huh so one of the
games I've been playing is this game
called set I was playing this game with
my friends Susan I want to cut and go
over all the rules with you first so
it's a card game where by the way the
New York Times has a set game that you
can play it's kind of fun so you should
go check that out set is a card game
it's a matching card game you're trying
to make sets of three cards each all
these cards have a color number pattern
and shape I can toggle this on really
quickly for any colorblind people so
amongst these attributes what you're
going for is a set of three cards where
the attribute easy they're all the same
or all different and so here this is
this is an example of a set where the
number is all different one two and
three the pattern is actually all the
same they're all striped their shapes
are all different and the colors are all
different so this is a pretty fun game
until you play with somebody like my
friend Susan who I want to give you a oh
no I don't have music I want music on my
thing maybe
oh no music so this is what playing set
against Susan is like it's where she
just starts taking cards and I'm just
observing and telling her hey good job
wow wow great great take their Oh almost
had that one oh okay oh wait do I see
something no no not at all
so it's trust me it is a fun game but
until you play against somebody like
Susan so how does that make me feel as a
protagonist of this talk
how does my protagonist satisfied with
their life I can't beat Susan instead
what would it take for my protagonist me
to find satisfaction this is their goal
beat Susan at Seth what are my
protagonist biggest fears and character
flaws well probably three things first
is that I don't really want to put the
effort into getting good at this game so
I could sit down and just play alone a
lot until I get really good at this game
because it's tricking your brain into
kind of seeing these sets as quickly as
possible but I am pretty lazy and so not
interested in that
I'm also impatient I just want to take
the time to do that
I'd rather do it and use it to anything
else and last of all the mother of all
character flaws hubris I'm pretty sure I
I was reading about Starcraft being beat
by deep mind and alphago and alpha zero
and the like I can do this I can solve
set with my computer and I can finally
beat Susan this way and so I started
thinking about how this problem is
formed and so for a human it's
interesting because if you think of the
problem in two parts the set making is
the hardest part for a human when you
you can quickly see these cards and you
know what they are but you can't make
the sets very quickly out of them for a
computer it's actually the opposite the
set making is super easy and it's the
vision part that's the hard part
so as you can tell we're going to
confuse your vision the crap out of this
problem
alright here's the plan we I want this
to run on my phone so I'm gonna use the
camera on my phone get into a browse
put on the canvas and then I get pixels
from those pixels we're gonna identify a
bunch of cards in the thing from there
we want to extract out every card and
then put each card we want to identify
the color number of shading and shape
and then eventually we're just going to
boot forth and solve for this at the end
all right so let's get into it
so this is my test image for this for
this thing as you can see bunch of cards
getting this out the things that you
kind of care about are these ones so
this get image data is what we care
about here we get out this big huge
number array so this looks kind of
foreboding and creepy it's got one no
eleven million pixels in it so let's go
over what this image data object is
really quickly you can think of images
as a series of pixels here we have not
six pixels six huge pixels and then each
pixel is itself basically four numbers
there's a red green blue and a
transparency or alpha component or GPA
so that's four numbers times six pixels
that's 24 numbers so that's our 24
numbers that we see here and then it's a
flat array so it's one dimensional but
if you know the width that you can you
know basically we wrap it and so this
array corresponds to this picture up
here and if I change this that trend so
that's transparency so that goes away I
can bring it back I can make it black I
can make this one red so you can see how
those think go so we're gonna be using
this image data object a lot throughout
this talk so I gotta get familiar with
it
so first step is we want to threshold
out we want to simplify this picture a
whole bunch when I basically changed
into straight black and white and that's
to make our job a lot easier later on to
do this this is called threshold incan
vertebrae are G B value into a gray and
then if that gray is over below a
certain value I either set it to white
55 or black zero and so if I change this
this slider you can kind of see how it
affects the image and then an
entertaining thing to note here is that
you'll notice that this is not a linear
transform from our G
Beata gray and that really quickly is if
you look at these green and blue values
here so this is actually the same amount
of green and blue like hitting your eye
but because of how your retinas all
jacked up you perceive the green a lot
more than the blue and so you think
green is a lot a lot lighter than blue
and so that's take into account with
this formula here alright so we have a
black and white image now we want to
find those cards again to do that we go
to NPM and just find somewhere in the
package that finds finds contours for us
but what this package does it's called
more neighborhood tracing which sounds
scary because it has a dude's name on
the front of it and uses neighborhood in
the context that makes no sense
whatsoever
but you can think of this as say you're
an aunt and you really like black pixels
and your exploratory ants so you're just
out off for a walk and you're meeting
with article so you're just scanning
through all this black and then you all
of a sudden hit this white pix whoa whoa
what is this this is new let's figure
this thing out so you kind of back up
and you go to the side you kind of hit
it again like Oh white pixel again you
back up you got you go to the side you
do that a whole bunch until you're kind
of like making it way around this whole
thing until you're all the way back to
the beginning again and so if you've
paid attention and like late couples
down you've basically made and trace the
path around this card so that's more
neighborhood tracing you'll notice that
I then so I take the threshold image
we're going through that I get a bunch
of contours and then I ended up
filtering based on area here and so if I
kind of change some of these values
which now you'll notice that I'm
twiddling values a lot of these values
have to do with the source image so I'm
kind of like picking numbers out of the
air based on kind of the parameters of
what I'm assuming and that's gonna be
important later on so here I've like
made the area threshold go away lower so
I can basically pull out smaller shapes
but I actually don't want those so let's
go back to here and only get 12 of them
so I'm back to my original 12 cards and
let's see where are we we want to now
pull out each card
to do this we need to simplify that path
a little further instead of so that path
is like a whole bunch of thoughts around
that card ultimately what I want is this
rectangular string or quadrilateral that
consists of only four vertices to do
that there's probably a cool name for
this thing but I made up some random way
to do this what I end up doing is
basically removing pixels so I want to
remove redundant pixels along a line and
so I just keep removing and down until
I'm down to four the way I just choose
which pixel to remove next is basically
the one that results in the these
minimum least area change and which
results in a pretty good rectangle this
point now I want to pull out every card
so this is now where I'm going from here
to here so this is called perspective
transform which again go to MPN you just
search for perspective transform you get
this thing this is basically linear
algebra
I'm not guys playing this one but linear
algebra matrix math you can pull out
basically and transform from perspective
into this perspective all right we got
every card it's kind of oh so we need to
correct for the lighting in the room
basically you'll notice here that this
is I took this in like a yellowish lated
room sometimes you get warm lighting
sometimes you get cool lighting
ultimately want to kind of correct for
that so we want the background of the
cards to basically be as like 255 - to
565 white to do that we can assume that
our white point so we need a white point
we can assume that the background of the
card is white and so what I've done is
just average the top and bottom borders
of every card and called that my white
point and then using that I can shift
every pixel in the image in RGB space
into basically where the background is
of averaging out to be more white than
the rest of the card and all the all the
other colors and the slightly
brightening also which is actually good
okay so recapping where we are we got
our image we identified some cards we
extract with cards and now we want to go
for all of the colors and all the things
in them so here we have the spectrum of
the required so I've gone through and
basically just grabbed every
pixal mapped it to a hue and then
throwing out all the white so as you can
tell this is actually pretty pretty
indicative of kind of all the cards I
ran through it's pretty easy to tell
that there's a purple card a red card a
green card this weird one here is kind
of interesting so that one was actually
this one here we can kinda see some
rainbow in going on here which is still
fine there's a lot of strong signal here
still in the purple so totally fine next
one is number so for number again I just
throw contouring out again for each card
I run through the contour finder and I
get contours and I end up looking at the
percentage area or sorry the percentage
width of the contours versus the entire
card so if there's one shape on there
it'll end up around 25% when there's two
it's around like 55% and when there's
three it's around 85% and we can go
ahead and correct for this kind of case
where I miss a contour by just using
this and mirroring it over here and
assuming that deducing basically that
this is going to be three okay we got
two things down next one is shading for
shading we want to grab basically the
inner core of every contour and look at
its average luminosity so to grab its
inner core you end up just taking a
bounding box of the entire contour
cutting that down to like three fourths
of the size of the thing and then
averaging pixels again inside of that
internal box to give you basically so
for an outline you end up with white
pretty much so everything above 55
four-striped you end up with somewhere
between 230 and 240 and then for solid
you end up with everything below like
180 again you'll notice that like I'm
kind of like hard coding these values in
at some point shape is a little bit more
complicated so shape we can kind of do
the same thing and take that bounding
box and look at the percentage area of
the contour versus the shape and then we
get this so diamond is actually a lot of
doesn't take up a lot of that area
squiggle to explode but more an oval
takes up the most this though is getting
a little close for me so like this the
difference between Oh
those were iffy so I decided to throw in
yet another dimension here so this is a
convex hull so this is this chart down
here is the same as this top one but
I've added a Y dimension so it pulls out
some of these values the convex well you
can think of is like basically taking
some saran wrap and wrapping these
shapes in saran wrap and so that's the
shape you get for oval and diamond you
end up with basically no change in the
convex hull and then for squiggle you in
to put more so this pulls it out a
little better and makes the
classification a lot easier finally
we're down to like the boot floor set
finding here's where the computer
actually has a structure information in
order to go ahead and look for all the
sets when you do the math it's 12 cards
3 cards per set that's only 220 possible
sets which is absolutely nothing to a
computer a crunch it crushes this in 20
microseconds and checking all those 200
many possible sets for four sets to kind
of put this more into perspective when
you think of sets per minute I'm at like
one set per minute Susan's good she's at
like 20 sets per minute and the computer
is at 300,000 sets per minute so it's
got us beat there by far so then knowing
the sets you just draw them back on the
source image and got this cool looking
thing here so how does this play out how
did I do
so let's see I'm gonna give it away but
this is playing again against Susan with
with my first build or so you know I'm
not doing so well still actually there
was no sets there oh never mind there
was one she took it oh I got one that's
way better than I was doing before and
it was basically a pretty even match
actually basically when Susan's slow I
can get a set if she's fast then I still
lose and that was actually due to due to
no this way so we lost in this example
still it was very sad so what happens is
that like we trained this thing on
basically one or two or three source
images which is exactly the problem with
overfitting so I
we built the model based on like a
couple source images I didn't take into
account a whole lot of different
lighting that I might come into contact
with shadows lots of other different
things and so the way the fix this is
basically you get a lot of source images
so I went back and I built into my thing
a way to quickly grab a lot of images
and then tag them and then based on that
I can actually find stuff or sorry I can
train this a little better so it's
basically taking all those all those
images and using those who in four
models magic numbers which I call poor
man's machine learning which is exactly
actually machine learning so don't let
anyone tell you otherwise
it's basically machine learning is doing
so I put together a training montage
plays Fitz on YouTube oh it does ok I
don't need sound I'm gonna do this live
start over cuz I got time this right
so many times you change your passion
for coal
don't longer dream of the pathless fight
justice
I have the tiger it's the thrill of the
fight rising up to the challenge of our
rival and the last known survivor stalks
their prey
I got a
I love the tiger
[Music]
[Applause]
oh I went out of full screen no full
screen although all we're doing is a
live demo next so okay I did I get so I
have a plant in the audience and
everyone welcome Anne and then could I
get a dealer as miles around or anybody
who wants to deal deal some cards
what software update to my phone trying
to all this nice wait
I guess quit iTunes that decided to open
on me oh look at some random images it's
good so anyone wants to cheating us
their friends go to set that Visnic calm
and that is my screen alright alright no
dealer huh we need a deal there a deal
it would be great ooh Billy hi Billy I
work with Billy four by three twelve so
well this I have a chance of losing
easily because the lighting up here is
interesting
I tried this earlier and I think I was
okay but also Anna's pretty good this
game and so we'll see we'll see how this
goes are we ready we ready to go seven
oh I'm cheating I can take it because
I'm a cheater Oh No these cards are
weird
Justin threshold Inge is there anything
here that's not when it's a false
positive
these cards are too close to each other
wait wait wait for it no there's a sec
what yeah this is why humans still have
a chance
cuz computers can't see set I think the
pressures getting to end the stage
pressure oh there's a nice set yeah go
wow that was a harder set yeah set okay
I think we get the idea cuz this is
gonna take a while to go through but
there's nothing right there that Han got
all right I'm gonna declare that a
victory
thank you again so great we built some
networks actually prepared for both
versions of this but we'd won so that's
good
this is a video just in case of showing
that I also beat Susan
so I completed my my accomplishment so
the remaining things actually through
here so I did all this in CPU in the
browser there's you could do all this in
WebGL which would be a lot of fun it's
totally dense itself the shaders and GL
so given that I have this like training
set now I can actually throw as a
tensorflow which would also be a whole
lot of fun the other things I want to
highlight really quickly are other
versions of things that have kind of
built actually did not build medium
snowflake thing but so this is actually
a totally different game this is like a
crazy game where you have crazy dice you
have figures and you kind of build your
character up over time
one thing about building your characters
you're gonna have these like item buys
and you're trying to figure out what use
is better than 44 that's 11 they use
random dice they look the same so I
decided I'm gonna root for this too so
I'm gonna run simulations against all of
this and these are damage curves for
each of these weapons at different
distances another example that I want to
highlight is actually stool called
medium or a cool called snowflake from
medium so this is actually a
visualization of the engineering ladder
this is a little bit looser as far as
like actual tools to hack your life with
but to me this is a tool that you can
use to hack your career with so you can
kind of see where you sit on their
ladder for here seriously is really good
at project management and initiative and
accomplishment but she is very low on
the community scale which probably stop
killing people and blowing them up but
she can like have a one-on-one with her
manager which I don't think she has a
manager and they can talk about how you
know she could probably target just
doing this easy zero to one on
well-being and like this is zero to one
on community and she's actually gone up
a level so this is just a tool that I
feel like you can you can basically very
quickly figure out like if you've ever
had a one don't want your manager and
you're trying to figure out like it
what's the next best
for me to work on what would happen if I
like just got really good at some of
these other things where would that put
me this answers it for you so yeah so
again why I again love cheating and
mischief and just solving problems in
ways that shouldn't be solved and I feel
like it leads the basic innovation so
that's kind of why I hope I've inspired
you to actually build something and kind
of shown that like I just I've never
done computer vision things before this
and this is my first pass at it so I
hope you can do that too and then
ultimately all of these slides are on
this website observable HQ there's
enough cheap
that's me there's no and there's
actually blog post content inside of
this so if you share it then it should
make sense to all your friends to thank
you
[Applause]