[Applause]
hi everyone thank you for coming to my
talk oh my gosh this is so many people
yeah so I'm just gonna start with a
disclaimer that this is a tech talk
about my necklace so I'm just gonna talk
for 30 minutes about my necklace and I
can do that because there is a ton of
tech involved in this necklace and it's
responding to people around me so yeah
you can see I have a lot of electronics
currently strapped to my body so I have
this choker with a camera attached in
the cameras attached to a Raspberry Pi 3
B plus and the Raspberry Pi is running
nodejs
and it's running tensorflow j/s and it's
with tensorflow JSS running an image
recognition model the machine learning
model and the PI determines how many
people it sees and it sends that
information to an Adreno and the Arduino
is controlling the lights on this
necklace and it updates these colors and
patterns of LEDs based on parameters
I've set for how it should respond like
Paul said I am a not a hardware hacker
for work I am a web developer so I
mainly do react HTML CSS and
accessibility I work on Microsoft to do
which if you want to organize your day
it's a great app or if you want to use
it like me and make lists of weird
German words and a few names for future
pets you can use that too and I go by
she/her pronouns okay um yeah I'm really
fascinated with making myself income
uncomfortable in public so a few years
ago I started making lighted clothing
because I just thought they were really
beautiful but that's also a burden for
me because I love them but it's kind of
hard to deal with sometimes all the
attention that they draw but I've tried
to learn how to channel that and from
anxiety into more like fascination and
excitement and also try to use them to
come confront my discomfort
of drawing attention to myself and I
just want to start off by showing you
some of the projects I started with so
so the first project I ever made was
this fiber optic dress and made this a
few years ago because I was going to a
festival and I thought this is what you
were supposed to do and I basically add
this link is linked to the kit I bought
it's actually a fiber optic rave whip so
it's like a flashlight with like 200
strands of fiber optics attached and I
guess you sling it around or something I
don't know but I coffee Dan
Instructables tutorial I bought this kid
it has just like this flashlight with
preset programs so I didn't code any I
didn't saw it or any and my mom kindly
made this dress for me and I sewed these
two hundred strands to the dress and
made this really cool-looking jellyfish
dress and it people really loved it I
got a lot of compliments I felt
extremely awkward I wasn't it was first
time wearing anything lit and I wasn't
used to all the attention but it was a
positive attention so gave me a
self-esteem boost which was like the
trade-off I wish it could have been more
alive though because I set a program and
I was that was it there was no like
interactivity aspect to it so last year
I made this project which I called with
Paul helps me name this project it's
called Hall it's Couture and so it's
just multiple coordinated pieces and I
gave that link is I gave several
versions of a talk about this so that's
a link to my talk at Jas coffee you last
year so I have these multiple pieces
they're all coordinated together there's
a an app using react that you could
control the lights with anyone on the
web could change the colors and the
patterns it was a lot of coding I had a
web app built in react and then I had a
node.js server it was using MQTT
messaging protocol to relay information
and
had each piece had an Adreno in it and
it was a ton of bartering i soldered
each of these pieces sometimes I had to
rebuild and had to do it multiple times
because that's with hardware yeah so
with this project I purposely wanted to
explore creating this temporary
experience that connected a viewer
directly to an the artwork for a brief
period so to kind of blur the lines
between who's the artist
and who's the viewer and I wanted to
explore the reactions of people when
they realized that the lies could be
changed with a web app that they had
complete control over my outfit um and
it was an interesting experiment but
what I realized that I was doing it more
to please other people
which is ok but it was less of the
direction that I wanted to go in I
wanted to be more focused on me and what
I was experiencing um the last project I
built last year was this speech to image
necklace I know there's no sound but I
have basically the caption I'm saying I
like turtles
and then it's showing the meme if you
know of like the kid dressed as a zombie
and he says and I like turtles
so so this is the last project I made so
it was just it's meant to show
accompanying images as I talk in hopes
that people would find me interesting
and stay engaged speaking to me so it
has a Raspberry Pi with a touchscreen I
just plugged in it so the Raspberry Pi
is just behind the touch screen there
and on that PI I have a no js' server
that sins and it has a microphone in a
sends audio to the Google speech
recognition API and then it sends those
key words to the Google Search API which
I have a custom image search criteria
set up so then it would show an image
correlate to what I'm saying so I liked
that it was responsive to the
environment but because I was using
these api's I still needed Wi-Fi so that
I kind of wanted to get rid of like the
Wi-Fi dependency on future projects and
a side note I didn't realize I was
really working with machine learning on
this and this is relevant cuz I'm gonna
talk a lot about machine learning so I
just wanted to point out that ironically
I didn't realize always working with
machine learning I just thought you know
until it really looks back at my project
I know that's a bit naive
but I also feel like machine learning is
so pervasive you just sometimes feel
like you're using api's but you're
actually using machine learning because
I just thought I was sending data and
get a response so my current project so
I wanted to take the things I liked
about other projects and go in this
different direction I wanted to be
responsive to people around me but it
will it increases my discomfort unless I
can conform to social norms so what that
means is very innocent flashy and
attention-getting when I'm alone when
I'm off by myself and it only calms down
and eventually turns off when I'm
sociable and surround myself with other
people so it's less about
purposely attracting people it's more
about me wanting to get rid of the lies
as fast as possible
and I think I might have convinced you
that with this progression of work that
my progression of work through my
projects it was a convincing reason for
me to build this Nexus but I've misled
you a bit there was actually like a
really pivotal moment that I decided to
build this so like I said I work at
Microsoft to do in Berlin and I started
this job almost a year ago and I was at
this really first time working at a
really big tech company and I was very
insecure because I'm not I don't come
from a computer science background and
everyone else had like masters and been
doing computer science since they were
like 12 years old and folks started a
machine learning study group so we were
doing a big online MOOC course and I was
falling behind from the first study
group session and I felt really insecure
about that but I wanted to prove mainly
to myself that I could learn all these
concepts
do it in my own style and for me that
meant I really need to build something
physical like I have a physical use case
that was relevant to me and my interest
and this is so the real reason I built
this project is because I couldn't keep
up with a machine learning course at
work um so there was a couple courses
that work for me the first one is this
machine learning for artists course by
Jean Cogan he's really influential
creative machine learning artist and he
teaches at NYU but he puts all his
course works online for free and so I
just watching the videos for the course
it really opens my eyes to what artists
and non-traditional technologists are
doing like playing Tetris with fate your
face and it really inspired me and gave
me a lot of ideas on how I could
integrate machine learning into a new
project and then because I'm a
JavaScript developer of course like I
wanted to do everything in JavaScript
which in hindsight was kind of awful
because if it would have been a lot
easier just cease Python because there
was already stuff and tensor faux faux
Python was already pretty well
documented for the PI but I don't like
to make things easy for myself if you're
not familiar with a tensorflow it's the
library for training and running machine
learning models mainly in the browser
where you can also run it in node and
this is dan Schiffman from the coding
train and this is honestly how I wanted
to use sensor flow GS his videos are
great and he just basically kind of
plays around with tensor flow and that's
how I learned so when I started this
project I knew I wanted to use
JavaScript to do the machine learning
but there was two things I wasn't sure
of first of all I needed to run the
machine learning models in node because
I would be running on a PI and not in
the browser and then I also needed to
run these models and note on the PI and
the model I wanted to use was this model
called tiny yellow and yellow stands for
you only looked look once and it's
called tiny because it's one of the
image recognition libraries for the
browser and since the flow but yes for
the browser it wasn't set up for nodes
so the first thing you wanted to do to
see if I could get this model running
just a node so to do that I just luckily
just had to do some minor code changes I
just bought the repo for this project
and then I needed to replace use the
attention flow node specific package and
then store all the models they were
originally in the crowd a cloud I had to
store them locally changing like
import/export statements to require say
minutes things that you do for node and
I couldn't use like the web camera media
devices API I needed actually load the
file in node and then send it to the
model so I was able to do those things
you can see I was so happy I sent out a
tweet about this and yeah you can see it
sees it like thinks deceased ties which
I'm not sure why but it also saw people
which what was in the photo that I was
sending it so I got that done that was
pretty exciting but I needed to actually
get it didn't matter if I got it running
if I couldn't get tensorflow is running
on PI and I didn't do too much research
into this beforehand but I got pretty
lucky that the tensorflow J s team was
actually making tensorflow J s like the
same week I started working on this they
had a branch that they made the
architecture for the PI which is a ARM
architecture they were making changes
and they were going to allow it to be
ran there so it wasn't released in yet
but I was some help from my colleague at
work I was able to just clone it and
like use this package called y'all y'all
to link it to my project and as you just
I could build a tensor flow J s on the
PI and it worked
and I think this is like probably one of
the first instances of actually building
on that pie because it was like right
after they added identify so I was kind
of cool I was doing something like
really bleeding-edge and I have a
disclaimer that I haven't touched this
since I did this so it's running on
tinder fudge is 0.14 too to give you
context the latest release is 1.2 0.8
and they're also prepping release
candidate too but you know what I did
this months ago it's for my project
it works like really I look down to make
sure so it works really reliably and if
it's not broken why fix it like I have
other things like in four hours into so
yeah so just disclaimer if you want to
try this yourself I'm using an older
version of tin so fledge is um so yeah
my two biggest hurdles we're done so I
could start working on the LED necklace
and as a maker I tend to just buy in
bulk and I always see cool things that I
like and I buy things a lot and I have a
huge drawer of just like extra things so
a while back I bought this kit it's like
an intro soldering to kit called blinky
tiles
it sees just shapes that you can solder
together and it comes with this pre
program it's meant to be used with this
pre-programmed
micro control that comes with it but I
really like the aesthetic of it and I
actually had played around before with
just using their micro controller to try
to use something that you're supposed to
create and set on your desk into a
wearable itself so i had made this
before this got completely destroyed
when i moved to berlin so I just took it
apart but I didn't want to use their
preset controller so um I feel like
you're gonna hear me say luckily someone
had did this but this is kind of like
how it goes with making luckily someone
had controlled these blinky tiles before
with a different Arduino and it's
basically using if you're familiar with
stage lighting and stuff there's is
called DMX lighting but they had used
this library called DMX simple and I
could just kind of see the code that
they had written for their Drina so I
was just able to kind of know what
microcontrollers they used and I just
they used a TTT board so I just bought
that exact board and this is just like
my test of making sure that I can
control with my own
Adreno and it worked and then I just set
out on designing what I would design I
wanted to make you can see like the
design on the left is not what I went
with I went with the one on the right
and tomatoes I just I have a mannequin
and I just pin a I solder on the
mannequin I just spin things out and cut
wires and it's super tedious but I just
do it on my mannequin um so I created
the LED necklace but then I needed to
send that image data from the PI to the
jury no to control the Y's and I didn't
want to rely on Bluetooth or anything
that could go wrong so I just decided I
wanted to hardwire it and for that I was
going to use serial ports I never sent
any messages between devices with serial
ports but the PI there's two pins
there's one for sending and receiving
and then I could send the data from the
PI or my data pins of the Drina so it
seemed pretty straightforward but I
struggled a lot so I was trying I had a
lot of problems just getting the serial
on the PI to work so I had trouble this
if you look at C on the Left I just have
a wire going into the send pin to the
receive and this is called a loopback so
it's how you can test that your serial
port is working and I spent like a week
and had like desperate tweets like
trying to figure out what was going on
and because you know it's like the PI is
supposed to be super kid friendly but I
had a lot of problems with this and I
basically just I had to change like
disable a bunch of stuff configuration
on startup and I wrote this huge gift so
I would like potentially remember how to
do this next time but yeah I basically
had to disable stuff like on startup
every time so it's like has the Pens
like active um so things were not going
I figured out things but things still
weren't going great I had some problems
that I needed to deal with before I can
move for further with like connecting
the two so as you can see like that is
like bad it's not supposed to be
flickering like this and at the time
like yeah I'm not like an electronics
like expert I had a hunch it was like
power issues but it really was like
outside of my abilities at the time so I
just switched boards with another board
I had and I switched from the teensy to
an Adreno called the Adafruit mini Metro
and I got lucky that this board actually
worked with that DMX simple library
because it only works with a few but
sometimes you don't learn things until
after and that's okay I got really lucky
but I also think it was it was a power
issue where like the teensy Borg is like
runs on 3.3 volts but it's five volt
tolerant but the mini Metro is has a 5
volt output and the blinky tall's need 5
volts so it just happened that it
matched that better but really it was
because I had one lying around and I got
really really lucky so I fixed the power
issues but then I had another problem
when I was running the machine learning
model after about 30 minutes the this is
a CH top to monitor things the memory
would fill up and then the swap memory
the Phillip which is really bad and then
it would crash because it had it had a
memory leak and I like come to my
program looking for like memory leaks
and I always just stopped I could just
see there was like you know a node I
have called at Jayenge well do you
remember this project I was using the
same SD card and Raspberry Pi and it
turns out this project had a memory leak
it was so it was running at startup with
I had it running with p.m. to boot and
yet
this one had a memory leak but because
they were both called a PJs it was kind
of hard to debug yes so I figured out
that and I turned off the other no js'
half elf is running and then it was okay
yeah so I I got those bugs fixed up so
then I could get back to actually
sending this data from the PI to the
lights and then how did another the
decision to make so what formats would I
use to send data from the PI to the
lights and as a JavaScript developer I'm
like hey I just use JSON every year I'll
just use JSON here and yet JSON on
Torino is a thing there's a library um
and it worked but in retrospect this was
a really bad decision because adrenals
are tiny they're very like limited and
resources of memory wise and the lights
would pause while it was processing and
parsing the JSON it received and I
thought I was so clever for using JSON
on a Adreno but it was just a really bad
decision on my part so I had to redo
this and this is code from my nodejs
app so I switched to I had to think
about what really the Arduino needed I
didn't need an object with like
information about all the things that
the camera was seeing I just needed
really to know how many people it was
seeing so to do that and to do that I
only really needed a byte I needed a
number so I condensed it down and I'm
just sending three bytes now to the
adrenal so I have just a starting and
end by just so on the drain oh it knows
when it because some over serial just
gets a lot of garbage and so it knows
these bytes is getting an incoming
message and then the second the one in
the middle is just the number of people
and the tiny yellow model only can
categorise up to 20 things so I wouldn't
I know I knew that a byte would be
sufficient for that so
Oh finally I could send information and
then it was time to work on programming
the lights and this is just a lot of
like non JavaScript C++ ish
Audrina stuff and thankfully like
Charlie showed a little bit of Adreno
cosy so you can see there was like a
setup and then there's a loop and inside
this loop I have some methods that are
checking one is receiving you can know
if it's receiving a message and things
and this is I just want to show a
snippet of code so this is some code I
wrote that when it knows is receiving a
message and they can call this like set
program method and basically I wanted to
show that when there's no one around I
have this like it's really chaotic and
then as the people there sees more
people then it goes to more gentle light
program and then eventually it goes to
dark mode once they're sixty people and
if you're interested in looking at
adrena code this is the link for that I
also have the links to my project at the
end if you don't catch it in time and so
I had it sending things but then I had
another problem because this is like
this is how Hardware goes so I had just
been testing it on my mannequin but then
when I actually put it on myself it was
having some fit issues so like as I
moved it was like flashing because like
some of the wires were like shorting out
and I originally envisioned putting all
the tech and just like a sani pack not
really not realizing how big a pie and a
battery pack and everything was so it
was no way that I was gonna fit and look
cool and I have fanny pack um so I had
to resolder something so after I started
stopped crying about that I just
restarted and that was fine and then I
decided to just kind of strap everything
to my body on the skirt oh and this
actually works quite well
and solves the problem of like too much
stuff um and then I decided to add one
more feature that hadn't really planned
for so sometimes I get really
overwhelmed wearing this and I just want
a way to turn it off completely and give
myself a break without having to be
around people so I added this button
ring because so I have two things I can
do with the button ring um the first is
I can flip the light programs so it's
actually when I'm around no one it's off
and then as more people come around and
gets more chaotic so I just completely
flips them if I do this short button and
then it can come back on and then when I
just get completely exhausted by
everything I can do a long button press
and I can just completely turn it off
but then I can turn it back on just to
like it's kind of like my escape hatch
if I just can't rip this off of me and I
just and that's it
this project was hard and I got
demotivated a lot it's I originally
envisioned doing this in three months
and it took me eight months not because
I was working on it constantly but
because I took like some month-long
breaks because some of these problems
like the messaging and the power issues
and things like recently wore me down
and of course as always stuff I could
have done differently but I'm really
happy with what I've learned and I'm
excited to continue doing machine
learning things with wearables and other
projects and I love to talk about non
tech stuff dogs fashion all the outfits
of the day any of those things but we
can talk about tech to just if you're
around and you want to say hi just I
know I can look unfriendly sometimes but
I'm actually friendly I'm just nervous
most of the time so say hi and that's it
thank you so much
[Applause]