[Applause]
okay thank you thanks to that yes well
thanks everybody for being here well
this morning or this afternoon it's 12
so I don't know how but so this title is
this talk is about textile based
interactions in JavaScript so this talk
is not really about what you should do
or what you will do there's more about
what you can a lot of the times when
we're are conferences we talk about the
future of interfaces like with react or
web components or the new web api's or
things like that but this talk is more
about the other side of the interface
the actual thing that you can use to
interact with it so I already got
introduced but my name is Charlie Gerard
I'm a front-end developer I live in
Sydney in Australia and outside what I'm
going to talk about today has nothing to
do with what I do for work but outside
of that I'm also part of Google
DevExpress delight tech speakers I know
that there's a few around at this event
so if you have any question about how to
get into these programs or what they are
feel free to come talk to me afterwards
yes so what we're going to talk about
today is not about me but it's about
textile interfaces so what does it
actually mean just a quick definition
it's using textile as an input to
interact with interfaces and or devices
so the main inspiration for this talk is
actually something that I came across
about three or four years ago
that's called a project jacquard so for
those of you who don't know what this is
this is a collaboration between Google
and Levi's so the brand of jeans to
create a connected jacket so instead of
just spending some time explaining tick
to you I'm just gonna play the I'm just
gonna play video okay why so there is no
video great so there was a video five
minutes ago and now there's no video
anymore
so I'm just going to talk about what the
video shows so Google partnered with
Levi's to embed a jacket with
electronics so that you could interact
with your phone while you're riding a
bike so if your phone rings and you're
supposed to you don't want to take the
call you could just
touch the fabric and it would it would
send a signal to your phone and you
would have you be able to interact with
a few different interfaces with that you
could switch the music or get the next
direction from google map just by
interacting with the jacket and after I
saw things like that I was like to me it
was brand-new at the time like three
four years ago I didn't really know how
to make this and I go be like I got
really interested into understanding and
this talk is about how you can actually
build something similar while just being
a JavaScript dev so why am I actually
talking about this there's a few reasons
so the first one is the evolution of
interactions we still most of the time
interact with technology using a laptop
or our mobile phone but there's more
like we can see some people start using
voice interaction people have tried VR
or AR and I do believe that there is
some potential in textile interactions
as well and as technologists I think
it's our role to we don't have to build
these things but at least keep in touch
with what's going on I know the reason
is that it's a lot of exciting
possibilities hopefully you'll feel the
same way at the end of that talk as well
but I know that in technology well in
our job we build things that people will
use for money and things like that but
to me technology has a lot more exciting
opportunities and that's why I love this
industry and I'll show you a little bit
what you can do with textile and then
finally and that's a personal challenge
is to identify what you can come across
online when you look at innovative
innovations so every time that I come
across something that I don't really
quite understand I don't want to just
look at it and not figure out how to
build it myself
I don't want to think that you have to
have a PhD to build certain things or
you have to be backed by Google or
Facebook I believe that everything that
we see we might not understand at first
there is a kind of cheap hockey version
that we can do at home without having to
think that you have to work for Google
for to do stuff like that so now we
let's talk quickly about the potential
of textile so first of all it's flexible
when we think about our laptop or our
phone is quite rigid if you try to bend
it well it's going to break but with
textile when you embed electronics in it
you actually can well it's quite
flexible so you won't you have less
chances of breaking it easily
very lightweight most of the time we
don't even realize that we're wearing
clothes by the way it textile doesn't
have to be clothes it could be this
textile on the table here you have
curtains at home or towels or blankets
or pillows or whatever it's in the end
we don't really look at it that way we
kind of just take it as well it's
textile it just covers our body but it's
actually a lot of potential there as
well and it for it being lightweight it
means that we would maybe be able to
kind of walk around without having to
carry a heavy phone or a heavy laptop I
started speaking at conferences last
year and even if I try to travel light
at the end my luggage is or my backpack
is actually quite heavy because of the
laptop and the phone and stuff like that
so being able to embed that kind of
interactions into textile means that we
could actually travel even a lot lighter
it is quite modular so if you want to
build your own things you can add
different components you could add
lights or flex sensors or all that stuff
and you get to build it yourself so you
rather you get to decide where you want
it to be if even if it was on a t-shirt
it could be the shoulders or the back or
whatever so you don't have to actually
buy something as it is like the phone
you know I mean well Google I think was
working on a modular phone but in
general when you buy a piece of tech
it's kind of like you know you just buy
it and that's the way it is and you
don't get to decide what parts you want
but by building your own textile
interfaces you get to decide the
interaction and the sensors that you
want to embed in it it's responsive to
adaptive touch so this is in this talk I
kind of had to made up make up some
words because so adaptive touch is well
that I made up because I didn't really
know how to explain what I wanted to say
so adaptive touch what I mean Dada is
that so when you for example for the
jacket by Google you interact with it
and it sends a signal but adaptive touch
means that you don't have to press it
very hard to then send a signal you can
even just slightly touch it and you
could have different interactions based
on how you decide to touch the jacket so
this is what I mean by adaptive touch it
is very low-power if you've been playing
a bit with Arduino or Raspberry Pi
depending on the amount of components
that you have in your circuit it's not
something that you'll have to recharge
every day like you do with your phone
and then finally it's pretty cheap so
I'll cover that a bit later as well but
if you compare the price of
a Macbook - something that you can do
well obviously you will do a lot less
with interactive test I'll let you do
with your laptop but it's to get started
in that space you don't have to spend a
lot of money okay so how does it
actually work at the core of it is
conductive thread so this is normal
thread that it makes that is mixed with
stainless steel so it means that it
carries the current through a circuit
the same way that Y is do so again well
I don't know if you've been playing with
Arduino but usually you connect your
microcontroller and your sensors with
wires in between and you can do exactly
the same with conductive thread the main
difference though is that usually wires
are coated like they're wrapped into
some kind of plastic so you don't really
interact with them whereas the
conductive thread it's bare and as your
body conducts electricity as well it
means that when you use conductive
thread in your circuit you can know the
kind of like electric charge that goes
in a circuit and as soon as we interact
with it so as soon as you touch the
thread you have a change in this
electric charge in a circuit that you
can track and then use that as as an
input for your interfaces and devices so
this is like the latest version of the
one of the things that I built so I'm
just going to go through a little bit
about how it works and the modules that
you have to use to do that so I built it
three different times that's just the
latest last one you don't have to build
it that way if you're interested there's
a few different microcontrollers that
you can use or things like that but at
the core of it is a flora board the
reason I used that one is because it's
good for wearables if you look at all
the holes around the board they are big
enough to have to be able to put a
needle through them so at first I didn't
know and I worked with another board
that wasn't made for wearables and I
realized a bit too late that I couldn't
even put the thread through the hole so
they was trying to use this so if you
want to do wearables I think the flora
is a good one then I I did a Bluetooth
module because I didn't want to have to
be tethered to my computer obviously I
want to be able to walk around and be in
the street or whatever and communicate
to my phone or my laptop via bluetooth
then you have high level resistors I'm
not going too much into that but to be
able to filter through and get data from
the amount of touch the capacitive
sensing you have to use resistors then I
have two pieces of thread and finally a
battery to make it totally portable okay
it's 48 dollars so when you want to buy
all these components online it would
only be about 48 dollars so when I was
preparing this talk I realized that
Google actually partnered with another
brand called Santo ha to make a backpack
so now they also have an interactive
backpack but the thing that looked at it
after and I was like 685 pounds are you
for real and I'm like normally you do
that for 80 48 dollars
so it is the instead of instead of
having to buy something that is super
expensive you could actually start
playing with that technology and put
like interactive stuff in your backpacks
or accessories or whatever for a lot
cheaper ok so in terms of code now we
assembled the whole auto Hardware
together so it can't you can't do
everything in JavaScript so that's a bit
of Arduino but I'm gonna go through it
line by line I mean red block by block
so it is a that's pretty much all the
code that you need to upload to the
Arduino what's going on here is that we
accentuate the Bluetooth communication
we use there is a capacitive sensor
library for Arduino so you don't have to
write it yourself you just create two
capacitive sensors for the two pieces of
threads that I used and we use a
threshold so as I said in in circuit
there is always an electric charge but
you want to be able to track when you're
touching it so you put a certain
threshold so that when it's over this
one is a thousand but you could decide
whatever you want really based on the
attraction that you want the as soon as
the value coming back from the Arduino
will be over a thousand it means I'm
touching a piece of thread so do
something in the browser the main the
setup function that runs once when you
upload your code to the Arduino you just
kind of reset the value of your
capacitive sensor you set the Bluetooth
mode to data that you know it's with the
library so that you don't yeah it's fine
and then the loop function is going to
run continuously and what you do you
just read the value from the two pieces
of thread and then as soon as they're
above a threshold you send that via
bluetooth so in just a few
of course you have the capacitive
sensing running on the Arduino and now
what we have to do is deal with it in
the front end so as i said i'm using
bluetooth so i decided to be able to
connect to the browser using web
Bluetooth so what we're gonna do here is
we have the name of our of our device
that's called the Adafruit bluefruit le
you can reset that name if you don't
want to call it that way but that will
be on the arduino side you so for web
Bluetooth it works all the Bluetooth
protocol works with services and
characteristics so they have certain new
ideas that you have to use to be able to
connect to and for web Bluetooth knew
you for security reasons you have to
start the connection to a device with
user input so you can't just run your
browser and it will connect straight
away so let's say on webpage we have a
button and as soon as we click we're
going to request nearby devices and we
add a few options to filter down the
device that we want to connect to so
that would be our Adafruit module and
the services that we want and then
finally we connect to it we get the
primary service we get the
characteristic and at that time here
we'll start to get the data from the
arduino so we call the start
notification method on on the
characteristic to start getting the data
every time it changes from the arduino
so every time that the data that yes
that the data from my touch is changing
i actually have an event listener and i
get that and i have to decode it so at
that moment here we're just getting a
certain value in the browser and then we
decide to do with it at the moment it
will just be an integer and this is
where we could to decide the type of
interaction that we want to have so
let's talk about the interactions if we
consider again that have a piece of
fabric and i have two conductive threads
on it we could create an interaction of
swiping down i know if i get the value
of thread one at thread two in the order
obviously I'm swapping down and then you
could have reversed you could have
swiping up so if I touch the bottom
thread first and the top one later it
means I'm going up then you could have a
tap or a touch times two because we have
two different threads so if I'm only
touching the first one maybe do
something if I'm only touching the
second one do something else then you
could have a double tap so if over time
I touched if for example in half
second I get value from the bottom
thread twice it means I'm doing a double
tap and things like that
and finally we could have pressing the
both threads at once so with only two
pieces of conductive thread we could
already have seven possible interactions
of ways of things that we could do in
the browser or with a device so I'm
going to try and show you how it works
so now all the demos are going to be
live because I think I couldn't really
bring everything but so the first thing
that I built was just I tried to augment
my couch with conductive right so I
wanted to be able to switch songs on
Spotify by just touching the thread so I
don't know it looks like videos and not
okay so I don't really know okay so well
that's not the slider you can watch
later so what was supposed to happen
here is that I when I'm touching threads
one by one I would switch the song to
like the previous one or the next one
and if I'm touching them if I'm touching
a piece of thread long enough the volume
goes up or goes down so yeah if you're
interested in watching it's like a
10-second video anyway but you can do
that later I have no idea whether video
work but so the second one I'm gonna
dive a bit more into is this example
that was when Google was working on
project jacquard they also showcased
that at a conference so people could try
the fabric and have that visualization
and at the time I only knew that I was
only thinking about simple interactions
so like oh if I touch one piece of
thread do something or the other one but
this one I was quite interested because
I was thinking how do they get the
location of where I'm touching and
things like that so I built a very like
it's not as good right but the it kind
of works so I'm gonna try to demo it
live and then I'll explain a little bit
how it works because I feel like in
terms of reverse engineering that was
very interesting to work on so now is
the time where hopefully I tested it
like many times so okay yes okay so I'm
just gonna run my server so for this
particular for this particular example
I'm not doing Bluetooth because I had a
lot of issues with Bluetooth at
conferences so this one is tethered to
the
that's my first prototype so if I touch
yay okay so if I saw well its reverse
for you the where I'm touching baby
Seacliff I'm touching on the bottom left
I should do yeah bottom left if I go to
the top right yeah well that was a bit
of a bug here but if I go in the middle
you can actually well that's just a
visualization it doesn't really do
anything but it means that it's just a
piece of thread like I could just like
flick it around what this wires
everywhere so I'm not gonna do that but
it's very like I can fold it like I it
was in a really tiny bag in my luggage
and well I I only did a visualization
because I wanted to reproduce what
Google did but you could really do
anything with it you could turn it into
a MIDI controller for life Cheers
but yeah I know I've been trying to join
last year's for a while but they don't
think I'm cool enough so maybe if I make
them a textile media controller maybe
that will work
yes so now let's go just a little bit
quickly into how how many it happen so
instead of just sewing threads just like
horizontally just like separately I
turned them into a matrix so agreed and
one thing that is really important is
that the thread should not touch so when
a piece of thread goes over the fabric
the one that is on the y-axis should
actually go under the reason for that is
because if you're touching one piece of
thread and that piece of thread touches
another piece of piece of thread then
your reading is going to be wrong so I
do edit it in terms of a matrix because
all of a sudden your threads can be
changed into coordinates obviously you
want to be mapping where you're touching
the thread to a location on the screen
so your vertical threads could be X 0 X
1 X 2 and then y 0 y 1 y 2 for this
particular example I only used six
pieces of threads but you could see that
because I did it manually and there
already took me a while so if I had a
swing machine I could actually do a lot
bigger and better stuff but what is
going on after that is that when you're
touching for example the top left corner
you get data so you could be like okay I
get a value from my first thread my
first vertical thread I get a value from
my first horizontal thread so I could be
that the
is then x0 and y0 and then with
coordinates you could you could add that
to canvas in the browser and stuff like
that so for example that as I showed
here as I'm touching the middle thread
and the bottom thread like vertically
presently I apply that to then my
visualization in the browser yes and
then you could actually also have that's
the second made-up turn temporal
gestures it's not lip smart I was I
don't know how to call it so but if it
sounds madam I'll take it
so here when I showed you the demo I was
just touching one at a time and it was
impacting the visualization but you
could assert you could program swipes
like I said before and the way you would
do that is you would track data that you
get over time for example over half a
second or over a second you would push
everything into an array and then every
second or every half a second you would
check okay if the first value in my
array was X 0 Y 2 and then the last
would be X 2 y 2 then it means I'm
swiping right and you know the opposite
for swiping left you could do up and
down and things like that so then you
could create a lot of different gestures
that would mean different things so even
with only six pieces of threads you
could actually have quite a lot of
different ways to interact with an
interface and okay this is not me but I
wish it was me and that uses the same
principle so but it's a lot it's not
done by hand you can see that it uses a
machine and the reading is really really
good so it is a great agreed as well and
the where you're touching on an X and Y
axis is then showed by lighting up
certain squares on the screen so I put
the link to the blog where there's like
a lot of different other interactions
that you can do a six style so you can
check that afterwards but ultimately
that's kind of what I would like to be
building because it's also like a super
lightweight and it's like on your skin
so you could really just interact with
interfaces with that so that's really
cool different a few different
applications this is where I wish the
videos were working but okay so
accessories so as I one of my prototype
that I built was to augment a couch but
you or you have some brands that are
already working on accessories
wanted with interactive textile this
brand has a pair of speakers and instead
of having buttons
it's a textile that you can press a
certain way to trigger our next song put
the volume up or down and things like
that you can make musical instruments so
so I don't know if you heard of the meme
gloves I think it's a project that
started about seven or eight years ago
by an artist called Imogen Heap and she
also has a team of engineers that have
been building these musical globes so on
top of having conductive thread it's
also gloves that are embedded with
flexed senses and position senses so if
you are interested I really I really
advise me to actually watch that video
after it's really really cool so by just
being on stage and moving hands a
certain way you can change the frequency
or the beat or the instrument and and
that's a good total new way of having of
playing music of having keys
accessibility so this project is it has
the same concept where a pair of clothes
is embedded with Flex senses and it's
just a team of students who have been
working on training machine learning
model to recognize to be able to map
sign language to actual spoken words so
this is a prototype but it's something
that you could do as well as JavaScript
developers and hopefully you feel like
you know that hopefully it makes a bit
more sense now that I explained to you
how it works and finally and I realized
that if you just a couple of days ago
you can store data into textile like so
the way this one works and I wish I
really wanted to get it to work for
today but I didn't have the time it's
like if you magnetized conductive thread
you could actually encode bits into it
and you can just like send messages so
this particular demo is actually around
gesture recognition with just magnetized
thread so it's not in the camera it's
reading the magnetometer in your phone
and when it senses the change in
magnetic field you could actually have
encode like build different types of
gestures just by having a little piece
of magnetized conductive thread on your
finger and that works through your
clothes so it means that you could
actually have your phone in your pocket
and you
by doing it putting your hand a bit
closer just trigger things and if you
have a piece of thread on all of your
fingers you could really map so and just
use quite quite nicely and and it's
really cool that you don't have you
don't have to have a PhD to do things
like this like I was I'm quite convinced
that I could figure out how to do this
it's gonna take me probably a bit more
time but I was googling yesterday like
how to calculate the magnitude of
magnetic fields and I was like oh my god
you know it's so cool I didn't figure it
out yet I need to find the formula but
you have a web you have the sensor API
in the browser that allows you to have
access to stuff from the magnetometer
and your phone it just doesn't it just
doesn't give you the magnitude so you
have to calculate yourself so maybe I
will come back one day and I would have
figured it out then of course you know
I've been talking about the
possibilities and how excited I am about
this but there are limits so I have to
mention that as well it is partially
waterproof we're talking about textile
so obviously you know if you wear
something on a jacket and it rains you
have to be a bit careful but you you can
add a certain coating so there's a spray
that you can add on conductive thread to
make it waterproof a connective thread
works for two years maximum so I think
it works like well for two years and
then like that it doesn't work anymore
all of a sudden but it means that the
reading might change over time and
finally quite interesting the
capacitance changes with external
factors and that will impact the way
that you're going to code that will
impact the threshold but what it means
is that depending on the weather if it's
humid or dry the reading that you're
gonna get the capacitance is going to be
different so your threshold you know
mine was a thousand but if it was like a
dry winter day it might have been
different so depending on what you
actually want to build if it's a jacket
that goes outside you'll have to take
that into consideration if you have like
a couch at home while you will be less
affected by external factors so it
should be all right but yeah so that's
to me that's the main limits that I
found also if you want to do this I
would not advise to do it by hand
because sometimes it takes hours to just
put stuff together and that was very
frustrating for me and a few extras so I
made most of my experiments with
conductive thread but it's not the only
material that you can use so there's
actually quite a lot you can use
conductive rubber stretches sensors so
you can
detect how the rice conducted but also
how much you stretchy so that's pretty
cool I don't know what I would do with
it but that's nice the second one is a
piece of conductive fabric so instead of
just having the thread you could
actually cover a bigger area of textile
by cutting fabric into different shapes
or or just cover or have an entire shirt
that's conductive instead of just a you
know piece of it
you have pressure-sensitive conductive
sheet where instead of just having the
connectivity you could have the pressure
how much you're pressing on the sheet as
well and then finally you have stretch
resistive fabric where you can read the
amount of stretch by by stretching it
and the what that would allow you to do
as well is to play with something that I
really want to do
that's called electrical to typography
where instead of just so it's about
finding coordinates as well but for me
to find coordinates and map them to a
screen I made a grid of thread and I
converted that into coordinates but if
you have a sheet of fabric there is
actually a way to try and find
ultimately where you're touching it by
doing electrical topography so you're
adding sensors all around the sheet and
they're going to give you a certain
value of capacitance and as your body
touches the conductive sheet it's gonna
it's gonna change the electric charge
and by reading the data coming in all
sensors you can approximately know how
close you are to certain senses rather
than others and what I really like about
that is that it's not something that I
thought JavaScript devs would be able to
do like usually just build websites but
now I know that by just going with a bit
of hardware you can actually come up you
this probably there's probably a formula
to actually calculate this I just need
to buy the sensors and then figure stuff
out and and just play with stuff like
this which i think is really cool and
then finally before I finish some final
words of wisdom that are not by me by
never entertain imagination is more
important than knowledge for knowledge
is limited to all we now know and
understand while imagination embraces
the entire world and or there ever will
be to know and understand and I read
this quote while I was reading a really
short essay called the usefulness of
useless knowledge and what I really like
you don't have to believe me
you when I say that this is really
interesting and important to look into
this
but if Einstein says that it's important
then you know you kind of have to
believe it but the reason why I'm saying
that is because I feel like sometimes as
developers we're not really using our
full potential in what we can build I
think we might we might forget that a
lot of people who are not devs have to
just take the world as it is and just
buy the stuff that other people have
built without really having the chance
to make it more personal or you can
imagine but you can't build where as
developers even just JavaScript
developers it doesn't you don't need to
to like have like a master's degree or
know all the languages in the world it's
just about being curious and wants you
not only imagine what you could do with
tech but you also get to build it and I
feel like that's that's extremely
powerful we don't know what kind of
interactions are going to be in the
future but we can actually get to tinker
and maybe even you know get to influence
how it's gonna be and that's like one of
my favorite part of take if tech was
only about building react components at
work I might not want to be there I mean
it's cool too but I the fact that you
can use technology to build a lot more
and whatever comes to mind is really
really cool to me so that's all I had
today thank you very much thanks a lot
the organizers for helping me set up
that was a bit that was a bit of a
hassle but thank you everybody for
staying with me for half an hour and if
you have any questions about that or if
you have any ideas of stuff that you
want to build if you just want to talk
about that I will be around for the next
two days and otherwise I will be sharing
the slides on Twitter there's a lot more
resources and the videos will work
better than they did here because they
didn't but anyway thank you thank you so
much that was it
[Applause]