[Applause]
so last year my colleagues and I did
something pretty exciting using only the
browser a little bit of JavaScript and
something called the Web Audio API we
were able to decipher brain waves now
that sounds a little pretty a little bit
sensational but it's mostly true so
today I'd like to tell you the story of
our journey through the lows and highs
of web audio on a quest to decipher
brain waves in the browser so this is
story of discovery it's about finding
out what the web can offer it's a story
of exploration about pushing the
boundaries of web technologies to differ
to make a difference and last it's well
it's a story that has a really cool demo
at the end so stay tuned but before I
can tell you that story have to tell you
this one so my name is Braden Moore I'm
a full-stack developer at an Australian
medical technology startup called C R so
C R provides Australia's largest
epilepsy management and diagnostic
service in fact the tagline of our
corporate website is epileptic diagnosis
made easy now I didn't know anything
about epilepsy before I started working
at C R and I guess that's kind of true
for a lot of people so let me give you
some of the facts so epilepsy is a
chronic condition that affects over 6
million people worldwide now the
symptoms of EPs can vary from person to
person but could range from loss of
awareness loss of bodily control to
full-body seizures now these events can
vary in frequency summer times happening
multiple times a day up to once a month
or even just as rare as once a year but
the results can be devastating
regardless and there's no known cure for
all types of epilepsy but with their
diagnosis some of the symptoms can be
treated so the way that epilepsy
diagnose is by reviewing a patient's
brain waves in what's called an EEG or
electroencephalogram so each of these
traces they represent the electoral
signals from certain parts of a
patient's brain captured using
electrodes on the head so a clinical
team uses EEG just like these to help
provide a diagnosis now the service that
we offer to see it looks like this
patient comes to our clinics where they
get connected with his electrodes to
monitor their brainwaves over the next
seven days we send them home they can
live their life in a comfort of their
own home while being monitored
then after that time they come back to
our clinic they get disconnected and
then we spent a few weeks looking
through our data and looking for
something that might give us a diagnosis
so we've got seven days with the data
what is it that the scientists are
looking for well look sometimes they're
looking for the obvious they're looking
for something like these minute long
seizures that are pretty easy to capture
on tape but if these are very rare
events happening once a month or even
once a year it's not guaranteed that
will capture that during the seven days
of getting wanted so more often they're
looking for something a bit smaller
they're looking for something like a
needle in a haystack well in epilepsy
there's something called the
characteristic signal of epilepsy which
is called a spike wave so it's these
little spikes and little waves that
around half a second long each and so
while a patient might only have a
seizure once a month they might have
dozens of these little spike waves every
day and so it's these needle and
haystack so they're scientists are
looking for in fact you're looking at a
whole kind of run of little spike waves
right there okay that's great but what
does a web developer have to do with it
I mentioned before that our clinical
team gets this data and then they start
reviewing it but that's not the full
story like every second Plattner company
these days we have our own cloud
platform and it's a platform for secure
management of storage and review of
medical data so and one of the powerful
things about this platform is it allows
a clinical team to upload this EEG brain
wave data and then review it anywhere at
anytime or through a browser one of the
major technical challenges of this
building this platform is the ability to
view or they start on the platform so to
tackle this problem we've built
something that's very creatively called
our viewer to view the data an Emmy
because it's hard and it looks something
like this in fact it looks exactly like
this this is a screen shot I just took
you know a few minutes ago and so it's
this data viewer that our clinical team
uses to review these seven days worth of
data
and to label events and to eventually
deliver a diagnosis one of the big
challenges in getting us scientists to
move from traditional so one of the big
challenges sorry that we face in splat
form is getting neurophysiologists as
they called to move from traditional
desktop based software's into a cloud
platform now
they often neurophysiologist often flick
through EEG data very quickly they're
going for pages a second so flick flick
flick that is looking very quickly
through this because if you want to get
through lots of data you don't wanna be
spending a lot of time on each page the
way they are able to accomplish that is
because they're trained to look at for
patterns they're looking for certain
things like these spike in waves
remember that's a half second thing in a
10 second page that you're looking at
for these pages per second so you're
looking for a needle in a haystack but
what happens when a needle doesn't look
like a needle what happens when the EEG
data that you're trained to look for
doesn't look quite right well in cases
like those those could lead to
misdiagnosis and awful patient outcomes
so this is our problem how can we take a
ETA review from desktop to the browser
so there's one particular aspect of
displaying this EEG that's very
important but also a little bit tricky
and that's called signal filtering so if
this problem of signal filtering that
we're going to explore today so let's
illustrate the problem on the Left
here's some EEG data that has signal
filters applied that's great on the
right is that same data with filters
turned off so clearly there's a very big
difference between these two so why is
this you know where we store it's still
you know recording the brain here well
like most electrical signals when you're
recording something you're not just
recording that brain activity you're
also recording a bunch of extra noise
and artifact so some of this noise can
come from let's say hi if we're gonna
see muscle contractions could come from
like a DC drift or it can come from like
the mains electricity noise so all of
these things aren't brain waves but they
still show up in this EEG so what does
it filter after all well no it's not
that one let me explain what a filter is
by talking a bit about music we all like
music so this is an audio spectrograph
it's a visual representation of all the
frequencies of a piece of audio like
your favorite song at a given point in
time so it has like low frequency is
like the Bass & Drums mid frequencies
like another spoken voice and maybe
higher frequencies like wind chimes or
pan flutes or other you know super
common instruments that you'd hear so
just like audio your brain wave is also
composed of a range of different
frequencies all happening at the same
time
so think about music imagine you're
producing a song but you've got this
kind of bass noise you're not really
wanting in the air or there's too much
pan-flute well one way to get rid of
that is by filtering out like the lower
frequencies or filtering out the higher
frequencies and so for eg there's kind
of three main filters we'd like to apply
you can have a high-pass filter which is
like this it lets everything higher than
the filter frequency pass through
there's a notch filter which notches or
cuts out the the frequent range of
frequencies and the low pass which is
the opposite of your high pass filter
everything lower than the filter if we
can see passes through you can also
combine them you can have multiple of
these filters running at once and that's
exactly what we'd like to do so okay how
can we go about implementing filters in
javascript in the browser on the cloud
well look first could we not have a
filter at all you know the best code is
the code you don't have to write but now
remember a couple of slides back with
filtering without filtering that's it's
kind of night and day right there so
that's a bit of a no-go I mean how would
have been happy with it but so could we
write our own filter implementation well
it's kind of a maybe and because of
stuff like this and and this and all
this stuff so I trained as a particle
physicist not an electrical engineer so
don't know too much about filtering but
I've been told that there's a bit of
subtleties involved in implementing a
filter and I'm sure you could agree that
JavaScript probably isn't the best
language to handle you know complex
mathematical processes so put that on
the back burner maybe could we use a
package you know where JavaScript
developers after all it's what we're
trying to do so how about this one this
is promising
DSP kit it's a digital signal processing
library it seems like Inc process
signals it is not intended to be used
it's good Fairplay to them how about
this one
DSP Jas now actually this one is I think
it's the most starred signal processing
library in JavaScript so you thought you
think this is probably the way to go but
it hasn't had a meaningful update in
nine years I mean I wasn't even coding
nine years ago so I think of this
happened in the web since then so maybe
maybe that's not the way either
so those three options are exhausted now
what are we left with well if you've
read the title of this talk you probably
know what's coming next I want I won't
leave you in suspense so introducing
there's very quiet drum rolls that you
could've just heard them but the Web
Audio API so this is a Web API that's
designed for processing audio it's
standardized while maintained as decent
browser support and it's gonna active
group of contributors and users so look
you're looking for something that works
now Web Audio is probably the way to go
well when I was first starting to use
Web Audio are the Mozilla developer
Network or mdn Doc's were invaluable and
actually just learned at this conference
that many of these Doc's are written by
live GS is very own routes so big shout
out so Ruth for helping me learn this
but let me take you through some of the
basic concepts here so how do you use
the Web Audio API how to actually use it
to process this data
well first you create an audio context
this is kind of the canvas where
everything happens inside this audio
context you create a bunch of nodes one
of which being your input node so this
might be an oscilloscope or this might
be like you're feeding microphone into
there next you can make a bunch of a fix
nodes so you could do things like reverb
or compression panning and many more all
the things that you know greatly with
audio maybe you wanna put reverb on your
IEG just just me you could also and then
you've got to specify the destination so
where is this all getting output to
conventionally that's gonna be your
speakers alright so after you do that
you string all those things together you
know one two one two one and then you
process the entire thing and there you
go stage your process audio okay that's
good but I might have jumped the gun a
little bit here what does an audio api
have to do with processing eg data like
why why did I jump from you know
packages and JavaScript to this well it
all comes down to one of the effects
nodes so the effects nodes can do really
cool things on your data such as reverb
compression padding it can also do
filtering so we had that audio
spectrograph where those filters were
applied you know turns out filtering is
a thing that musicians and music
engineers actually do so it makes sense
that they'd have really well implemented
filters in this API so if we could
somehow leverage these filters and use
them for our energy filtering then we'd
be set okay so in particular we want one
one method that's really useful for us
called the create IR filter method now
quick electro engineering lesson I R
stands for infinite impulse response
it's all you need to know you know
anything about it else about that but
it's the filter you can generate with
this crate I feel to know it crates an
IR filter and that's super configurable
so you can make it high pass low pass
notch whatever you want in there so
here's the plan first we need to make
some way of reading or getting the web
audio to read out EEG data once it's
there we can process this data we can
apply the filters and finally we output
this back under brainwaves displayed on
the screen cool that's all well and good
but how exactly do you read eg with an
audio API I'm glad that you asked you
you asked that question because we're
about to find out so here are the six
steps that you need to decipher your own
brain waves in the browser with the Web
Audio API let's go alright entering vs
code mode now so first we need to set up
an audio context remember this is the
canvas where everything happens next you
need to you know create an audio context
that's as simple as you know calling a
constructor you pass in the number of
channels the number of samples for each
channel and your sample rate on there
now this is an offline audio context and
kind of this - audio context you can use
the offline audio context in the Web
Audio API or something called the audio
context makes sense so the thing about
the audio context is this is the one
you'd usually want to use if you're
processing audio if you're making like a
multitrack mixer or you want to apply
some reverb in the web that's great the
problem with it though is that it plays
back the audio more or less in real time
so it's good if you actually you know
want to apply your effects in real time
on music but if you've got 7 days with
an EDD artist look through you really
don't want to be waiting you know 7 days
for that to process so that's maybe not
the way to go there's another type of
offer of audio context you could use
called the offline audio context so it's
quite similar to the regular audio
context except just taken directly from
and again in contrast with a standard
standard audio context and offer an
audio context it doesn't render the
audio to the device Hardware instead it
generates it as fast as a cannon and
outputs the result to an audio buffer
now that sounds brilliant
so basically audio context is slow often
open to this super fast what we're
trying to do so that's why we're going
with it
cool we've made this audio contacts next
we have to set up our source notes
pretty straightforward create a buffer
source and note that these methods are
all coming off the audio context so you
can't actually create nodes in the web
audio without actually being attached to
this context next up we need to create
and fill this audio buffer so first up
we're going to create a buffer it's got
the number of channel samples sample
rate that we want go to loop over our
channels and then for each channel I'm
going to kind of get the channel buffer
that I just created to remove this is
like an empty empty audio buffer so get
the channel buffer for that channel
gonna iterate over all the samples and
then right here I'm going to you know go
to our channel buffer I'm going to fill
that with pal EEG data which is in the
very credibly named data array so our
EEG data is just in an array of an array
of floats in this data and something
really exciting is happening here so
right there on the left
that's an audio buffer that's audio that
stuff you can process through the Web
Audio API on the right is your EEG data
and in the middle is an equal sign so
what you're doing is you're moving your
EEG data into an audio buffer which
means right now you can put filter and
reverb and compression all this cool
stuff that you really shouldn't do on eg
but you can just like that that's really
cool next you just assign you know two
audio buffer to your source and there
you go you've got your input you've fed
your EG data into the Web Audio API okay
next step we need to generate the
filters well a bit of a bit of
background on this
so the create IR filter method remember
that's kind of super configurable but
it's super configurable in the way you
have to provide certain what's called
coefficients to define what the filter
looks like now there many different
filter types one can use and are all
slightly different from our research we
found out is that desktop software's
usually use something called a
Butterworth filter and that's you know
pretty standard across a lot of filter
types surprisingly you know JavaScript
doesn't have its own generate
Butterworth filter coefficient
implementation so we kind of had to
write our own which looks something like
this and this and this and it just keeps
going so many more pages there's one
more page there's about two hundred
lines of code there
users like complex numbers and polls and
transforms and actually ported it from
MATLAB code one of our colleagues wrote
into JavaScript so it's possible a bit
messy a bit long but there you go so
we've got our butterworth coefficients
once we have that we can generate to the
IR filter by passing these coefficients
in so there you go that's you know six
lines we have our filter nodes next up
we need to connect the source of the
destination so what's happening here is
we're connecting the source to the low
pass the low pass the high pass high
pass to the notch not to the destination
all the way through simple proof finally
we need to process an app with this
audio so we just set up this big audio
graph as it's called now we need to
render this audio graph so step one is
setting what our start point is you know
makes sense that you want to start
processing from the start which is
what's happening here then we need to
call this start rendering method so this
is on the offline audio context you can
do this and it's super cool because it
returns a promise which resolves to your
process to buffer what I'm gonna do now
is you know just initialize where the
output data is going to go and then kind
of do the reverse of what we did to feed
it in gonna loop over all these channels
get the channel buffer and get the here
the channel buffer for those but now
this is full of data instead of empties
before and for some reason here this
channel buffer is returns a typed array
rather than a regular ol array now the
way to convert from a typed array to a
regular array is build a stack overflow
and type how do I to convert from a
typed array to a regular array and copy
paste of the code they tell you right
there so that's that's the simplest way
I found it works cool now we'll return
our process can loop data and there you
go six steps we put our dial we've fed
it through a filter that we process it
this is filtered EEG data right there so
after our six steps we're able to turn
that to that and we've got so much
mileage out of these these images is
like the third time we've used at this
talk so look you know now is what we the
time where I would you know bring out a
demo and show you how it all works and
then get on a plane back to Australia
but you know life life isn't that simple
we there was a few kind of problems with
this and one of the biggest problems is
it was very slow like very very slow so
some further now our scientists they
want to review like four pages of data
second so that's 250 milliseconds per
page okay well to apply these filters it
takes 300 milliseconds so already we're
over budget and then you need to factor
in you know fetching and downloading
this data
you're doing the filtering and then
rendering that all then clearly you know
you're nowhere near what we need to get
that goal
so this is what I call you know just
some minor roadblocks so what's the
problem the problem is this crate IR
filter method you know it takes around a
hundred milliseconds to generate a
filter which is look reasonable because
you have to provide these coefficients
generate a new filter super configurable
from scratch that's okay but you're
doing that you know not just once but
one two three times that's four three
times which is you know three one
hundred three hundred milliseconds the
pain doesn't end there because now you
have lots and lots of things because
every page of data that you're looking
at has to generate its own filters so
one page 300 millisecond another page
300 milliseconds so it just goes on and
on and on so clearly this is not gonna
work so where can we go from here well
look we could use web workers maybe I'm
just putting that there haven't actually
tried it but you know even if we could
get web workers to use web audio it
still has the problem of the brows are
still taking that 300 millisecond so it
might be off the main thread but it's
still that waiting time there use
webassembly maybe you know rust didn't
see those languages are far better
suited to writing your own filter
implementation than javascript which is
what we talked about before but you know
we're so close we're so close for almost
got there we've got our data filter I
don't want to jump ship and go to you
know rust or C just just right now so
maybe could you feel from the backend
you know similar to web assembly perhaps
there's server-side languages that are
better at writing filtering permutations
but there's a bit of latency in having
to you know change the filter fetch some
data change the filter again fetch some
more data it's just a lot of data flow
through maybe is this something else we
can do all right well when you're in the
end and so I'll tell you about a Taylor
to filters now so what is that problem a
problem is this the great IR filter
filter method it's good it's super
configurable it you know represent the
second order filter configurable with
several different common filter types
but it's a bit slow to make so what's
the solution
well the solution is
there's an old proverb to hide a tree
use a forest so you know how about you
know the je s come for the past 2019
take on it to solve a filter problem use
a filter alright just two notes about
this I don't know if my my proverb
actually makes any sense maybe not and
secondly I thought that top when I was
writing this I thought the top row was
actually a super you know real ancient
wise proverb but it turns out it's just
from ninja boy in Pokemon emerald so
it's just something it's stuck with me
for 14 years but you know there you go
so anyway filters there's another filter
that we can actually use in Web Audio
and it's called the by quad filter node
now if you look if it seems like that
description looks pretty similar to the
one we just saw before you know you'd be
right those are the different the two
descriptions written the mdn Doc's give
you a second to spot the difference
there's not really much there so let's
do our own kind of comparison so are
they both do they both generate
second-order filters yep that's good
that's important because desktop
software's use second-order filters can
you create your own custom filter with
it well yes for the IR filter know for
the bicoid filter you kind of use what's
given to you okay that's that's alright
is it a Butterworth filter so remember
we generate our own Butterworth filter
with the create our filter method but
for the life of me I couldn't find out
what exact filter algorithm was used to
create the bicoid filter and that's
important because if something is
filtered slightly different it's gonna
look slightly different and it's looking
slightly different you might not see it
when you're flicking through four pages
a second finally kind of the big the big
one is the time taken to generate these
filters so clearly the bicoid filter is
a big winner here taking ten
milliseconds versus the hundred
milliseconds for the great IR filter
note so the question still remains is
does this filter filter the way we
wanted to do after a bit of research we
can say with with success yes
so the filtered data coming out of a
bipod filter is basically more or less
the same as the data coming through our
desktop software so my quad filter node
we just works
another big upside on this is that
remember before we had our Butterworth
filter code 200 lines of matlab ported
we transforms in complex numbers well
there's the coach a writer Butterworth
filter a bipod filter yeah that's nine
lines
let's change if you you know can
compress things a little bit so I call
this an absolute win so success things
are working so let me give you a demo of
these filters actually being applied so
that's not the way all right so right
now we have some of our EEG data and
this is you know usually what the
clinicians would do and would look at so
right now none of the filters are
applied and usually they'd be here
looking through this data and looks
alright you know without a filter but
sometimes you get really noisy parts
like this this might be where you're you
know muscle artifacts coming in or
signal noise so if you're trying to
review something in here there could be
all these spike waves underneath this
noise and you'd never see it so what you
can do though is you can just turn on
your filters yeah and right there that's
so much easy to see this is so much
closer to what the real EEG data looks
like you know you can also specify
exactly what filter you want you want to
make a you know I forty Hertz low-pass
you want to put a high-pass at say no
three cool you can do that on the fly
now right here I've got another bit of
EEG so this is without filters applied
let me just turn these filters on and
what you see underneath all of this
right here there's a spike and way so if
you're looking at this and it's a bit
noisy and you're flicking through page
after page you might not see this appear
right here but if you have these filters
applied you'll be able to see the
underlying epileptic signal there and be
able to give the correct diagnosis so
filters in the web
all right so we've come a long way
together we've learned a bit about
epilepsy about EEG about brainwaves and
how we can use the Web Audio API to the
help decipher these brain waves now
before I fly back home to the land of a
great beyond let me leave you with some
closing remarks so why does this matter
you know why am i giving a talk about
Web Audio and brain waves well look
there's many kind of other better talks
that could have been given about Web
Audio could have taught you how to make
your own motor track mixer and how do
you know
apply reverb and do these cool things
instead I gave you this one well the
reason is at my company seer you know we
help we try to help people you know
we've helped over a thousand patients
over our last two years of operation
we're seeing over 40 patients a week
coming through our services and so at
the end of the day what they want what
these patients want is a diagnosis and
as we saw if you don't see the right
data you're not going to get the right
diagnosis you don't get the right
diagnosis you can't get the right
treatment and you know in some small way
the Web Audio API is actually helping
with this so kind of technology you can
make a difference here you know it's at
the heart of what we're doing and it's a
heart of what we and I mean all of us
are kind of doing so I'm not sure that
the riders of the Web Audio API spit
kind of envisioned you know filters
being used for epilepsy diagnoses but it
is and it's getting done right here so
take nobody can make a difference but
it's only when you apply technology in
the right way that really can so at the
end of the day it's you know we're the
ones that making the difference we get
to choose how this technology is used
you know we get to choose where I spend
our time and look look we're not we're
not all you know volunteering in foreign
countries are working off for profits
you know but we can still use the web
technologies to make a difference in the
lives of others all right on a bit of a
lot of note I'd like to leave you with
one more final demo so perhaps there's a
question on everyone's mind previously
we did this we converted brain waves to
audio back to brain waves but what if we
did this brain waves to audio let's
leave it there all right well for the
first time on on any or at least this
particular J's comp stage I'd like to
present to you the the symphony of the
mind let's see how we go
you know one house there it is alright
I've got a super cool enable cool demo
button here cool demo is enabled
alright let's begin
put on the noise
[Music]
like an airplane noise now it's very
peaceful slow just got it down to
nothing
all right there you go
[Applause]