I became obsessed with the relationship between the brain and the mind
after suffering a series of concussions
playing football and rugby in college.
I felt my mind change for years after.
I was studying computers at the time,
and it felt as though I had damaged my hardware
and that my software was running differently.
Over the following years,
a close friend suffered a serious neck injury
and multiple friends and family members were struggling
with crippling mental health issues.
All around me, people that I loved dearly were being afflicted by ailments
of the nervous system or the mind.
I was grappling with all of this
while pursuing an MFA in Design and Technology at Parsons
when a friend and fellow student showed me an open-source tutorial
on how to build a low-cost single-channel EEG system
to detect brain activity.
After a couple long nights of hacking and tinkering,
I saw my brainwaves dancing across the screen for the very first time.
And that moment changed my life.
In that moment,
I felt as though I had the possibility to help myself and the people I loved.
And I also realized that I couldn't do it alone.
I needed help.
So in 2013, in Brooklyn, with some like-minded friends,
I started OpenBCI, an open-source neurotechnology company.
In the beginning,
our goal was to build an inward-pointing telescope
and to share the blueprints with the world
so that anybody with a computer could begin peering into their own brain.
At first, we were an EEG-only company.
We sold brain sensors to measure brain activity.
I thought that's what people wanted.
But over time, we discovered people doing very strange things with our technology.
Some people were connecting the equipment to the stomach
to measure the neurons in the gut
and study gut-brain connection and the microbiome.
Others were using the tools to build new muscle sensors and controllers
for prosthetics and robotics.
And some were designing new devices
and peripheral add-ons that could be connected to the platform
to measure new types of data that I had never heard of before.
What we learned from all of this
is that the brain by itself is actually quite boring.
Turns out brain data alone lacks context.
And what we ultimately care about is not the brain,
but the mind, consciousness, human cognition.
When we have things like EMG sensors to measure muscle activity
or ECG sensors to measure heart activity,
eye trackers
and even environmental sensors to measure the world around us,
all of this makes the brain data much more useful.
But the organs around our body, our sensory receptors,
are actually much easier to collect data from than the brain,
and also arguably much more important for determining the things
that we actually care about: emotions, intentions and the mind overall.
Additionally,
we realized that people weren't just interested
in reading from the brain and the body.
They were also interested in modulating the mind
through various types of sensory stimulation.
Things like light, sound,
haptics and electricity.
It's one thing to record the mind,
it's another to modulate it.
The idea of a combined system
that can both read from and write to the brain or body
is referred to as a closed-loop system or bidirectional human interface.
This concept is truly profound,
and it will define the next major revolution in computing technology.
When you have products that not just are designed for the average user
but are designed to actually adapt to their user,
that's something truly special.
When we know what the data of an emotion or a feeling looks like
and we know how to make that data go up or down,
then using AI,
we can build constructive or destructive interference patterns
to either amplify or suppress those emotions or feelings.
In the very near future,
we will have computers that we are resonantly
and subconsciously connected to,
enabling empathetic computing for the very first time.
In 2018,
we put these learnings to work
and began development of a new tool for cognitive exploration.
Named after my friend Gael, who passed from ALS in 2016,
we call it Galea.
It’s a multimodal bio-sensing headset,
and it is absolutely packed with sensors.
It can measure the user’s heart, skin, muscles, eyes and brain,
and it combines that capability with head-mounted displays
or augmented and virtual reality headsets.
Additionally, we're exploring the integration
of non-invasive electrical neural stimulation as a feature.
The Galea software suite
can turn the raw sensor data into meaningful metrics.
With some of the sensors,
we're able to provide new forms of real-time interactivity and control.
And with all of the sensors,
we're able to make quantifiable inferences about high-level states of mind,
things like stress, fatigue, cognitive workload and focus.
In 2019,
a legendary neurohacker by the name of Christian Bayerlein
reached out to me.
He was actually one of our very first Kickstarter backers
when we got started, early on.
Christian was a very smart, intelligent, happy-go-lucky
and easygoing guy.
And so I worked up the courage to ask him, "Hey, Christian,
can we connect you to our sensors?"
At which point he said,
"I thought you would never ask."
(Laughter)
So after 20 minutes,
we had him rigged up to a bunch of electrodes,
and we provided him with four new inputs to a computer.
Little digital buttons, that he could control voluntarily.
This essentially doubled his number of inputs to a computer.
Years later, after many setbacks due to COVID,
we flew to Germany to work with Christian in person
to implement the first prototype
of what we're going to be demoing here today.
Christian then spent months training with that prototype
and sending his data across the Atlantic to us in Brooklyn from Germany
and flying a virtual drone in our offices.
The first thing that we did was scour Christian's body
for residual motor function.
We then connected electrodes to the four muscles
that he had the most voluntary control over,
and then we turned those muscles into digital buttons.
We then applied some smart filtering and signal processing
to adapt those buttons into something more like a slider
or a digital potentiometer.
After that,
we turned those four sliders
and mapped them to a new virtual joystick.
Christian then combined that new joystick
with the joystick that he uses with his lip to control his wheelchair,
and with the two joysticks combined,
Christian finally had control over all the manual controls of a drone.
I’m going to stop talking about it, and we’re going to show you.
Christian,
welcome.
(Applause)
At this point,
I'm going to ask everybody to turn off your Bluetooth
and put your phones in airplane mode
so that you don't get hit in the face with a drone.
(Laughter)
How are you feeling, Christian?
Christian Bayerlein: Yeah, let's do it.
Conor Russomanno: Awesome.
This is a heads-up display that's showing all of Christian's biometric data,
as well as some information about the drone.
On the left here, we can see Christian's muscle data.
Christian is now going to attempt to fly the drone.
How are you feeling, Christian, feeling good?
CB: Yes.
CR: All right. Rock and roll.
Let's take this up for a joyride.
Whenever you're ready.
CB: I'm ready.
(Applause and cheers)
CR: All right, take her up.
And now let's do something we probably shouldn't do
and fly it over the audience.
(Laughter)
(Cheers and applause)
Alright, actually, let’s do this.
I'm going to ask for people to call out some commands in the audience.
So how about you?
Straight forward.
(Laughter)
Alright. How about you?
Man: Up!
(Laughter)
CR: Not down.
Oh, he's doing what he wants right now.
Amazing.
(Cheers and applause)
Alright, let’s bring it back.
And what I'm going to do right now is take control of the controller
so that you guys know
that there isn't someone backstage flying this drone.
All right, Christian, you're alright with that?
CB: Yeah.
CR: Unplug.
Forward.
And we're going to land this guy now.
CB: I think I was better than you.
(Laughter)
(Applause)
CR: Amazing.
(Applause)
Now I'm going to unplug it so it doesn't turn on on its own.
Perfect.
Christian has repurposed dormant muscles from around his body
for extended and augmented interactivity.
We have turned those muscles into a generic controller
that in this case we've mapped into a drone,
but what's really cool is that joystick can be applied to anything.
Another thing that's really cool
is that even in individuals who are not living with motor disabilities,
there exist dozens of dormant muscles around the body
that we can tap into for augmented and expanded control interactivity.
And lastly,
all the code related to that virtual joystick,
we're going to open source
so that you can implement it and improve upon it.
There's three things that have stood out to me
from working on this project and many others over the years.
One,
we cannot conflate the brain with the mind.
In order to understand emotions
and tensions and the mind overall,
we have to measure data from all over the body,
not just the brain.
Two,
open-source technology access and literacy
is one way that we can combat the potential ethical challenges we face
in introducing neural technology to society.
But that's not enough.
We have to do much, much more than that.
It's very important, imperative, that we set up guardrails
and design the future that we want to live in.
Three.
It's the courage and resilience of trailblazers like Christian
who don't get bogged down by what they can't do,
but instead strive to prove that the impossible is in fact possible.
(Applause)
And since none of this would have been possible without you, Christian,
the stage is yours.
CB: Yeah, hi, everybody.
Audience: Hi.
CB: I'm excited to be here today.
I was born with a genetic condition that affects my mobility
and requires me to have assistance.
Despite my disability,
I'm a very happy and fulfilled person.
What truly holds me back are not my physical limitations.
It's rather the barriers in the environment.
I'm a tech nerd and political activist.
I believe that technology can empower disabled people.
It can help create a better,
more inclusive and accessible world
for everyone.
This demonstration is a perfect example.
We saw what's possible when cutting edge technology
is combined with human curiosity and creativity.
So let's build tools that empower people,
applications that break down barriers
and systems that unlock a world of possibilities.
I think that's an idea worth spreading.
Thank you.
(Cheers and applause)