Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today, we are going to read some minds.
A few months ago, our first video appeared on brain machine interfaces.
It was about a paper from Neuralink, which promised a great deal.
For instance, they proposed a brain machine interface that could read this pig’s thoughts
as it was running on the treadmill.
And what’s more, they not only read is thoughts, but they could also predict what the pig’s
brain is about to do.
So this was about reading thoughts related to movement.
And to be able to use these brain machine interfaces to the fullest, they should be
able to enable some sort of communication for people who have lost their ability to
move or speak.
So about writing or speaking?
As impossible as it sounds, can we somehow restore that, or is that still science fiction?
Many of you told me that you would like to hear more about this topic, so due to popular
request, here it is, a look beyond Neuralink’s project to see what else is out there.
This is a collaboration between Stanford University and a selection of other institutions, and
it allows brain-to-text transcription, where all the test subject has to do is imagine
writing the letters, and they magically appear on the screen.
And now, start holding on to your papers, and just look at how quickly it goes!
90 characters per minute with over 94% of accuracy, which can be improved to over 99%
of accuracy with an additional autocorrect technique.
That is absolutely amazing.
A true miracle.
90 characters per minute means that the test subject here, who has a paralyzed hand, can
almost think about writing these letters continuously, and most of them are decoded and put on the
screen in less than a second.
Also, wait a second, 90 characters per minute?
That is about 80% as fast as the average typing speed on a smartphone screen for an able-bodied
person of this age group.
Whoa.
It is quite remarkable that even years after paralysis, the motor cortex is still strong
enough to be read by a brain-computer interface well enough for such typing speed and accuracy.
It truly feels like we are living in a science fiction world.
Of course, not even this technique is perfect, it has its own limitations.
For instance, we can’t edit or delete text, have no access to capital letters and the
method has a calibration step that takes a long time, although it doesn’t get significantly
worse if we shorten this calibration time a bit.
So how does this work?
First, the participant starts thinking of writing one letter at a time.
Here you see the recorded neural activity, this is subject to decoding.
You can see the decoded signals here.
And we can’t just give this to a computer to distinguish between them as is, we project
these into a 2D latent space where it is easy to find which letter corresponds to which
region.
Look, they form relatively tight clusters, therefore it is now easy to decide which of
the squiggles corresponds to which letter.
The decoding part is done by using a recurrent neural network, which is endowed with memory
and can deal with sequences of data.
So here, in goes the brain activity, and out comes the decision that says which character
these activities correspond to.
Of course, our alphabet was not designed to be decoded with neural networks.
So here is an almost science fiction-like question - how do we reformulate the alphabet
to tailor it to maximize the efficiency of a neural network decoding our thoughts?
Or simpler, what would the alphabet look like if neural networks were in charge?
Well, this paper has an answer to that too, so let’s have a look.
The squiggles indeed look like they came from another planet, so what do we gain from this?
Well, look at the distance matrix for the English alphabet.
The diagonal is supposed to be very blue, but what is not supposed to be blue at all
are the regions that surround it.
Look, the blue color here means that in the English alphabet, the letters M and N can
be relatively easily confused.
Same with the letters O and C, and there are many more similarities.
And now, look, here is the same distance matrix for the optimized alphabet.
No dark blue in sight outside the diagonal!
Much easier to process and decode.
If neural networks were in charge, this is what the alphabet would look like.
Glorious!
Also, the fact that we are talking about squiggles is not a trivial insight at all, traditional
methods typically rely on movement in straight lines to select letters and buttons.
The other key thought in this paper is that modern neural network-based methods can decode
these thoughts of squiggles reliably.
That is absolutely amazing.
And wait a second…note that there is only one participant in the user study.
Why just one participant, why not call in a bunch of people to test this?
It is because this method uses a microelectrode array, and this requires surgery to insert,
and because of that, these studies are difficult to perform, and are usually done at times
when the participant has brain surgery anyways for other reasons.
Having more people in the study is usually prohibitively expensive, if at all possible
for this kind brain implant.
And note that research is a process, and these papers are stepping stones.
And now, we are able to help people write 90 characters every minute with a brain machine
interface, and I can only imagine how good these techniques will become two more papers
down the line.
And don’t forget, there are research works on non-invasive devices too!
So, what do you think?
Let me know in the comments below.
What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!