Computers that run the Von Neumann architecture store
their programs and data in the same memory bank.
Since both have to travel the same road to get and from the CPU, we find that the
system is ultimately limited not by the CPU or GPU's computational limits but by said road.
This is the famous Von Neumann bottleneck.
In a previous video I talked about in-memory computing as
a way to bring a computer’s memory closer to the compute.
But making a computer that thinks like the brain - a neuromorphic system as
it is called - entails far more than just memory.
For this video, a look at these brain-inspired systems and
the fundamental differences between computer and brain.
## Why the Brain First things, first. The brain.
Computer scientists have long had the desire to replicate the brain.
But why? What is so special about the brain? Aren't computers just better?
Computer scientists have long admired the brain's ability to function at very
low energy. The brain operates at about 12 to 20 watts of power - 20% of the body's metabolic rate.
The desktop computer on the other hand does about 175 watts. Leading edge AI
accelerators like the Nvidia H100 use anything from 300 to 700 watts.
The brain also does not operate at a very fast pace - with a clock frequency of about 10 hertz.
Though this varies depending on what the person is doing at the time and their mental state.
So the brain does not use a lot of power
and doesn't operate very quickly. And yet it is capable of so much.
Imagine a bee. A bee's brain has less than a million neurons and runs on
less than a watt of power. And yet it can fly. It can navigate to flowers
and back home. It can socialize and maybe even calculate things.
A bee's brain is just as capable as a 18 billion transistor system on chip,
and it can do all that with just a million neurons and virtually no power.
Perhaps we should start using brains and not silicon chips for learning? Oh wait ...
The biological brain's powerful capabilities - achieved at low power - is the single most
significant motivation for building neuromorphic hardware today.
## Insane Parallelism A brain accomplishes these with parallelism.
Your brain's 86 billion neurons operate without a central clock. By which I mean
that they fire their signals - referred to as "spikes" - on their own time,
based only on the spikes they receive from their neighbor neurons.
Measured in floating-point operations per second,
the brain is an exaflop-level compute device on par with the most advanced supercomputers.
An Nvidia H100 by comparison can only do something like 60 teraflops - depending on the variant.
The brain’s lack of synchronization is in contrast to digital circuits like a
CPU. A CPU relies on signals from a central clock as a reference by which to coordinate
their calculations. It lets them crunch certain tasks very quickly, but synchronicity has costs.
A central clock system costs time and energy to distribute the clock
signals across the system. And there is waste as each system component does not
execute its task until it is told to do so as per the central clock signal.
## Chaos as a Feature
No coordination implies that neural activity is chaotic, to say the least.
No doubt about that. But brains make that chaos a feature not a bug. The
neural environment is full of noise - spikes firing from neuron to neuron but going nowhere.
When a neuron receives a spike from a synapse, the majority of the time
it does nothing - noise. But there are so many extensive connections - with
so many synapses - that enough spikes get through to the right neurons to carry on.
Again, in huge contrast to the digital computer, which works hard to make sure that every signal
matters. A modern 3.2 gigahertz Intel CPU sends a "noise" signal once every 24 hours.
But as always, there are tradeoffs. You use a lot more power to achieve this very low
signal-to-noise ratio. Imagine the work you need to put in to synchronize billions of transistors.
That is how the brain achieves its efficiency. Neurons literally throw things at the wall and
are not afraid to make mistakes. In doing so, they find something that works. I hope that made sense.
And by living amidst chaos, brains also become shockingly resilient and
flexible - even in situations of massive damage.
It is not hard to see the value of this for computer hardware systems.
## Neuron
People who want to copy the brain often start with its fundamental unit - the neuron.
Neurons are cells in the brain and the rest of the nervous system. When they receive spikes from
their neighbors, they can - if they so choose - send their own spikes on to their neighbors.
All neurons share several common features - dendrites, soma,
and the axon. Spikes enter the neuron's cell body - formally called the soma - through dendrites.
Dendrites are the neuron's input pathways. A typical neuron in the outer layer of the
cerebrum - the largest part of the brain - has 10,000 inputs.
A typical neuron in the cerebellum - the second largest part of the brain,
but trying hard to get to number one - has up to a quarter of a million inputs.
If sufficiently stimulated, the neuron sends a spike to its neighbors. Or a
series of spikes - a neural code, though how the code works remains somewhat of a mystery.
This spike is sent through an out-path called an axon. The axon's primary goal is to ensure
the signal is forwarded faithfully - though it does not always do this.
And then from there, it enters the neighbor neurons' dendrites through what
we call "synapses" - an electrochemical structure for connecting two neurons.
Such a description understates their role in computation. Variations in
synaptic structure over long time periods - a thing referred to as "synaptic plasticity"
can subtly change the neuron spikes and how they are received. We should not ignore them.
## Diversity
So those are the common structures, but the brain's 86 billion neurons exhibit
incredible diversity. As you might expect with 86 billion of anything.
Some neurons - due to the complicated chemistries of their axons - can send
signals faster than the Taiwan High Speed Rail. Other neurons, slower than a Taiwan sea turtle.
Some motor neurons can stretch up to a meter long. The ones in a giraffe can be multiple
meters long! Other neurons on the other hand might just be 0.1 millimeters long.
And neurons - just like us - have their own preferred stimuli. In 1981,
David Huber and Torsten Wiesel won half of the Nobel Prize for Medicine for showing that
some neurons fired most rapidly when shown lines going in one direction over others.
As I mentioned above, the majority of the time a neuron receives a spike
from a neighbor neuron, things get left on read.
So if we are to sum it up enough to get us to the next section of the video,
the brain works by propagating huge, random waves of spikes throughout its billions of
diverse neurons. Many of these spikes end up as wasted noise, but many are relevant too.
Neurons and Synapses merge together the work of computation and memory - doing a
bit of both. Memories are stored in the relative strength of the
synapses between neurons. But those synapses can do calculations as well.
That is why the brain does not suffer the
Von Neumann bottleneck. That's their secret, Cap. They aren't separate.
## Going Neuromorphic
So in order to create a proper neuromorphic computer,
we not only have to implement artificial versions of neurons and synapses - But also the way
they very tightly bind together the memory, compute, and communications between the two.
Now, you might be thinking, "What about artificial neural networks
like those running in ChatGPT? Can we call those neuromorphic?"
It is true that these neural networks started from
our understanding of how the human brain works. So many of the concepts overlap.
Perceptrons for instance are a simplified mathematical model of the meat neuron. It
approximates the neuron's behavior by taking in a weighted sum of inputs, applying an "activation
function" to mimic the neuron's stimulation, and fires off an output to its neighbors.
But virtually all of these artificial neural networks - especially the ones running out there
in the real world - run on Von Neumann hardware - which means dealing with the bottleneck.
Changes can be made to the hardware in order to improve performance and power consumption,
of course. That is why we GPU and TPU. But the tantalizing possibility remains of
getting game-changing benefits by running this neural "software" on actual neural "hardware".
The neural software - often referred to as "spiking neural networks" to
differentiate from Von Neumann-style ANNs and Deep
Neural Networks - shall be discussed some other day. Let's talk hardware.
## Silicon Neurons Many industrial and academic players have
demoed neuromorphic hardware created with traditional CMOS transistors.
So normal semiconductor manufacturing processes like that for Von Neumann
computers. Let me note a few of these "silicon neurons". Many are - like today's
artificial neural networks - programmatic approximations of the neuron's behavior.
We have IBM's TrueNorth project from 2014 - the first widely distributed
neuromorphic chip. It is capable of running inference - recognizing that
a person is doing something in a video or controlling a robot.
TrueNorth is a special CMOS integrated circuit with 4,096 cores - each with
256 programmable neurons. The whole chip has 256 million programmable synapses.
A big point that IBM made was how it uses far less power than most
computer systems - the chip's 5.4 billion transistors consume about 70 milliwatts.
Other semiconductor companies have shipped silicon neurons of their
own too. Intel Labs has their Loihi and Loihi 2 neuromorphic AI hardware.
The European Human Brain Project was a massive ten year science project initiated in 2013,
looking to do groundbreaking work in the neural sciences.
One of their projects was the BrainScaleS project, which was first released in 2011.
A second version was released in 2020 with improved local learning capabilities.
It is a mixed-signal ASIC chip that uses analog electronic circuits to
mimic the spiking neurons of the brain. Very interesting
though its analog neurons are not as flexible as its digital counterparts.
And then in 2019 we have the Tianjic project, a hybrid chip created by scientists at China's
elite Tsinghua University. It is a hybrid platform that attempts to run various types
of neural networks - including those designed for neuromorphic hardware.
## Crossbar
How do we implement a neuron network in hardware? Let us
briefly look at how IBM's TrueNorth does it.
TrueNorth is fabbed on a 28 nanometer CMOS process. As I mentioned, it has 4,096 of what they
call neurosynaptic cores. It is capable of running widely adopted convolutional neural networks.
Each basic TrueNorth core has 256 axons, 256 neurons,
and synapses fully connecting them in a 256x256 crossbar structure, implemented using SRAM.
Each of the axons are given a synaptic weight depending on the neuron that
they are connected to. And each neuron has a state or "membrane
potential" as well as a particular threshold for sending on a spike.
The system works on cycles. During the cycle,
spikes travel to the neurons through axons, which affects the spike's value.
The neuron collects the incoming spikes into buffers and then evaluates them.
They then update their own membrane potentials accordingly. Once having done that, they compare
it against their threshold. If the membrane potential meets or beats the threshold, the neuron
sends a spike of its own. Spikes can be sent to local neighbors or outside the core itself.
Just like the real brain, TrueNorth does not have
a global clock. The elements in each of the cores work asynchronously in cycles,
doing things only in response to events. They also operate at low frequency - 10 hertz.
And to implement the neuron's inherent randomness and noise,
each core has a random number generator that can raise the thresholds for creating
a spike. Or randomly delay or even halt a spike's transmission through a synapse.
It's like that episode of It's Always Sunny in Philadelphia. Wildcard!
## CMOS Shortcomings
A significant benefit of building these "silicon neuron" chips using
CMOS processes is that we get to draft in the wake of the chip giants.
They share some of the benefits of the brain. TrueNorth for instance looks to be very scalable,
sips relatively lower power compared to GPUs,
and can run widely used neural network software to get accurate results.
But there are drawbacks. A test of TrueNorth's performance found tradeoffs between energy
efficiency and accuracy. If we want the model to give us competitively accurate
results - which can make a big difference in a commercial context - we need to use more power.
These circuits are also quite large, since each of the cores have to have their own memories.
Von Neumann machines benefit from having a single, very dense central memory bank. Having denser,
higher capacity allows us to accommodate the larger models that are so in vogue.
And lastly, the brain is an analog device - digital devices are an ill
fit for replicating their behavior. So we need to incorporate the analog element,
which limits the system's flexibility.
These disadvantages have pushed scientists to look beyond CMOS for new ways to
implement neuromorphic devices. The most popular such approach is the memristor.
## Memristors
In 1971, Leon Chua - then a professor of Electrical Engineering and Computer
Sciences at UC Berkeley - published an article proposing a new type of circuit.
By looking at the relations between the three major circuit elements,
he proposed the existence of a fourth - the "memory resistor" or memristor.
This paper was very difficult to read and I will freely admit that I did not get it.
So I am not going to explain any more than that. Anyway, The idea fell on the wayside until 2008,
when scientists at Hewlett-Packard announced a physical implementation of the memristor.
The original 2008 memristor was a simple metal-dielectric-metal sandwich with two
terminals, or points of connection. In this case,
the metal electrodes were made from platinum and the dielectric was titanium oxide.
If we are to apply a voltage pulse to the memristor, the film can switch its electrical
resistance - flipping between an insulating and conducting state in a non-linear fashion.
What is it about the memristor that makes it so suitable for neuromorphic computing?
Their key characteristic is that the value of that electrical resistance
is dependent upon the history of the voltage passing through it - ergo the name "memory".
Even better, it can remember that history even when the power goes off. This makes
the memristor a form of "non-volatile memory" like flash memory or a hard drive.
People quickly drew connections between memristor behavior and that of biological
synapses. Scientists envisioned them as a non-volatile memory for in-memory computing.
For the past 15 years, such devices have been at the center of neuromorphic study.
But there are a few challenges. The first has to do with manufacturability. It can
be very difficult to produce enough of these memristors uniformly and at
scale. Can they handle many cycles of resistance switches? And how long can
they store their data? Engineers are still working through these questions.
Also, the changes in the memristor's resistance are non-linear, which makes
it somewhat challenging to program for. New software paradigms in neural network
programming may be needed for memristor-based neuromorphic computing to work. The work goes on.
## Conclusion
I think the biggest challenge with these neuromorphic systems though is the competition.
Nvidia and other adherents of "Huang's Law" are leading the way. According to Huang's Law,
silicon chips powering AI more than double in power every two years.
In the past 10 years, AI inference performance in the GPU has improved
1,000 times. It might be hard to compete against this. But maybe we don’t have to.
Considering each system's advantages, we might start seeing more hybrid systems.
Perhaps we can put neuromorphic and Von Neumann chiplets together so to give us
the best of both worlds. Like a mullet. Business in the front. Party in the back.