Nvidia's GPU evolution kicked off the neural network revolution. However,
while GPUs run neural network algorithms quite well, they are not specifically designed for it.
So companies have started to develop hardware
customized for running specific AI algorithms - dubbed AI accelerators.
Today, the AI accelerator hardware market is estimated to be worth over $35 billion.
Venture capitalists poured out nearly $2 billion for AI chip startups in 2021.
TSMC considers AI accelerator hardware as one of their top secular drivers in revenue for the
near future. In this video, we are going to look at what these weird things actually are about.
## Beginnings
In 2011, a team from New York University and Yale
University presented a paper at a 2011 conference called
CVPR. The paper discussed a scalable hardware architecture for big and deep neural networks.
By now, the AI research community had broadly accepted GPUs as the de facto
standard hardware to run and test their algorithms. But the paper points out that
custom hardware can improve performance and power consumption by two orders of magnitude.
Traditionally, deep learning models are shown processing
images with a sequential pipeline of steps within a neural network.
It reminds me of how your phone processor turns the raw image data it receives from its sensor
into the finished photo that shows up on your screen. Watch that video if you haven't already.
The 2011 paper proposed an architecture that processed each individual step in parallel,
and applied it to a computer vision algorithm. They implemented the
datagrid architecture on a Xilinx Virtex 6 FPGA - a type of chip that
allows for custom configurations - and saw substantial improvements.
The interesting thing to note is that FPGAs are development and testing devices,
so they tend to be slower. So the team also presented a hypothetical 45-nanometer chip design
fabbed with an IBM process. And it blew its competitors out of the water.
## Functions and Inferences
We should discuss neural networks and what they are before we go too far.
I talked about this in my video about Nvidia's GPU
evolution - which I recommend you watch. But let us refresh ourselves on neural networks.
Strip away the exoticness of the name and what we find is
that a neural network is basically a titanic mathematical function.
It uses simple processing elements to model complex relationships between many inputs.
A neural network is represented with matrices.
You would first turn the input image into a representative matrix of data.
Then, you "send" that input data through the neural network's many layers.
This is done by multiplying the input matrix with weight matrices. Then you
apply a special math function - referred to as an "activation function" - to the resulting values.
Yeah, This is a massive simplification of what companies are doing in production but there we go.
There are two roles that AI accelerators have here. First is in the aforementioned
training phase, where we use large data sets to
determine the weight values that go into the network's many layers.
The second is the inference stage. That is when the trained neural network function
is being used to infer a result from the input data. In other words, when you turn the image into
a data matrix, and start multiplying weight matrices and applying activation functions.
This is the moment when you show the neural network a photo of a
dog so that it can tell you that you have a dog in the photo. Hopefully, at least.
Neural network accelerators work better than their more generalized cohorts
like CPUs and GPUs because they are tuned to handle matrix multiplication
and convolution operations - which take up some 90% of the work.
And there are a lot of calculations to do. Google back in 2016 - over five years
ago - surveyed how many. Some networks had up to 100 million weights. I think it makes sense
then that you would build your own hardware to handle this. But we will get to that later.
## Catching On
The concept caught on, and soon a small community of AI hardware projects sprouted up.
I'll mention just a few of these forerunners here. They include the 2015 project
ShiDianNao, run by the Chinese Academy of Sciences' Institute of Computing Technology.
ShiDianNao is a piece of hardware tuned to run a specific type of
neural network - one suitable for computer vision applications.
Their 65 nanometer accelerator consumed 4,700 times less energy than a GPU and
was small enough to be embedded alongside any commercial image sensor in a product.
CAS would expand their hardware into four lines for basic,
multi-configuration, training/learning, and computer vision purposes.
Another notable project is the Eyeriss flexible accelerator for
mobile devices running neural networks - an MIT project with some funding from DARPA.
Eyeriss‘ chip was for running inferences on images using a pre-trained model.
Furthermore it did it with 10 times more energy efficiency than contemporary GPUs.
Finally, one notable startup was TeraDeep,
a project founded by members of the original team that presented at CVPR 2011. Their processor - the
nn-X - wanted to revive old webcams and other hardware by giving them deep learning powers.
## Edge and Server
The three projects that we just discussed have one big thing in common - their
chip products have generally focused on inferring results from pre-trained models.
Hardware specialized for inference. But they differed in where they performed
those activities. And this location is important in classifying the chip.
So in addition to this training-versus-inference split, the AI hardware industry has another
classification that depends on the environment in which they operate: Edge or Server.
"Edge or Server" kind of sounds like a fancy hipster cafe somewhere in Brooklyn,
where the lattes cost $8.
Edge AI chips are placed directly into the device - smartphones, cars, IoT devices, or wearables. As
a result, designers have to take into account the power and size restrictions of this form factor.
For server chips, companies are most concerned about the cost-performance ratio.
Power is a substantial portion of the chip's total cost of ownership,
so data centers want the best performance they can get at a certain cost.
As a result, these chips are higher-end products near the leading-edge. Developing
such designs cost a lot of money and resources, which raises the upfront investment needed.
In the end, all AI accelerators are classified using these two axes that we have talked about.
They are either doing training or inference,
and they are doing it either on the edge or in the server/data center.
Up until then, AI accelerators were mostly still research projects
percolating in small startups or universities.
The market would not see wide prominence until Google made a big announcement in 2016.
## Google and the TPU
Google had thought about building their own chips since 2006.
But the effort really kicked off in 2013 when they realized the immense computational demand
training and deploying neural networks would have on their servers.
So they started working on the Tensor Processing Unit, or TPU. Quickly designed and fabbed with
a 28 nanometer process, the first TPUs entered Google's data centers in 2015.
They publicly unveiled its existence a year later in May 2016.
Chips like the TPU are referred to as ASICs,
application-specific integrated circuits. And that application is matrix multiplication. At
its very heart is something called a "Matrix Multiply Unit". It contains
65,536 multiply-accumulator circuits or MAC units, arranged in a 256 by 256 array.
The multiply-accumulator circuit is designed to do just one thing. It
multiplies two numbers and then adds it to an accumulation sum. That's really about it.
When the TPU is in use, the Matrix Multiply Unit receives and processes two things.
The first is the subject image data from the TPU's host CPU
or main memory. This is the thing you want to run inference on.
The second thing is the neutral network’s secret sauce - the millions of weights - which
are loaded from the attached DDR3 DRAM memory into the Matrix Multiply Unit.
Other server chips of the time include the 18-core Intel Haswell CPU and the Nvidia Kepler K80 GPU.
The TPU has far more Multiplier Accumulator units than either
of those chips and more on-chip memory for storing intermediate results.
Of course, CPUs and GPUs are designed to handle far more generalized tasks than the TPU.
I don't think this TPU can run video games or Photoshop for instance.
The TPU design's overarching goal is to ensure that the Matrix Multiply Unit is in constant use.
More recent iterations of the TPU are capable of trillions of floating operations each second.
## TPU Benefits
Stripping it down to the bare bones, the TPU is not terribly
complicated - a massive linear algebra machine - but it is very specialized.
Because of this specialization, Google's first generation TPU can run inference
15-30 times faster and 30-80 times better energy efficiency than its contemporaries.
Google filling their data centers with better performing TPU hardware keeps them from having
to buy CPUs and GPUs. And that saves them millions of dollars for their data centers. Good for them.
It is important to note that Google also has their own machine learning software framework -
TensorFlow - which lets them exactly customize the hardware to what the software is doing.
With these two, Google built out an end-to-end machine learning system that lets it do
amazing things. Like object searches on 8-10 years of uploaded Google Photos.
Google didn’t make their TPUs available for sale - rightfully considering it a competitive advantage
of theirs. But people can rent its computing power through their Google Cloud service.
## Server Competition
Google essentially created the server AI ASIC industry.
But other companies have rushed to do the same.
Amazon is the biggest cloud hyperscaler out there with AWS. They have already
proven their chip design chops with their internally designed Graviton chips - which
are ARM-based CPUs that have started to become a real alternative to Intel's server CPUs.
In December 2020, they announced their own Trainium devices, AI training accelerators.
These options are a challenge to GPUs and other hardware makers
with AWS-hosted instances like Habana.
Habana was founded in 2016 with a data center product
for training machine learning models. They were acquired by Intel in December 2019
for $2 billion. Considering the Amazon announcement, I think they timed it right.
Nvidia has their own custom AI hardware too: The Tesla V100. This is not necessarily a GPU,
but an AI processor with over 20 billion transistors and 5,120 cores. Their
software competitive advantage is quite strong, so they are a formidable player.
For startups targeting the AI server market, this intense competition between billion
dollar startups and multi-billion dollar tech giants has made navigating these seas
rather challenging. When the biggest buyers start making their own AI chips and positioning
its instances more favorably, there is some reason for concern.
## Neural Processing Units
So the data center market is becoming increasingly competitive for outsiders.
Yet things on the other end aren't getting any better either.
Pitchbook noted that half of the AI chip market is in specialized AI chips in mobile
phones. My first and uneducated guess would be that these chips handle the
increasingly computation-heavy operations of image processing.
But the mobile CPU processors are trying to eat up that market opportunity themselves
by growing and expanding their chips' onboard neural networking hardware.
Apple again led the way. In 2017, the iPhone X's A11 Bionic chip
had dedicated neural network hardware built into it. The Neural Engine. The
first iteration had 2 cores, and was capable of 600 billion operations each second.
Again, Apple's control of the entire software stack makes this sort of integration ideal.
The first Neural Engine was only used for Face ID and Animojis, but the next year
Apple opened up the second version of the Neural Engine to users of its CoreML API.
Other chip vendors are adding dedicated AI functionality to their chips too.
For instance, Taiwanese fabless chip maker MediaTek and their Dimensity 5G mobile processors.
There are still many opportunities left for inference on the edge - IOT devices,
autonomous driving, and what not. But few mobile devices have margin like phones and
the technical effort behind those low-power and low-latency use cases will be a bit trickier.
But as my dad used to say, challenges are just opportunities in disguise.
## Silicon Photonics and AI
Even with these specialized architectures, today's AI accelerators can take hours or even longer
to train a single machine learning model for production.
One suggestion for what’s next has to been to implement new AI chips using silicon photonics.
Servers in today's data centers operate much like any other computer. There is a CPU that
processes information and a separate memory bank that stores the data and instructions.
This setup is called a von Neumann architecture and it has been around since the 1940s.
Neural networks challenge this architecture. Beyond the fact that they are highly parallel,
the separation between the processor and the memory slows things down a great deal. It is
like if the brain stored its synapses and neurons in two separate locations.
Silicon photonics products use light rather than electricity to send signals. Light can
travel through optical fiber at the fastest speed possible, and without additional heating concerns.
I discussed silicon photonics in a previous video - but only briefly discussed the technology’s data
processing potential. I plan to have a more detailed video on it later down the line.
## Conclusion
It is interesting to compare the AI accelerator hardware industry with
the Bitcoin mining hardware industry. Those guys have taken a similar path,
first with CPUs, then GPUs, and then increasingly more powerful ASICs.
For the bitcoin miners, the next step seemingly has always been to go to a more advanced node.
But I don't know if future AI accelerators have to go down that same path.
If so then the big chip companies like Nvidia will always have greater access and resources for that.
The more interesting AI accelerator approach will be those taken by companies finding new ways
to achieve similar results on inference or training without an advanced node.
Whether that be silicon photonics or something else like advanced parallelism, we shall see.