When we last left Nvidia, the company had emerged victorious in the brutal graphics
card Battle Royale throughout the 1990s.
Very impressive. But as the company entered the 2000s, they embarked on a journey to do
more. Moving towards an entirely new kind of microprocessor - and the multi-billion
dollar market it would unlock.
In this video, we are going to look at how Nvidia turned the humble graphics card into
a platform that dominates one of tech’s most important fields: Artificial Intelligence.
In 1999, Nvidia released the GeForce 256 graphics card. Its processor had 17 million transistors
and was fabbed by TSMC.
Most notably, when Nvidia unveiled the GeForce, they called it "the world's first Graphics
Processing Unit", or GPU.
A new term that must have struck people as a bit of clever marketing. At its introduction,
Nvidia defined the term GPU to mean:
> "a single-chip processor with integrated transform, lighting, triangle setup/clipping,
and rendering engines that is capable of processing a minimum of 10 million polygons per second."
To understand what was so special or groundbreaking about this particular Nvidia product, we need
to dive into the graphics pipeline. Yes, we need to talk about graphics.
The job of a 3D graphics system is to turn a scene into images. 60 times a second, sometimes
more, with minimal lag. Here is a very simplified example of how they did it. Like, super simple.
Early real-time graphics systems broke everything down into triangles. More complicated squares
and quadrilaterals? Triangles. Curved patches? Triangles. It's triangles all the way down,
baby!
At the start of the pipeline, you have a bunch of 3D coordinates data and 3D model or shape
data.
The pipeline first ingests those 3D coordinates and turns them into 2D window coordinates.
This is called the "geometry stage".
The geometry stage can be further subdivided into two sub-stages: The first is "Transform
and Lighting", and the second is "Triangle Setup and Clipping".
In Transform and Lighting, the pipeline needs to adjust the 3D coordinates data and apply
lighting effects to take into account the user's own personal perspective.
The Triangle Setup and Clipping stage is where we process the coordinate data into something
that the rendering engine can render.
This especially matters in scenes where multiple 3D objects are overlapping one another. The
overlapped object would need to be clipped, ergo the second part of the name.
I want to pause here for a quick rest stop. When the graphics pipeline is processing and
manipulating these coordinates, they use a mathematic structure called a "matrix".
A matrix is a rectangular array of numbers. You can perform operations on these matrices
- like multiplication or addition - in a standard defined way.
We don't need to know exactly how matrices figure into manipulating 3D coordinates within
the overall graphics process. Fear not, this is not a math class.
But we do need to know that they exist. And that the graphics pipeline - and thus graphics
cards running them - were heavily optimized to process and run mathematical operations
on them. It will come up again later.
Okay. Next, is the "rendering stage". Here, the pipeline fills the pixel space between
the translated 2D coordinates with image pixels representing the object's surface.
The setup engine passes the render engine all the information about the object's color
and texture at a particular spot as well as how those are affected by lighting conditions
and perspective.
The final result is then sent to the monitor for displaying to the user.
As recently as 1996, Nvidia's graphics cards handled just a subset of the graphics pipeline
- the render stage.
The CPU handled the rest.
This made the most sense for the company's early hardware since rendering was simple
and repetitive. You receive the coordinates of the triangles' lines. Then you just draw
the right lines with the right colors, many times over.
But very quickly, Nvidia's cards started to take on more of the graphics pipeline on its
own. In 1997, their cards took on Triangle Setup and Clipping.
If you recall, this stage was about computing the actual triangle coordinates for the render
stage to process.
The CPU previously handled this, but the work was repetitive and became a bottleneck in
the pipeline.
Now freed of the lame work of setting up triangle coordinate data, the CPU could focus on the
geometry stage.
And it did this part quite well, throwing more triangles at the graphics card than it
could handle. At that point, the card became the bottleneck in the pipeline.
The solution Nvidia's engineers found for this bottleneck was parallelism. They converted
the work into a pipeline of sequential steps. Then they added many identical pipelines and
lined them up to work in parallel.
For some reason, I am reminded of bees in honeycombs, making honey. Each individual
bee in its little comb works independently of its fellow parallel bees. No need to wait
for this other bee to finish up his bee work first.
And no, I have no idea how honey is actually made.
3D graphics work was ideally suited for parallelism because the task is repetitive and data-independent.
The system's processing speed is almost linearly proportional to the number of available honeycombs,
or in the GPU’s case, processing cores.
This finally opened the door for Nvidia to take on the last part of the graphics pipeline:
Transform and Lighting calculations.
With the GeForce 256, Nvidia succeeded in taking the entire graphics pipeline onto a
single chip. No more need to transfer data back and forth with the CPU. This major structural
change is why the company deemed it worthy of a new name.
The GeForce 256, the first GPU, was a major achievement and the market started to really
take off.
However, this generation of GPUs ran what were known as "fixed function" pipelines.
Once the programmer sent data to the GPU, that data could not be modified.
Kind of like an escalator, you cannot get on or off in the middle. This fixed functionality
prevented developers from taking advantage of new features in graphics APIs and the like.
So the next natural step in GPU evolution would be to introduce additional customizability
into that pipeline. This came about in 2001, two years later, with the GeForce 3.
To do this, Nvidia broke apart the fixed graphics pipeline. Programmers can now send data to
customized programs called shaders.
Vertex shaders replaced the Transform and Lighting stage of the pipeline.
Pixel shaders operate at a pixel level and help with rendering the image. Programmers
can now swap that bland artificial look with more organic surfaces.
These shaders were written in a low-level language, and had their limits. But it opened
the door for more generalized programming capabilities down the line. That line came
in 2006.
In 2006, Nvidia introduced the GeForce 8 series GPU as well as a new, proprietary software
framework.
Compute Unified Device Architecture, later dubbed just CUDA, gave programmers a set of
libraries, debuggers, and APIs that made it substantially easier to program a GPU.
With this, the old fixed function graphics pipeline was now completely wiped away. The
GPU was no longer a piece of specialized graphics hardware with bits for textures, bits for
lighting, and bits for triangles.
Along with its open source cousin OpenCL, CUDA turned millions of commodity GPUs from
game machines into generalized processors. Each equipped with many thousands of cores
capable of manipulating large chunks of data in parallel.
Compare this with the other major category of general purpose processor in the consumer
market, the CPU. The CPU has far fewer cores, and is best at running operations that need
to go sequentially.
GPU performance have scaled much faster than CPUs as of late. This is for several reasons.
First, Nvidia has been able to benefit from TSMC's progression in commercializing new
process nodes.
Because a GPU's processing power is linearly proportional to the number of processing cores
it has, Nvidia could scale its GPUs' processing power by scaling its transistor counts. Some
recent Nvidia GPUs have over 20 billion transistors.
Second, Nvidia was able to scale up frequencies, with the biggest jump coming in 2016. That
year, Nvidia's Pascal GPUs used TSMC's 16nm process node for the first time.
That node was TSMC's first to use a new type of transistor - the 3D transistor called a
FinFET. It allowed chips to run at a much higher clock speed.
Third, Nvidia ramped up their GPUs' power use while at the same time benefitting from
improving per-transistor energy usage trends.
This scaling law is referred to as Dennard Scaling and it means that new chips use the
same amount of energy despite having more transistors on the same sized space.
Dennard Scaling has slowed starting in 2006, but the trend continues to some extent and
this allowed Nvidia GPUs to get much more powerful.
And fourth and most lastly, Nvidia found a new market demand for insanely powerful GPUs.
One unrelated to the gaming market.
ImageNet is a large dataset of one million images maintained by Stanford. Starting in
2010, they held a contest between various computer vision models to see who has the
best image classifier.
Like as in, identify if the thing in the image was a strawberry, blueberry, or pigeon. So
on.
In 2010 and 2011, the top performing models used hand-engineered computer vision methods.
Those didn't perform too well, with an error rate of 25% in the top-5 performers. A normal
human with a little bit of training can probably do about 5%.
Then in 2012, three researchers Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton participated
in the contest using something called a deep neural network.
Trained over 5-6 days on two GTX 580 3GB GPUs, it took first place by a mile with a 15% error
rate - ten percentage points better than the runner-up.
Since then, the winning model's error rate has dropped to the point where it regularly
performs better than a trained human.
Then in 2016, four years later, deep learning hit the mainstream when Google Deepmind's
AlphaGo won a 5 game match of Go against 18-time world champion Lee Sedol.
As you might expect, a deep neural network is based on one conceptual understanding of
how the biological neural network works. I use that phrase carefully because nobody knows
how the brain really works.
But underneath all the philosophizing is simply a titanic mathematical function. It uses a
network of simple processing elements to model complex relationships between many inputs.
The network is made up of many small "neurons". Each neuron takes in many inputs, applies
weights to them, and like some sort of judge, spits out an output.
The network can have multiple layers of these neurons, all feeding information to one another.
Structuring these in different ways leads to varying results.
In the field, people represent a network's layers of neurons using matrices. The inputs
and weights are basically just numbers within a big matrix. This takes advantage of the
GPU's relative strength in handling matrices to train networks vastly faster than ever
before.
Neural networks as a concept have been around since the 1950s. But researchers never got
good results with neural networks before. The reason has to do with the sheer amount
of data required to get reliably good results.
For instance, HBO’s Silicon Valley trained a neural network for its “Not Hot Dog”
mobile app. In order to train that simple app the developer had to collect 3,000 photos
of hot dogs. And 147,000 photos of things that were not hot dogs.
For many years, researchers had to use CPUs to train their neural networks. This took
an impractically long amount of time, because CPUs are not meant for parallel operations.
They have too few cores for doing that work.
Furthermore, they have limited amounts of memory cache. Neural network training is quite
memory intensive. You have to store the input data, parameters, and more as the inputs work
through the network's layers.
Anyway, training a network with 128 images using a dual ten-core CPU from 2012 took about
124 seconds. Extrapolate that out to a million images or more and it takes 11 days to train
a single machine learning model. That is not practical. You would never be able to iterate
on your work.
GPUs on the other hand can do it 8.5 times faster than a CPU - a million images in about
a day. A massive improvement. In other use cases, the speed improvement can get even
better - up to 40 times faster.
Scientists had noticed this possible application for years as GPUs' power - as measured by
floating operations per second - vastly scaled up. But prior to CUDA's release in 2006, there
was no easy way to actually program these operations on a GPU. Researchers had to fit
the neural network into the confines of graphical operations, which was about as hard as it
sounds.
Machine learning first helped computers read handwriting, recognize images, and play the
game of Go. Okay, who cares about that right?
Companies quickly realized however that they would be able to apply the same techniques
to more financially lucrative tasks. Things like autonomous driving, content moderation,
revenue optimization, and more.
As you might expect, these killer apps motivated businesses to quickly adopt deep learning
techniques into their revenue models. Almost all of this work has been done on Nvidia GPUs
and coded up in CUDA.
The company has invested substantial resources developing the space, correctly believing
that it would lead to more high margin GPU sales.
Their flagship GPU for enterprise-level neural networks is the A100, with some 54 billion
transistors and fabbed by TSMC. These are sold to data centers at something like $20,000
a pop. The data centers then rent access to their hardware through the cloud for as much
as $33 an hour.
The company's biggest revenue generators are still consumer GPUs for gaming or crypto.
But starting in 2017, its data center revenues started accelerating too.
In the third quarter of 2021, the company made $3.2 billion from gaming, and $2.94 billion
from data centers.
But the data center division is growing much faster, 55% year over year compared to 42%
in Gaming.
Nvidia GPUs helped enable the deep learning revolution, but competition in the hardware
space remains pretty tough.
On the GPU side, AMD bought Nvidia's former graphics card competitor ATI. They then evolved
their own line of GPUs, along with a CUDA of their own - Radeon Open Compute, or ROCm.
Unlike CUDA, AMD has opted to make ROCm open source, which offers programmers a greater
amount of access to internal GPU details.
On another front, a variety of startups have emerged offering ASICs - chips designed for
one purpose only, like machine learning. Cerebras and Graphcore are a few notable names. Both
companies have raised multi-million dollar funding rounds from big VCs, looking for a
piece of the Nvidia deep learning lunch.
Other companies have opted to create their own deep learning hardware, mostly for economic
or strategic reasons. Google, Amazon, and Tesla for instance.
This is an option few companies have the expertise and scale for. And the tech giants in particular
are fortunate since so many control their own popular deep learning frameworks.
Despite all of this competition, the machine learning revolution remains centered on Nvidia
and their ecosystem of hardware and software products - both of which have been integrated
together to deliver the best experience and performance.
Furthermore, the company’s close collaboration with AI researchers in the industry has helped
make its products the de facto standard.
This evolution reminds me of how Apple evolved the iPhone. When it first came out, there
was no App Store and third party apps could not exist. Steve Jobs imagined it to be a
phone with a few narrowly defined extra functions.
The first iPhone sold well, but it didn’t become a phenomena until Apple opened up the
iPhone hardware to outside developers, turning it into a powerful computing platform that
you can fit in your pocket.
Likewise with Nvidia and its GPUs. It took longer than the iPhone’s opening up, but
the results have been just as ground shaking. Nvidia probably didn't see that its newly
generalized hardware would eventually power the latest AI revolution.
But they didn't really need to. That's why a platform can be so powerful.