In an older video, I talked about the immensely
powerful and beautifully complicated image processing pipeline in today's smartphones.
I artfully left out the chips themselves.
Mostly because I found those image sensors too difficult to explain and I was tired.
But I have received many requests about this fascinating silicon technology. So
for this video, let’s take a look at the chips that make digital cameras possible.
## Beginnings
In 1889, the German physicist Heinrich Hertz noticed that ultraviolet light
somehow knocked electrons of certain atoms out of their normal orbits.
The number of electrons emitted from the metal atoms was directly
proportional to the light's intensity. It happened almost instantaneously.
And the kinetic energy of the ejected electrons was independent of the light's intensity. This was
a strange phenomenon that classical physics could not explain, and it took many years to reconcile.
Finally, in the early 1900s,
a theoretical physicist proposed a theory to resolve this discrepancy.
He took a concept first proposed by Max Planck. The concept that light
consists of tiny packets of light called photons.
His theory said that the energy in that light is proportional to its frequency.
And that it only knocked out an electron if its energy was above a certain threshold.
This theory had many of its tenets proven in later experiments and won the 1921 Nobel.
That theoretical physicist? Albert Einstein.
## Classical Devices
Really though, the photoelectric effect
as it would be called underpins how our modern electronics devices sense light.
The concept of solid-state photosensing or imaging is to have light fall onto a
two-dimensional array of photosensitive elements.
That light ejects electrons from the array’s materials,
changing its electrical properties. We can then measure this for indirect information about the
light itself. If we can properly interpret this data, then we can create an image.
One early device that used these principles was a vacuum tube-based gadget called the
photomultiplier tube or PMT. It was suited for detecting ultra-low levels of light.
A photocathode at one end of the tube emits electrons when exposed to light, via the
photoelectric effect. Those electrons then travel towards an anode on the other side of the tube.
Along the way, they are consecutively multiplied using 5-7 sets of dynodes.
As a result, you can turn 1 target photon into 100,000 or even 10,000,000 electrons.
Which is exactly the behavior you want for a photo-detector.
PMTs and other such classical devices paved the
way for their silicon-based descendants down the line.
Silicon is an excellent material for an image sensor, capable of responding to a
wide range of lights. So researchers have wanted a solid-state silicon image sensor
for a very long time, but it was not until the 1960s that a viable contender emerged.
## CCDs
In the late 1960s, a pair of scientists at Bell Labs - George Smith and Willard
Boyle - were researching semiconductor bubble memory.
This type of memory has a bunch of electrodes sitting on top of
a silicon substrate with a layer of oxide insulation in between.
If you apply a certain voltage to the electrode,
then it creates a "potential well" in the substrate right underneath the electrode.
These potential wells are capable of storing a charge by capturing energy within it. The charge
in turn can represent a bit of data, if you were to use this technology as a semiconductor memory.
Another nifty thing that you can do with this series of
electrodes is that if you were to pulse them in a certain sequence,
you can move the potential well and the charge it is storing across the substrate.
Doesn't sound all that special, right? Ultimately,
bubble memory was too slow to compete as a viable memory technology. But researchers
realized that these wells can also collect and convey analog signals.
So rather than using these wells to store charges as bits of memory, what if you were to use them
to collect and transport packets of electrons ejected from silicon via the photoelectric effect?
And there, we have the operating principle of the CCD. You take an array of electrodes,
which are modified to serve as your pixels.
When exposed to light, silicon in what is called a "photodiode" ejects electrons which are then
collected by the electrodes. A voltage cycle is then applied to carry those packets of electrons
down or across the array - like ants carrying leaves - towards another unit at the array's edge.
This unit then converts and boosts these electron packets from the whole
array into a sequence of voltages which external electronics can interpret as a
2-D image. If necessary, the voltages are also digitized.
The whole thing is synced to a clock and happens in a cycle.
## Rise of the CCD
One of the biggest benefits of the CCD was its sensitivity.
The big challenge with prior solid state image sensors was their efficiency. In other words,
how much of an electrical signal can they generate when exposed to light.
Most image sensors had light efficiencies that were too low, which meant that they were
not all that sensitive to light. This makes it harder to see the scene in front of it.
CCDs are sensitive enough to light. Combined with superior dynamic range,
low levels of noise, and smaller pixels, CCDs became the dominant
solid stage image sensor technology after their invention in the 1970s.
A variety of CCD makers entered the market - including General Electric, IBM, Hughes
Aircraft, Texas Instruments, RCA, Fairchild, Phillips, Hitachi, Thompson, Intel, and Sony.
## CCD Downsides
CCDs quickly took most of the imaging market,
providing the sensors for things like camcorders and digital cameras. But they had their warts.
One of the big downsides of the CCD is that if the
chip is still exposed to light during voltage conversion stage - sometimes
called the frame readout - then you can get undesired signal issues.
These issues manifest as "smearing",
and manufacturers have dealt with it with a mechanical shutter sitting in front of the
sensor or by separating one part of the chip into a dark area. Like a dark room.
The former is a moving thing that can break and also requires power.
The latter basically doubles the cost of fabbing the die.
Furthermore, when combined with its accompanying systems like the analog-to-digital converter,
the whole thing eats a whole lot of power. I remember having a camcorder when I was a kid,
and those things always seemed to run down their batteries in 1-2 hours.
So yes, the CCD was a solid solid-state image sensor and they set the standard
for image quality. But it also fell short of the slick and smooth dream
of having a convenient digital camera on a chip.
## CMOS
Complementary Metal-Oxide Semiconductor or CMOS image chips have been around for a long time,
sporadically researched since the 1960s.
Notably in 1964, IBM announced the invention of the scanistor,
a small dime-sized chip that can produce an output signal depending
on light. IBM said that its scanistors can scan documents for faxing purposes.
Scientists at Westinghouse, Fairchild and RCA also demonstrated similar setups that time,
but their photo-sensitivities were very low.
When CCDs were invented in 1970, it sort of sucked out all the oxygen from the
space. Few teams - notably Hitachi - continued studying CMOS image sensors,
mostly in the pursuit of a chip for camcorder applications.
Hitachi chased the dream for a while, introducing on-chip techniques to vary chip exposure times
and suppress the flicker you get with indoor light. However, they moved on after failing
to sufficiently suppress random image noise artifacts - a big deal in low light conditions.
So CMOS image sensors languished a bit while CCDs had their day in the limelight.
But the 1990s saw new interest and money flood back into the space. With improving processes,
people saw an opportunity for further integration with the rest of the imaging system.
This means faster readout, less power, better packaging,
and the image functionalities enabled by digital processing. The image processing pipeline!
CMOS image sensors can do this, while the increasingly specialized CCD can not.
The critical technological turning point was the introduction of second-generation "active
pixel sensors" ... and now it is time for me to explain how this accursed thing works.
## How It Works
A CMOS image sensor has an array of pixels.
Each of those pixels in turn has a photodiode - the
light-sensitive element which collects the silicon electrons ejected by light.
There are also control transistors included as well. These serve
several functions which we will discuss later.
When the system needs to receive and interpret the image,
we "access" or "readout" the pixel or row of pixels just like as if it were a memory.
Changing the way we did this is at the heart of second-generation CMOS image sensor technology.
With first generation "passive pixel" technology,
invented back in the 1960s by a Fairchild alumni, there is just one transistor inside
the pixel - the control transistor switch that is triggered when we want to readout.
After that control transistor switches on,
the photodiode's charge flows into a wire towards an amplifier. This amplifier at the
end of the wire converts the charge into a voltage for image processing off-chip.
The passive pixel approach gives you smaller,
simpler pixels with just 1 transistor each. But because the pixels are so simple,
we have no way of amplifying the charge before it flows into the wire.
The charge tends to be very small. A small charge into a big wire results in
a high amount of noise - over 12 times more than what you get from a CCD. Noise is bad.
All this noise means that the passive pixel approach does not scale. The more pixels we
put into the array or the faster we try to readout those pixels, the more noise we get.
Soon after the passive pixel was invented, people recognized that they can fix its noise problem by
adding a buffer or amplifier to the pixel to boost the charge before it gets processed.
This is the active pixel sensor - with multiple
transistors inside each pixel rather than just one.
In a way, this is quite similar to how the CCD works. It just packages more functionalities
into a single pixel. We no longer need to carry charges across the pixel array for
conversion in an support system. We do the work inside the pixel itself.
But easier said than done! Semiconductor processing nodes were not capable of actually
pulling this off in a commercial sense until really, the 1990s.
## Commercialization
An early implementation of the active pixel sensor concept called the photodiode
active pixel sensor was bandied around since the 1980s, particularly in Japan.
VLSI Vision Limited also contributed some critical patents in the space.
But it was not until 1993 that things really got going at NASA's Jet Propulsion Lab or JPL.
Spacecraft cameras at the time used CCDs,
which as I mentioned earlier needed a lot of support systems and used a lot of energy.
NASA wanted something "Faster, Better, Cheaper" so Eric Fossum - JPL's chief
of image sensor research - and his team developed their own implementation of
the active pixel sensor - the photogate active pixel sensor.
They also introduced a new technique called Correlated Double Sampling to
significantly suppress noise and improve image quality. This is where you sample
the pixel's image data twice and take the difference to filter out noise.
Back in the 1960s, NASA went all in on a strange new technology called integrated circuits.
But in the 1990s, they turned up their noses to CMOS image sensors.
So in 1995, Fossum formed a company called Photobit to commercialize their implementation.
Their pioneering PB-100 chip would power the Intel Easy PC Camera, Logitech's QuickCam and more.
They produced other chips for military, medical and industrial use too. Photobit was acquired
by Micron in 2001, spun off as Aptina Imaging, and then acquired by ON Semiconductor in 2014.
## CMOS vs. CCD
What occurred in the late 1990s and early 2000s is a classic disruption story. In the early days,
CCDs had superior image quality to CMOS active pixel image sensors.
Because there are more transistors within each active pixel, the pixels themselves
are physically larger which limits how many of them we can have on the array.
Furthermore, having more transistors inside the pixel meant that less of the pixel's physical
size can be allocated towards collecting light - referred to as "fill factor".
Early fill factors could be as little as 26%. This was in comparison to the CCD's 100%.
The counterpoint was that having multiple transistors in the pixel - like a charge
amplifier, exposure timer, and even an analog-to-digital converter - meant that
each pixel can deliver better image data faster.
Furthermore, CMOS image sensors delivered digitized data, were relatively cheap,
and ate less power. This was why mobile phone makers started using CMOS image chips in 1999.
## Fall of CCD
Moore's Law changed everything. CMOS image sensors are compatible
with traditional semiconductor processes while CCDs are not.
So when advanced process nodes breached the 0.5 micrometers or 500 nanometers
level in the mid-1990s, the industry achieved a transistor density per pixel
to be sufficiently competitive against existing CCD technology.
Pixel density experienced its own version of Moore's Law - with density
doubling every 18 months. Soon you were able to go from a few
thousand pixels on a sensor to what would eventually be a few million.
CMOS image sensors became the fastest ramping product category in semiconductor
history up until then - with a million 8 inch wafers produced from 2003 to 2005.
By 2007, CMOS image sensors had improved their image resolution weaknesses while
also retaining their cost, power and size advantages.
Soon CCDs only held the imaging quality advantage in low-light or non-visible light scenarios.
Cameraphones like the Nokia N95 and the iPhone paired these sensors with a mobile computer,
supercharging image sensor development and unlocking
the near-unlimited benefits of computational photography.
Smartphone applications take up to 70% of our modern CMOS image sensor market.
CCD manufacturers were slow to adapt to this rapid advancement. Sony - the
largest CCD maker - was one of the few to successfully make that transition. Perhaps
because they made their own digital cameras and saw what consumers wanted.
In 2014, Sony's IMX174 Global Shutter CMOS chip outperformed even their most
advanced CCD chips. Sales of the latter collapsed.
In January 2015, Sony said that they were shutting down their CCD
manufacturing operations and going forward with only CMOS. With this,
CMOS had swept the market, relegating the venerable CCD to the fossil bed.
## Conclusion
Now that CMOS image sensors dominate the industry, what is next for the industry?
For the past decade, the smartphone industry has demanded better cameras.
This means higher resolutions, more mega-pixels and thus smaller pixels.
But you want those smaller pixels without adding additional noise to the
whole picture. As the pixel gets smaller, its transistors get relatively larger,
and the photodiode gets smaller. This shrinks the fill factor, and thus adds more noise.
Maintaining that signal to noise ratio as pixels shrink is a big
challenge in the CMOS image sensor industry.
Each pixel size generation has a fundamental theoretical limit to its Signal-to-Noise ratio.
These smaller sizes are harder to achieve but it is still happening. Just this year,
the Chinese fabless semiconductor company Omnivision announced a 0.56
micrometer image pixel, using TSMC's N22 process.
A remarkable achievement that shows that better CMOS image sensors are still possible.