I recently got an iPhone 13 Pro. I love it.
For me, the thing that sticks out most - literally - is the camera.
For a long time, iPhones have had increasingly better cameras. The last few generations,
camera performance has headlined the marketing messaging: Get the new iPhone,
because the camera is a lot better. It is one of the few things that gets people to upgrade.
And indeed, the 13 Pro's camera takes really good pictures. But it is not just the iPhone. Virtually
all of today’s top smartphones can take images on par with anything you can get with a standalone.
The top smartphone makers are investing a lot into the space. Back in 2015,
60 Minutes reported that Apple employed 800 people to work just on the iPhone's camera. In 2021,
Xiaomi announced that they’re hiring thousands of employees to work on their phones' cameras.
How does your mobile phone camera work? And how did it get to be so good?
In this video we will look at the amazing computer and semiconductor engineering
that goes inside this impressive feature.
Traditional analog cameras work by exposing photographic film to a light.
Pretty simple. Even though the outcome is still the same - a picture - today’s smartphone cameras
work far differently than that. It is convergent evolution, but for imaging.
Your basic digital camera module has a simple physical structure:
A lens placed on top of an image sensor chip. Module makers and integrators often may add other
things like sensors and actuators to help with camera performance but that’s the core structure.
This digital camera structure is shared across all digital devices.
However, the smartphone setting offers certain advantages and disadvantages
compared to larger digital photography equipment like DSLRs.
The biggest downside is that the camera’s insides have to be a lot smaller and thinner.
Smartphone camera modules are typically housed in areas 7-10 millimeters thick.
This is because the smartphone itself needs to be portable. The device has to fit in a pocket.
Thus, modern smartphone sensors are typically about 5 millimeters by 4 millimeters in size. In
comparison, DSLR image sensors are much larger. Full-sized sensors can be 36 millimeters by 24
millimeters big. You want the image sensor to be larger so that as much light as possible falls
onto a pixel. Smaller sensors risk more motion blur, image noise, and reduced dynamic range.
Smartphone sizes also limit smartphone optics as well.
They are a lot smaller and less adjustable than what you can find in a typical DSLR.
This creates a couple major engineering challenges that camera manufacturers have to overcome.
The first is a fixed and limited aperture. Aperture refers to the opening of a lens
diaphragm. It is where light passes en route to the sensor. Smaller aperture means less light
falling on the sensor. And as I already said earlier, you want more light on the sensor.
The second is a limited zoom function. In the camera industry,
zoom refers to the ability to smoothly shift from a long shot to a close-up. Later on in the video,
we will talk more about how today's smartphones are able to achieve a pretty good zoom.
On the flip side however, the smartphone is capable
of offering much more computing power than your discrete camera.
Today's smartphone chips are as powerful as some laptops. And that has been one
of the key factors in improving the physical deficits of a smaller sensor.
The science behind "silicon photo-detectors" date back to the 1920s.
However, it would be Willard Boyle and George Smith at Bell
Labs who first proposed the idea of Charge-Coupled Devices or CCDs in 1969.
A year later, the idea became a technical reality. Boyle and
Smith would win the Nobel Prize in Physics in 2009 for their work. CCDs
captured the majority share of the early mobile imaging sensor market.
However, a new type of photo-sensing semiconductor - Complementary Metal-oxide Semiconductors
or CMOS - soon emerged with a lot more potential.
Over time, CMOS sensors began to replace CCDs in consumer and professional devices.
They consume less power, can be produced by conventional semiconductor foundries like TSMC,
and are not as expensive. Their inclusion in smartphones
solidified their dominance in the imaging industry.
With how important they are becoming to the overall device,
smartphone cameras and their sensors are a big business.
The imaging chip business is a large one. Estimated to be about $20 billion in 2020.
It is dominated by three companies - Korea's Samsung, Japan's Sony, and China's Omnivision.
Yet due to fundamental physical constraints, image sensor technology can only go
so far. At some point, the camera bump gets too ridiculous. Luckily,
more powerful computing devices are coming to the rescue. Computational photography is a fast
growing field and where some of the industry’s biggest advancements in recent years have been.
So you see a cute cat and you want to take a photo
with your iPhone. What happens when you hit the button on the Camera app?
So let us spend some time walking through the image processing pipeline.
I am not going to go through all the steps, but try to catch the major ones.
I also want to note that every smartphone manufacturer has their own unique image
processing pipeline. Even when using the same camera hardware,
smartphone companies can modify the parameters and algorithms. They might do this per their own
testing or to differentiate their camera performance from other manufacturers.
Alright let's get started.
The camera image sensor is made up of a 2D grid of photo-diodes.
A photo-diode converts photons into an electrical charge.
These electrical charges do not necessarily correspond to color by themselves.
So, each of the photo-diodes have a color filter laid on top of them.
Smartphones typically have dedicated image processing hardware. This hardware receives
the electrical charges generated by those sensors’ photo-diodes and maps them to the right colors.
These filters' arrangement depends on the sensor's manufacturer. It is referred to as a Bayer filter,
named after Bryce Bayer of Kodak - who first proposed it in 1975.
Today's modern smartphone manufacturers make color filters together into "multi-cell" pixel clusters.
These macro-pixels offer enhanced light sensitivity and flexibility.
The raw output from this stage is referred to as the "Bayer pattern image" or "Bayer
frame". And it kind of looks distorted or like a mosaic - with gaps between the different colors.
This is because each photo-diode can only correspond to one color.
So the image processor employs algorithms to help fill in the gaps of the raw Bayer image.
These demosaicing algorithms as they are called
employ a variety of mathematical methods to reconstruct the scene's actual color.
The simplest algorithm simply fills in the missing RGB values
based on the adjacent pixels. It works decently, but it does have aliasing issues at the corners.
So more sophisticated algorithms have been developed.
This methodology might seem a little weird at first glance. It implies that two-thirds of the
image data from our smartphone cameras are essentially computer generated!
However, it is mimicking what happens within the human vision system. The brain similarly compares
signals from different cone-type photoreceptors to create what we call "color vision".
Bayer himself even chose in the initial patent to include twice as many green
photo-diode filters as red or blue because that's how it is in the human eye.
After this, the image data goes through a variety of processing and finishing steps.
First, the image is processed for white balance. Our brain is able to
adapt our color vision to different amounts of illumination. Cameras need to accommodate
this and "correct" the colors in the image to look as if it was lit by a neutral white light.
Without this, the finished image looks unnatural. For example,
skin tones might look too warm/orange, or camera flashes will look too cool/blue.
This requires an algorithm to estimate the scene's illumination
and how the image sensor's color filter will respond to it.
The result is something called an illumination value and it is applied to the pixels' RGB values.
Then, the image processor might manipulate the image's colors in a proprietary way.
This can be based on the user's feedback. For instance,
the user might have selected a more "vivid" image setting.
Or it can be something pre-programmed into the smartphone camera by the
vendor to differentiate the look of their photos.
For instance, Samsung phones shoot images that look different than iPhones.
The image processor might then also apply another algorithm to reduce noise in the image. Noise in
an image refers to artifacts that are not in the original scene. Too much noise is distracting.
But if your camera is too aggressive in the denoising,
it might end up looking way too smooth and fake. Like something from Meitu.
Since they matter so much to the camera's perceived performance,
a lot of work has been invested in striking the right denoising balance.
Entire PhD theses have been written on denoising algorithms.
After this, the image processor resizes the data, adjusts the RGB values to make it more
presentable for the smartphone screen, and then save it into a JPEG, PNG, or HEIC file format.
That’s your image pipeline in a nutshell.
Now let’s talk a little bit on certain special situations and how much computation
goes into porting optical imaging features to the smartphone world.
Zooming in on something is particularly difficult to do with a smartphone. With traditional cameras,
you zoom in on something by physically moving the lens along the optical axis.
This is not easily possible with your small and skinny smartphone camera. There have been
some interesting attempts by Samsung, Nokia and Asus, but the resulting products have
a sizable camera bump. And despite this bump, the zoom performance is rather disappointing.
The dissatisfying result of optical zoom meant that most smartphones can only offer digital zoom.
This is mostly done by cropping the original image and upscaling what remains. Digital zoom values
tend to be pretty modest, since it further weakens an already iffy image resolution.
To improve the quality of digitally zoomed images, camera makers have employed
image algorithms that try to enhance the missing details. A simple one would just
try to fill in details using nearby pixel data. This hasn't quite gotten there yet.
Ultimately, the approach most modern manufacturers seem to
have gone with is to have multiple rear cameras: One with a wide field of view,
the wide camera, and another with a narrow field of view, the telephoto.
These dual aperture zoom cameras as they are called were first introduced in 2014 by an
Israeli company called Corephotonics. And it has been adopted by the leading phone makers - Apple,
Xiaomi, Oppo and so on. Now, users can swap between lens to achieve a zoom.
A few companies have gone with a different approach, called folded zoom. You have a
traditional optical axis, but use a 45 degree mirror to bend light sideways so that
you can get your zoom without making the camera thicker. A prism helps maintain image stability.
A few high-end phones by Oppo, Samsung, and Huawei have adopted this feature which they
headline as allowing for 100x zoom capability. It's a little weird, but there you go.
One of the most impressive things about the iPhone 13
has been its low light and night time performance.
It is here where the industry has really pushed the boundaries of computational photography.
The reason why I think they can do this is because the human eye kind of sucks at night.
When it gets dark, humans become color blind.
Thus, as a result, camera makers realized that they didn't have to be so faithful to the scene.
Rather, they seek to create the most colorful, noise-free low light photo they possibly can.
High quality low light photography was considered only possible with DSLRs.
They had larger image pixels and adjustable apertures with tripods,
which allowed for enough light to be captured. Smartphone camera hardware is not capable of this.
But smartphone camera makers realized that they can adapt another photography feature to overcome
this limitation: "burst processing". This means capturing and merging together many image frames.
It has its roots in astrophotography as "image stacking". A technique where
multiple night sky exposures are lined up to reduce noise and produce better images.
In smartphone burst processing, the smartphone camera captures and stores frames continuously.
Then when the shutter is pressed, it selects a frame close to the moment when you clicked
the button. Then it merges all the other frames together to create a single high-quality image.
This has its own challenges. For instance, trying to reliably align the sequence of images.
Doing it wrong can result in weird image distortion issues.
But when done right, it works really well. The 2018 Google Pixel phone first kicked
off this low light breakthrough with the Night Sight feature.
In creating this, the camera designers spent a lot of effort adjusting to jittery hands and
moving objects in the scene that could cause image blur. But the results were impressive.
Since then, other phone makers have used it as a general tool for better photography.
Burst photography has been used for denoising, increasing image resolution,
and high dynamic range compression.
Recent advancements in onboard AI processor technology have allowed camera makers to push
imaging performance to new heights. Here are a few prominent examples.
The first has to do with white balance correction. Researchers collected digital
photos and then had professional photographers manually white balance them. They then fed this
data into a machine learning model to create much more effective color constancy algorithms.
This machine learning method has been particularly helpful in low light conditions.
It was featured prominently in Google Pixel's Night Sight.
Earlier, I mentioned how camera makers tried simple algorithms to sharpen the
blur from digitally zoomed photos. Taking it one step further is to
train a machine learning model with high and low-resolution imagery data so that
the model knows how to properly sharpen and enhance blurry edges in a picture.
It works pretty well. Nvidia has applied the same concepts to digitally upscale video game
image assets to resolutions and frame rates the developers did not originally support.
They call it Deep learning super sampling or DLSS.
Another place where machine learning has really made an impact in
is bokeh. With smartphone cameras, the whole image is either in focus or not.
You cannot focus on one thing, like a person, and have the background blurred.
Computational photography has allowed mobile phones to generate a synthetic bokeh.
Modern cameras can use their second camera or a dedicated depth sensor to figure out how far away
the subject is. Then they can introduce depth blur to simulate the depth of focus effect.
This is the basis of iPhone's Portrait Mode, which at first was just for people
but now seems to have extended to other things like animals.
The phone uses AI to recognize a person or dog and blurs out the rest.
I have started to notice this bokeh effect leaking into normal camera modes too. For instance, point
the iPhone 13 at a cup on the table and it blurs out the background there too. Really interesting.
The thing that most surprised me when I started this overview was to find out just
much computer processing and image manipulation goes into creating today’s modern digital photos.
Whether we are talking about simple math operations or complex machine learning models, the
image data that gets saved and uploaded is heavily doctored from what the sensor actually “sees”.
This might be a really weird metaphor. But it kind of makes me think of a quote
in the movie Jurassic World. In it, Henry Wu, played by BD Wong, says:
> Nothing in Jurassic World is natural, we have always filled
gaps in the genome with the DNA of other animals. And if the genetic code was pure,
many of them would look quite different. But you didn't ask for reality, you asked for more teeth.
Every day, we take photos of the things around us. The image scene itself is a reality.
But the smartphone camera image of it, has over time
become increasingly less so. But that is what the people want, right? As long as it looks great,
who cares if it never looked like the reality we were looking at. Food for thought, man.
Alright that's it for tonight. Thanks for watching. If you enjoyed the video,
consider subscribing. Check out the newsletter or follow the Twitter.
Want to send me an email? Drop me a line at jon@asianometry.com.
I love reading your emails. Introduce yourself, suggest a topic, or more.
Until next time, I’ll see you guys later.