The first version of Photoshop i ever used was Photoshop 6.0 released an insane 20
years ago. As a graphic designer I've done my fair share of scoffing whenever some sci-fi thriller or
crime show would enhance some grainy CCTV footage and get a razor sharp image out the other end.
I've had enough experiences with clients providing thumbnail size jpegs and expecting
to miraculously blow them up to poster size to know the limits of software all too well. But
over the last handful of years something's changed. There's been a new suite of buzz words:
AI, machine learning, deep learning and neural networks. At first it seemed like marketing
hype - and often with commercial software it can be - but then I started watching
the channel Two Minute Papers and seeing the results being generated in machine learning
research. I even fudged my way through running a few jupyter notebooks on Google Colab for myself,
and to be honest, it's made me reassess some things. So in this video we'll take a critical
look at some notable examples of zoom and enhance tech from fiction, then we'll see
how close machine learning research comes to bringing those ideas to reality in 2020. Is AI
a magic bullet that will completely revolutionize image processing? Or is it just the latest flavor
of technobabble hand waving for lazy hollywood hacks? Well today i am gonna get to the truth.
[YEEEEAAAAHHHH CSI Miami style intro music]
To begin with, let's embrace the meme with this classic enhanced button treatment from CSI.
There's the robbers vehicle. Wait, wait go back a few frames.
Look at that, they drive by and then they back up before they park.
What about the car? Plate's obstructed. It's not much.
Wait, what's that in the windshield? Some kind of sticker.
Miami University.
This has many of the hallmarks of bad zoom and enhance: an impractical graphic user
interface controlled by keyboard without the mouse moving for example. Notice
that the cursor on screen never moves. But more importantly,
in the source video we barely can make out the original subject. To be generous
it might be 4 pixels worth of information. But let's use our imagination and pretend that this
security footage was actually shot in ultra HD 4K and they were just scrubbing through some low res
proxy footage somehow... in 2005... on a Pentium 4. If we start with the zoomed pre-enhanced image
things immediately aren't looking good. We can make out the zero five but the rest of the badge
is a vague brown stain. Traditional bicubic resampling just makes a fuzzier brown stain.
Can machine learning do any better? There are many research papers on upscaling or super resolution
using a single source image, but let's start with the user-friendly option
with some off-the-shelf solutions. Let's try Gigapixel AI from Topaz Labs, a standalone app,
and let's also check out Adobe Photoshop's new "super zoom" feature currently in beta under
their neural filters suite. Gigapixel tries in vain to find details to enlarge but there's just
not enough to work with. This image is only 22 by 16 pixels nowhere near enough to resolve text. As
context, for our Latin alphabet, 5 pixels is the shortest possible height for legible bitmap fonts.
What Gigapixel comes up with looks like a chunky tumbleweed. Photoshop does even worse. Super zoom
basically gives up at this tiny size. If i enlarge the image with traditional upscaling first,
then run the learning-based method, we end up with a brown stain that's only marginally different,
and no clearer, than the original pixel soup. So what's the minimum size which will give us
legible text? Well if we about double our input resolution Gigapixel starts to show some promise.
We can make out the zeros and twos well enough, but the one is still rough and LOT is starting to
look like LCI. Photoshop needs even more pixels to get something usable. But realistically at these
scales neither app really does much in improving legibility compared to the untreated image,
making the "enhance" step pretty much redundant. However in theory things might be different if
we used multiple video frames, not just one. In 2018 Google rolled out a feature on Pixel
phones called "super res zoom". The research paper behind it was later presented at SIGGRAPH 2019.
To boil it down simply: taking a photo on the Pixel actually takes a series of multiple images
in quick succession, like frames of a high speed video. Then using machine learning it
pulls additional image details from the minute (sub)pixel level differences between these
images and synthesizes them into a larger, clearer image. Now this leverages the involuntary movement
in your hands taking a photo on your phone, but in theory an outdoor security camera recording
a stationary subject might have enough variation thanks to wind movement to allow for similar post
processing. Unfortunately Google haven't made their code open source, so we can't test that.
So far we have seen so much fiction but let's see what's next.
Nat, how's it coming with the kidnapping footage? I'm just enhancing a reflection
from the car window. Yeah, I'd like three cheeseburgers,
three french fries and three cherry pies. Found a clean frame on one of the kidnappers.
Creepy thin man! Ooh good, now we have a bad
guy all we have to do is look for him. Find out if he works for Corwin.
So breaking it down, what's being enhanced in this scene is a reflection in a car window,
shot by a security camera. Let's put aside the issue of resolution we already dealt with
in CSI, how plausible is it to extract a clean image from a reflection? Well, a research paper
published this year called, "Learning to see through obstructions", proposed a method of using
a convolutional neural network to take a video input including a reflection and output separate
reconstructed videos of the reflection and of the clean background. This technique uses data from
multiple frames of video moving sideways in parallax, so it wouldn't work in this
scenario with footage from the security camera. The results do show some visible artifacting,
but separating a reflection and background is a task that would take days of intensive labor for
each frame for a human technician. Even for the highest skilled Photoshop guru or VFX artist,
and to me this is really quite impressive stuff! Then we have the upscaling of the face itself.
Now, our previous example from CSI showed that this pixel count is really too small for text,
but human faces are a special case. Machine learning requires huge amounts of training data
and some of the largest data sets have been human faces. So maybe there's some potential here? Well
a paper was released this year called "PULSE" which tries to do exactly this. With a little
bit of clean up, we can feed this low res image to PULSE and see what it comes up with. And...
I'd say most of these are plausible as human faces, but as for matching the input photo...
I feel the algorithm's imagination is a little too strong. The way the generative model works is, it
tries to make a result that when down scaled back to pixels, fits well with the original low res
input. None of these really resemble the output that Cameron Diaz got. But then that's likely
because that close-up is a completely different photo, that was added in post-production. You can
actually spot the original "enhanced" photo still in her hand in the wide shot. So what happens if
we take PULSE and work backwards from this other image? We'll, the results are actually worse.
This technique really works best with portraits that are taken straight on and this image has
Crispin Glover looking over his shoulder. Comparing the "ground truth" high res image
to the PULSE upscaled portraits, it's fair to say it does a pretty poor job at matching likenesses.
If the Angels had used this tech they'd have spent the remainder of the movie hunting for
a half-melted John Travolta. It's not a unique problem to this particular model, but a very
real problem in machine learning is underlying bias in training data. What comes out is going
to resemble what goes in and the training data is going to skew the results in a certain direction.
We needn't look far to see evidence of this issue. This tweet of PULSE turning Barack Obama white
went viral earlier this year, and there's a very important conversation to be had about the lack
of diversity and other forms of unconscious bias baked into data sets that train these algorithms.
The data set that pulse used for training is called "CelebA", composed of images of celebrities
and other public figures. So biases in the training data are not deliberate and malicious per
se, but they are reflective of biases in our real celebrity culture. That's a huge topic and not the
focus of this video, but if you're curious I've linked some articles down in the description. Our
final piece of Hollywood magic in this scene, of course, is going from a black and white image to a
color photo. Now we already saw some colorization in PULSE when we gave it a grayscale input.
The resulting faces actually have a very light skin tone and clearly red lips. This
again is a result of the training data bias, since it was trained only on color images,
it can only output color images, regardless of input. But there are also specialized,
learning-based techniques for automatic colorization. One of the best known projects
is called "Deoldify" and here's how it processed a black and white version of our final image.
Not bad as far as plausibility, but definitely it still has that colorized look. The skin tones look
like one flat plane, rather than the more natural variations of a color photo. But these techniques
can only give us a plausible output, not a factual one. Which leads us to our final clip...
This is the one showing promise. This is the security camera at the underwear store.
Freeze there. Rotate us 75 degrees around the vertical, please.
Freeze there. Times 10.
Focus on the drop.
Enhance, then forward frame by frame. All right, now just before the view
is blocked there's a shape change in Dean's bag. See the shadow variance?
See? The shadow's wrong. Zavitz changed
the configuration of Dean's packages. Is it a tape?
It's hard to say for sure, these things are... Can the computer take us around the other side?
It can hypothesize, Chris.
What do you think? It looks a lot
bigger than the tape, so maybe it's nothing. Maybe it's everything. Let's get it and find out.
Notorious! This scene is probably one of the most fantastical of all time. Surely,
this is pure science fiction? Or... perhaps not? A paper called "PIFuHD" was released this year,
which can take a single still photo of a person and generate a full 360
3D model. Machine learning can really do some amazing things with image processing! This is
really cool research and would have been nearly unthinkable, even as recently as five years ago.
That said, let's see what it can do with this security footage. Our first problem
is multiple subjects, so we need to isolate down to just the runner. The image is also cropped,
so let's do a quick fudge to make a full length image, and run it through our model and we get...
a monster made of oatmeal.
So what went wrong? Well, first is our angle. All of the examples we see from the research paper
are full length portraits taken at eye level, not from overhead. Second issue is
resolution - there's just not enough to work with here. And lastly the movie actually got one thing
dead on right: "It can hypothesize, Chris!". All of the results we've seen are purely theoretical.
As we've seen, machine learning techniques are bringing sci-fi closer to reality than
ever but there are some limitations that AI will simply never overcome. No AI will
be able to make information out of nothing. It can only ever synthesize a plausible solution.
Just like if I show you this image of John McClane from the front, given the huge data set of life
experience you have, you can probably guess pretty accurately what his back might look like. What you
couldn't know - and no AI could ever work out - is whether he has a gun taped to his back with only
the limited information you have. The failure cases with PULSE also illustrate this problem.
When we know that this is a low resolution photo of Lucy Liu this is obviously a wrong solution.
But that isn't the problem this algorithm is trying to solve. It's trying to synthesize a
plausible human face that fits this series of low resolution pixels. Even a theoretical general or
strong AI would still be susceptible to the same traps that humans can fall for. Sure,
PULSE gets this completely wrong but could you correctly guess the subject of this image? It's
not Obama, but his Indonesian doppelganger. It's impossible to tell at such a low res. Even a super
intelligent AI would need to work off assumptions and probabilities where information is limited.
That's why, impressive as it is, learning-based image manipulation is going to be a boon for
the creative sector, but not for forensics. There's a reason why the term "hallucination"
is used for extrapolated information produced by neural networks. If you think about, it it's not
actually information - it doesn't reflect any reality - it's more like machine imagination.
The techniques we've looked at today are very much still in their infancy.
Machine learning - often called AI for marketing purposes - will become increasingly sophisticated,
reliable and commonplace in creative workflows. A lot of the bugs will get ironed out and refined
over time. PIFuHD, for example, has poor temporal cohesion - that is what produces the flickering
between frames in this final animation. This problem's already been hugely improved on in other
image processing algorithms, and it's only a matter of time before further work fuses these
ideas and creates a more refined version. It seems unlikely that deep image enhancement will truly
be a one-click operation and human aesthetic judgment will always be an important factor.
But certainly the old days of manual Photoshop retouching are well behind us and a new suite
of tools is slowly replacing and sometimes augmenting the way the images are being processed.
If you want to keep up with the latest research in machine learning and AI, I cannot recommend highly
enough the youtube channel Two Minute Papers. Links also to all the original research down in
the video description. The vast majority of people talking around AI and machine learning these days
are, unsurprisingly, data scientists. People with a coding or academic background.
I hope as a designer who's worked for years with traditional photo retouching
and image processing workflows, I was able to share a different perspective on this topic.
If you're interested in seeing more videos about how design technology
and culture interact subscribe to the channel, if you haven't done so already,
and if you enjoyed this video please offer a like to the almighty algorithm.
Also let me know in the comments below what your favorite "zoom and enhance" scene is from a movie
or TV show. I'm wondering which of the ones that have stuck with people over the years?
And if you're interested in this topic, I have a lighthearted end-of-year video including some
neural filters and memes, which will also be coming up on the channel soon. As always,
my name is Linus, thank you very much for watching and I hope I'll see you in a future video.