I am an upscaling addict. But I’m not alone! The other month, when I was upscaling the
video for Gorillaz – Feel Good Inc, I happened upon somebody who had already done it, who
had posted his efforts over on Vimeo. I rejoiced, for I learned that I wasn’t
alone with my upscaling addiction, nor in my tastes about what I was upscaling! But
where we were different was how we went about our upscaling. On his Vimeo video, he had
stuck closely to the source material. He kept the framerate the same AND minimised shimmer
and artifacts by choosing a ‘blobby’ style of upscale, rather than a ‘sharp’ one.
Given the cartoon-style of the video, that’s probably sensible. And I don’t think anyone’s
going to criticise his upscaled version, whereas with mine, you COULD argue that the upscaling
artifacts, at points, make it look worse than the original.
Upscaling doesn’t give you something for free. The more you upscale something, the
further it deviates from the original, and the further it ventures into the weird, abstract
realm of neural networks. It’s why you can’t just upscale an upscale until it’s got infinite
detail. That extra detail is fake, and upscale too far and it makes the whole thing look
FAKE. But in my opinion, if I’m going to spend
several weeks of my life upscaling a video, I’m going to go as far as I think I can
with it! No half measures. I want to see just how much ‘detail’ I can extract from an
image, and I want to up that resolution and framerate as much as I like, even if it does
result in some bugs and artifacts. My choice, and if you don’t like it then YOU can upscale
stuff for yourself. And that’s when I found another upscaling
addict with EERILY similar tastes to myself. He has upscaled White and Nerdy, and is known
only as… the Art Master. I reached out to him to ask him how he got
so white and nerdy.
His story was a sad one. After Youtubing for 10 years, without warning, his account got
deleted, and he was left homeless. He spiralled downwards into addiction and dependency, becoming
a rampant upscaling addict. But cruel fates abound, he was barred from the world’s biggest
video-sharing site! Where could he turn to? It was then he discovered the safe refuge
that is Vimeo, home to other upscaling addicts. And he began to hone his craft.
He hand-upscaled each and every one of the 4,110 frames of the music video, White and
Nerdy. And the results are stunning. I wanted to learn the ways from this true master of
his craft! He said it seemed fitting to ‘digitally
restore’ White and Nerdy, because the copy on Youtube, that most people will see, is
in a tragic state! It’s not HD- only 480p. And from these tell-tale, interlaced artifacts,
it looks like half the video’s missing. If we compare the youtube original to his
final version, the difference is night and day. But stop for a second, and look- do you
see something? Something… suspicious? Yeah, the original version is cropped, missing
out on stuff at the top and bottom of the screen. As much as he’d love to take credit
for hand-drawing this extra detail… he didn’t. He found a better copy of the recording to
base his upscaling on.
He spent considerable time searching for the best version of the video before he began
the lengthy process of upscaling it. Because there’s nothing worse than spending almost
a year upscaling something, only to find a better copy somewhere else. He even checked
out Weird Al’s official, DVD version of the White and Nerdy video. But that turned
out to be just as bad as the Youtube version. The best quality version he could find was,
quite fittingly, on Vimeo! He explained to me that, while Youtube is
the most popular, other sites often have different versions of the same videos- and for older
videos especially these can often be better than the Youtube version, as was the case
with White and Nerdy. So while he has no idea where sites like Vimeo get their copy of the
music video from, he’s very happy that they exist.
He admits, he was a bit tempted to crop his upscale and to pretend he had worked miracles
on the Youtube version, but ultimately, he didn’t do this because he loves the video,
and wants to give you the best version of it possible. Why hinder the effort and confuse
people as to what restoration’s capable of. And that’s ultimately what this project
was for him- it was about collecting the fragmented remains of this music video, and bundling
them together into the best possible version available on the internet. So while the Vimeo
copy was obviously the best to base this project on, it’s not without its own problems. Which
we’ll be getting to shortly! Firstly, he upped the framerate from 24 to
48 FPS, using a program called DAIN, which uses AI to fill in the gaps between them.
Upping the framerate of something is itself, controversial. Most movies are only 24 fps.
The Hobbit was an exception to this and was recorded in 48. But people complained that
it was TOO smooth. That it made it look cheap, like a sitcom. There’s this belief that
lower framerates in movies makes them feel more epic, and in some ways, believable, because
by distancing it from smooth reality it uses the power of your brain to bridge the gap.
I guess the implication here is that fans of lower FPS have bigger brains or something.
He then went off on a lengthy tangent, explaining to me how they were WRONG. He insists that
higher framerates are ALWAYS better and that sticking to the slideshow that is 24, just
because we always have, is the digital equivalent of Stockholm syndrome. It may be what we’re
used to, but being a connoisseur of the finer things in life… he wants more. And from
the perspective of a video-restorer, it’s a lot easier to drop 48 to 24 than it is to
go from 24 to 48. So, at the risk of making his job redundant, he believes that EVERY
video should always be recorded in the highest quality and framerate possible, and then dropped
to cater to personal preference. Everybody’s entitled to their own opinion about stuff,
but if you disagree about this then you’re wrong. End of.
So White and Nerdy started at 24 FPS, and he upscaled it to 48. Or 47.952, to be precise.
You might notice that the version on Youtube claims to be 30. But that’s a lie! It was
uploaded with the wrong settings. You might think that 30 is bigger and therefore better,
but it results in one in 5 being duplicate frames which will make the video even choppier
than had it been uploaded in the lower 24 FPS. But he was cool with that, since it made
his remastered version EVEN BETTER. Why 48, instead of 60? Well, since the original
is 24, 48 means that it still uses all of the original images, and it’s just the new
frames between these that risk artifacts from framerate conversion- that’s half of them.
But with 60, since that divided by 24 is 2.5, that means that just 1 in 5 frames are the
originals, meaning 4/5ths of them are at risk of having artifacts! So it’s more computing
power, and you lose some of the original information. He knows that the average person will consider
60 FPS to be better. But he didn’t make this for the average person; he did it for
art masters. 48 FPS doesn’t just make things smoother.
He thinks it makes people move more believably… even giving the illusion of more detail?!
I for one am sure I picked up on more subtle movements on faces than I ever did from the
original. Just don’t slow it down, or you’ll notice
the weird, computer limitations of framerate conversions! This was one of the out-takes
he saved for me because he felt it was genuine nightmare fuel.
But nothing in life is ever easy, is it. Vimeo’s video might be vastly more detailed, but it
had one horrendous downside: and that’s its low bitrate. If you throw a complicated
scene at it, like one with lots of grass or movement, it doesn’t have enough bitrate
to go around and ends up HALVING the framerate! So bits of this video were only 12 FPS, which
means the bugs with the motion conversion get even worse.
He had a difficult decision to make: for these scenes, should he use the higher-quality,
but lower framerate vimeo copy? Or the rubbish, but consistent Youtube one? ‘WHY NOT BOTH’
he hears you cry. He says it was too obvious where they merged, though he got away with
it in scenes such as this one. But you can see the quality of the background surrounding
the Segway is quite a bit lower. But he felt it was worth doing, because converting
the high quality, but 12 fps, version of this clip made the wheel spin the wrong direction,
and made it look like the grass was eating the front of the machine. And yes, this was
using DAIN, which is regarded as one of the best framerate converters. That whole scene
was a vehicle-crash, motion-wise. This scene was uploaded at the wrong framerate in the
first place, hence the ghostly blur you see around it. There was nothing he could do about
that, unless he redrew each frame by hand. Which he says he seriously considered.
In THIS scene it got even more confusing! Most of the scene ran at 24 fps, but the bush
in the background was only 12, resulting in strange juddering motion. So he had to use
DAIN on that separately then merge it all together. He says working on this 3 second
clip was 5 days of hard work well spent.
Another example of using both versions of the video was with the dice roll. There was
too much motion, and too few frames, to use anything but the Youtube copy for this scene.
But then the moment the dice landed he switched back to the other copy again. Which you can
just about make out if you study each frame individually.
In other scenes you might THINK you noticed some unrealistic motion, but he can assure
you that you also missed many examples in this video where the conversion remained very
good. He thought at first it was only about 2 scenes in the video that had dropped to
12 fps, but actually there are like 7-8, some just a few frames long. It was a long process
of deleting all duplicate frames, then manipulating the converted footage about in the editor
to get it to sync up with the original again. He says it was crap.
He then went off on a lengthy tangent about public toilets, and how sub-optimal they are
in their current gender-divided state. He says if ‘safety’ is the reason then it
shouldn’t be by gender, but rather be split between adults and children. As it is, he
says you always see queues for the women’s yet none for the men’s. It’s incredibly
lopsided and a poor use of real-estate. He says he is all for the trans rights movement
getting their own toilet. In fact, he wants EVERY bit of the LGBT movement to get their
own toilets in the hope that the toilet situation ends up getting so confusing and fragmented
that the whole system collapses, leading to a golden age where large unisex toilet becomes
the norm.
…And in some scenes of his upscale, he wondered if these dropped frames were deliberate to
give the impression of comedic motion. Is it still right to smooth that out, or would
that be destroying the soul of the video? Ultimately, he felt he was over-analysing
it, and that probably even the editor of the original video hadn’t put as much thought
into this stuff as he had. He’ll tell you what- when you study every frame of a video,
you notice how many shortcuts and botches were made in the first place.
And he’s added a few of his own, too. He thinks that AI is a more useful, but less
powerful tool than many think. Even DAIN has its limits. Which is why, to achieve best
results, he used another program: PAINT. And he’ll admit here that if you pause the video
at certain points, you’ll spot these embarrassing image manipulations, like this drawn in racquet
during the Sour Cream scene. Slowed down, they may look rough, but in motion they actually
make things LESS distracting than the bugs the AI came up with in the first place.
SO! he had the whole video in 48 fps. Then he needed to upscale it.
He warns of the dangers of upscaling. He says that art is never finished, only abandoned.
You’re free to invest AS much time into improving it as you like but you face rapidly
diminishing returns which will burn you out if you’re not careful.
So with this project, he smartly discovered that halving the original resolution and upscaling
that to 1440p quality yielded the best general results, so he went and did the whole video
in that. This meant the whole video had already been upscaled to a decent quality, and could
be ‘released’ whenever he wanted- rather than to be left as a half-finished project
on his hard drives forever more, like so many of the doomed projects before this one.
Speaking of doomed projects, he then went off on a lengthy tangent about shrinkflation-
the process in which items seem to get smaller and smaller over time. He mentions how Toblerone
was rightly blasted for how they lopped a huge amount of chocolate off their design
the other year, but complains that Terry’s Chocolate Orange has managed the same but
more slyly by hollowing each segment out slightly. And he KNOWS that crème eggs are smaller
AND worse tasting than they were before being bought out by Kraft. And he swears that McCoy’s
crisps were 32g a few years ago, and have steadily gone down to 30, then 28.5, 27 and
now 25- the same weight as standard crisps are! That’s like, 1 good mouthful and then
you’re done! He no longer buys these products, partly because he refuses to reward this practice,
but mainly because he thinks buying them would signal to the companies that we’re stupid
and haven’t cottoned onto their sly scheme. And he claims to deserve more respect than
that.
…In terms of resolution, the original White and Nerdy video was 720p, but thanks to low
bitrate, parts can seem pixelated, or blurry. If you upscale this, it will still look pixelated
or blurry. But he finds that by reducing the resolution and by upscaling that, it can look
better! The question is: how low do you drop the resolution first?
Like he said, he found half-resolution 360p to be a good compromise. But he also made
a 540p, 270p, 198p and 135p versions of each scene, then upscaled each of them to see which
bits looked best. Faces and text look BAD if you drop the resolution, so he typically
left faces at 360, 540 or even original 720p resolution, but then upscaled to try and take
the edge off the pixellation. Other bits of the image, on the other hand, would often
look better at a LOWER resolution. We’re talking about things like grass, trees and
hair. So for these, he’d drop the resolution, then use AI upscaling to fill in details which
might not be realistic, but that look better at a glance. Particularly upscaled hair, which
looks great. Truly what upscaling was made for.
So he’d blend together 2, 3 or even 4 different upscaled images for each frame based on what
he thought looked best. Which is hard enough on a single image, but on some of the animated
scenes became incredibly time-consuming. Like he said, he didn’t think he’d bother doing
this on each and every scene which is why he had a generic 360p upscale to fall back
on in case he got lazy. He ended up manually tweaking about 85% of
the scenes in this video. Some he’d attempt several times before he was happy with them.
His main goals were to prevent faces and text from looking too weird, since that’s stuff
that people will notice. Those were the boring bits. The fun bits were scenes like the park,
with lots of leaves, trees and grass. He went to town on these! Sometimes he’d denoise
the upscaled pictures to increase clarity and to reduce colour bleed.
The other problem with upscaling is the shimmering it produces. Especially when the camera’s
moving, details will transition in and out of existence every frame, and this gets worse
as the resolution it’s upscaled from decreases. It’s an unfortunate side-effect from upscaling.
He wonders if it could have been reduced by upscaling BEFORE converting to a higher framerate.
But then, even that comes with downsides. It’s all a compromise- he expects to get
comments saying how he SHOULD have done it, but he can assure you, there would have been
compromises with those as well.
Speaking of compromises, he then went on a lengthy tangent about vanilla and how it’s
a great flavour yet, for some reason, is often used to describe ‘plain’ ice cream. He
thinks it’s one of life’s greatest injustices and, for all the horrible plain tasting vanilla
ice creams he’s had, he laments equally at the thought of all the nice ones he hasn’t
had in fear of them being plain. He wishes they could just be what they claim to be,
rather than to be some kind of sadistic ‘surprise mechanic’ to capitalise on thirsty people
on hot days.
…and at that point, I ran out of time so we won’t be able to discuss the upscaling
process any more than that, but the whole thing took him about 9 months and he is VERY
happy with his baby. This has been a passion project. A tragic tale of addiction. Like
so many before him, he says that this project was never finished- merely abandoned. But
he did it to a level he’s proud of- and more than that- to a level that he think represents
what digital restoration can hope to achieve, with the tools available in the year 2020.
He expects technology to improve, so it will be interesting to see what’s possible in
5 years or so. And this is why you should never delete the originals- upscaling is a
fun hobby, and produces the version that one he’d rather watch, but if given the choice
of just one copy, the original copy is the one that should be kept. Always. You can always
turn it back into the upscaled one again. You can’t necessarily get the original back
from the upscaled one. As a fellow upscaling addict, I can understand
his passion. With a project like this, there are different stages to it, each rewarding
in its own way. He enjoys making decisions about how to approach the project, then leaving
his PC to do the hard work is always satisfying. Then, at the end of that, he gets a sense
of satisfaction and surprise when he sees how good the end result looks. It’s an excuse
to revisit media from earlier in his life, to analyse it in greater detail than ever
before, and at the end of it, to be the proud owner of the highest quality version of it
in the WORLD. It’s a bit like the thrill you’d get from being a dictator of a country,
or world-record holder of sorts. But on a more pathetic, nerdier level. And with greater
risk of copyright infringement. As fun as it is, he says it also feels good
to finally move on from such a time-intensive project. Next he’ll probably do Jizz in
my Pants. And then upscale the video of it. Amish Paradise would be the next Weird Al
one of choice, but he doesn’t think the recordings have as much potential as White
and Nerdy did. I’ve personally tried upscaling Smash Mouth-
All Star, since the copy on Youtube is horrendous. But I don’t think you can do much with it.
That Youtube version IS the highest quality recording of it, by considerable margin. And
it’s already so fake and processed, there’s not much I can achieve by upscaling it. Unless
they provide the source files, what you’re seeing there is about as good as you can hope
for. But there is hope for the future! Digital
remasters are popping up for all of the old classics- Pet Shop Boys remastered a load
of their old videos just the other day. The way it’s going, I reckon it’ll only be
a matter of time before companies pop up to professionally remaster things with help from
AI. In a way, it’s already happened. When the
Hobbit came out in 3D, 5K, 48 fps, I should have known that Peter Jackson was at risk
of becoming an upscaling addict. And he has since become one, producing ‘They Shall
Not Grow Old’- that thing with footage from World War 1, all digitally remastered using
a number of clever techniques. It’s a very exciting time for this kind
of stuff. And I’m happy to have been able to speak to the Art Master, whose White and
Nerdy video truly is a work ofart.