This SciShow video is supported by Linode!
Go to linode.com/scishow for a $100 60-day credit
on a new Linode account.
At first glance, math and art might seem like total opposites.
Math is full of rigid rules, while art is free-flowing and creative.
Math can be churned out by a computer, while at least most art
still involves a human that can love and suffer and experience the world.
But as different as the two seem, sometimes math can be the key
to revealing secrets of art, and seeing deeper
into artwork than we otherwise could.
[INTRO]
Now, when we talk about using math to decode art,
we’re not talking about the kind of algebra
or calculus you can scratch out on a chalkboard.
The kind of math we’re talking about is mostly done by computers,
which can process images in ways that humans cannot.
This came in handy when it came to solving a mystery
tied to the masterpiece known as the Ghent Altarpiece.
The altarpiece is a 15th-century oil painting
by the Flemish brothers Hubert and Jan van Eyck.
It’s made up of more than a dozen different paintings,
with side panels on hinges so it can be opened or closed,
which lets people display different paintings for different occasions.
When the wings are open, it’s about as wide as
a two-car garage door and almost twice as tall.
And it is incredibly detailed. In the painting on one panel,
there’s an open book, just centimeters tall, with some barely visible text.
Art historians were curious if it was a real text,
but the words were completely illegible.
You have to understand, this painting has been through a lot.
During wars and riots, it was taken apart, looted, recovered,
and reassembled over and over again.
So it’s not surprising that the text wasn’t exactly looking its best.
But a team of mathematicians thought they might just be able to fix that.
The main problem was there were cracks throughout the painting.
This happens naturally thanks to changes in humidity.
As different parts of a painting absorb and release moisture,
they expand and contract.
This made it hard to tell what was text and what was just a crack.
But the team thought if they could identify the cracks in the painting,
maybe they could fill them in and make the letters clear again.
Fortunately, they did not have to start from scratch here.
There are lots of digital image processing algorithms out there
that can pick out little branchy patterns that kind of look like cracks.
Like blood vessels in medical images,
or rivers and roads in satellite images.
One way to do this is by using something called convolutional filters.
Generally speaking, a convolutional filter takes a plain image
and applies some operation to all the pixels in that image.
That process creates a modified image based on the original.
If you have ever used an Instagram filter before,
you’ve probably used one of these.
But in addition to adding sparkle to your Instagram posts,
convolutional filters can also be used to
identify certain features in images.
It works like this: The program starts with a small grid of values
called a kernel. The kernel is going to transform the image one pixel at
a time, using the pixels around the one it focuses on to do so.
As it passes over an image, it multiplies each pixel by the value that
the kernel overlays on it, then adds up the values you generate from it.
Then, that new value is divided by the sum
of the values in our kernel, like this.
That new number gets placed onto a duplicated version of the old image,
over and over again until the whole thing has been analyzed.
And changing the numbers that you put into
that kernel will help you identify specific parts of images.
For instance, if you want to detect edges,
you can use a kernel that has negative values on one side
and positive values on the other side, like this one.
If all the pixels in one area have the same value,
when you add up all the products, the negatives and positives
will cancel each other out, and the final result will be zero.
But if one side is darker or lighter, the pixels will have different values,
like this. And the result will be some positive or negative number.
These results help a computer figure out where an image changes
from dark to light which is what we see as an edge.
And in the end, it outputs a map of all the edges in the image.
Now that might not seem especially useful if your goal is
to identify cracks, not edges.
But a crack is essentially just two edges right next to each other.
So you can use these edge-detecting filters to pick out cracks.
And that is what the team did with the Ghent Altarpiece.
Now once the team had identified them, they digitally painted them in,
leaving behind only the text.
Which was actually legible! At least, to the art historians.
And not only that, they could even tell what book it came from,
a 14th-century religious text.
So, thanks to a little help from math,
we can analyze this masterpiece in a whole new way.
But sometimes, it’s not just the surface of a painting
that artists and historians are interested in.
Because a finished painting doesn’t always tell a full story.
Today, X-ray images reveal that many famous paintings
were painted on top of other artwork.
For instance, X-rays showed us that Picasso’s famous painting
“The Old Guitarist” has a figure of a woman painted underneath it.
This kind of thing fascinates art lovers,
because these hidden paintings can reveal what else
was on the artist’s mind, or what they wanted to cover up.
But even though X-rays can reveal that a hidden painting exists,
it’s really hard to actually recreate that painting.
Features from different images get mixed up,
and it’s hard to tell which lines belong to which painting,
especially if a canvas has been painted over several times.
But once again, math can help.
And some of the same researchers who worked on decoding the text
in the Ghent Altarpiece were able to develop a sort of answer key
for future research into paintings under paintings.
To be clear, the altarpiece itself isn’t
hiding any older artwork underneath the paint.
But it’s actually the perfect work for training a machine
how to separate two images because of how it’s built.
Like we said before, it was created to show different scenes
depending on certain holidays and functions,
which means that the panels on its shutters are two-sided,
there’s a painting on each side.
So if you take an X-ray image of any of these panels,
you will see both sides at once.
It’s like a puzzle with an answer key:
The X-ray image showing the two layers on top of each other
is the puzzle, and the regular photograph of each side is the answer key.
You can use this to train a computer to separate
a superimposed image into two different ones.
To do this, the team of researchers used what’s called
a convolutional neural network, a type of computer model that
can be trained to recognize images.
You basically give it a bunch of examples until it can pick out
patterns well enough to do the task on its own.
These models use different convolutional filters
to create maps of features like edges, curves, or circles.
So, they basically summarize any image as a collection of features
and then they match this collection of features to some known object.
In this case, the researchers trained a neural network
to tease apart two images in one mixed image.
And to get that to work, they sort of worked backwards.
They started with two regular photos of each side.
Then they had a neural network generate
two separate X-ray images based on each one.
And since these were based on photographs,
there was no overlapping image from the other side.
Next, it recombined those X-ray images to create a mixed X-ray image
and compared this to the real X-ray image
showing the overlapping paintings.
It was programmed to repeat this process over and over
until the two images were as similar as possible.
In the end, the team had a neural network that was extremely good
at pulling out two images that combined to make an overlapping image.
But cryptic messages and hidden images
aren’t the only mysteries swirling around old paintings.
One of the biggest ones is just who made it?
Like, remember how we said the Ghent Altarpiece was painted
by both Hubert and Jan Van Eyck?
Well, one question that has never been entirely answered
is which brother did what.
We know Hubert started the project,
and Jan finished it after Hubert died partway through.
But so far, no one has been able to sort out exactly
what each brother contributed.
And all over the art world, there are questions like this.
Not only are there plenty of unsigned or
otherwise anonymous paintings out there,
but there are plenty of forgeries floating around, too.
So if we can use math to help figure out who created an artwork,
there are a lot of potential applications.
Now, that’s not to say that we humans are totally in the dark
when it comes to finding out who a mystery artist is.
The same way you can tell apart people’s handwriting
by recognizing patterns, an art expert can sometimes tell you who made
a painting based on patterns in the brushwork.
But mathematical tools can take this up a notch.
They can summarize all of those intangible qualities that make up art,
like brushstrokes and texture, into a bunch of numbers and statistics.
One way mathematicians extract key features of a painting is
by taking a hi-res image and blurring it, first just a little bit,
and then more and more.
Each time they blur it, they subtract the blurry image from
the less blurry image right before it.
The difference between the two images is all of the information
that was lost when the image was blurred.
You can imagine that in some spots, like, the sky,
there won’t be a big difference between
the blurred version and the original,
because there’s not much detail there.
But in other spots, say, a garden full of flowers,
there will be a bigger difference because in the blurred version
you lose a lot of information about the details of
the flowers and the fruits and the leaves.
So, each time you blur and subtract,
you capture some key features of the original image.
As you do this with blurrier and blurrier images,
you extract more and more levels of detail.
And after many iterations, you can take all of the lost detail from
each level of blurring and add it together.
What you end up with is a summary of the original image
that just contains its key features.
They have a name for this, it’s called a wavelet decomposition.
Once a painting has been summarized this way,
mathematicians can use a number of different statistical tools
to see how it compares to another painting.
One approach is to use what’s called a Markov model,
which is a way to summarize patterns mathematically.
Essentially, this model breaks down a pattern
into a grid of little pieces, called states.
And then it looks at the probability that
one state will be neighboring another.
It can do this for textural patterns, too.
So, it’s a way to represent the patterns of
Van Gogh’s brushstrokes with math.
By comparing the Markov models of different paintings,
you can tell how similar the textures are,
and hopefully that gives you an idea if you’re looking at art
by the same artist or two different ones.
And a group of researchers put this method to the test,
using 101 scanned images from two art museums.
Most were by van Gogh, a few were most definitely not by van Gogh,
and 13 of them were up for debate.
The team used wavelet decomposition to analyze the paintings,
and while their results weren’t perfect,
they were pretty successful at separating the van Goghs
from the non-van Goghs, or van fauxs, if you will.
So far, this kind of approach hasn’t been used widely,
but in theory it could help solve the mystery of which brother
painted what in the Ghent Altarpiece.
Now, none of this is to say that math is taking the place of art experts.
But it can be another tool in their toolbox.
So as much as math and art might sometimes seem like
fundamentally different ways of exploring the world,
putting the two together can give us new insights
and new ways to answer age-old questions.
Thanks for watching this SciShow video, supported by Linode!
Linode is a cloud computing company from Akamai
that provides storage space, databases, cloud services,
and award-winning support to you or your company.
And when I say “company” I’m using that term as strictly or broadly
as your situation requires, because Linode can help out COOs
at any sized corporation, teachers at grade school or universities,
and self-employed content creators going it alone
or with a large team behind them.
Linode provides a variety of services that
can benefit each of those diverse roles.
And you can use Linode to meet the extent of your needs
without wasting your money on services that someone else might want.
Then, if you end up transitioning from the lone ranger
to a larger support team, Linode can scale with your growth
and still provide dependable access to your
streaming, grading, and analytics needs.
To get started with Linode you can click the link
in the description down below or
go to linode.com/scishow for a $100 60-day credit
on a new Linode account.
[OUTRO]