Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
I can confidently say that this is the most excited I’ve been for a smoke simulation
paper since Wavelet Turbulence.
Wavelet Turbulence is a magical algorithm from 2008 that takes a low-quality fluid or
smoke simulation and increases its quality by filling in the remaining details.
And here we are, 11 years later, the results still hold up.
Insanity.
This is one of the best papers ever written and has significantly contributed to my decision
to pursue a research career.
And, this new work performs style transfer for smoke simulations.
If your haven’t fallen out of your chair yet, let me try to explain why this is amazing.
Style transfer is a technique in machine learning research where we have two input images, one
for content, and one for style, and the output is our content image reimagined with this
new style.
The cool part is that the content can be a photo straight from our camera, and the style
can be a painting, which leads to the super fun results that you see here.
An earlier paper had shown that the more sophisticated ones can make even art curators think that
they are real.
However, doing this for smoke simulations is a big departure from 2D style transfer,
because that one takes an image, where this works in 3D, and does not deal with images,
but with density fields.
A density field means a collection of numbers that describe how dense a smoke plume is at
a given spatial position.
It is a physical description of a smoke plume, if you will.
So how could we possibly apply artistic style from an image to a collection of densities?
This doesn’t sound possible at all.
Unfortunately, the problem gets even worse.
Since we typically don’t just want to look at a still image of a smoke plume, but enjoy
a physically correct simulation, not only the density fields, but the velocity fields
and the forces that animate them over time also have to be stylized.
Hmm.
Again, that’s either impossible, or almost impossible to do.
You see, if we run a proper smoke simulation, we’ll see what would happen in reality,
but that’s not stylized.
However, if we stylize, we get something that would not happen in mother nature.
I have spent my master’s thesis trying to solve a problem called fluid control, which
would try to coerce a smoke plume or a piece of fluid to take a given shape.
Like a bunny, or a logo with letters.
You can see some footage of what I came up with here.
Here, both the simulation and the controlling force field is computed in real time on the
graphics card and as you see, it can be combined with Wavelet Turbulence.
If you wish to hear more about this work, make sure to leave a comment, but in any case,
I had a wonderful time working on it, if anyone wants to pick it up, the paper and the source
code, and even a Blender addon version are available in the video description.
In any case, in a physics simulation, we’re trying to simulate reality, and for style
transfer, we’re trying to depart from reality.
The two are fundamentally incompatible, and we have to reconcile them in a way that is
somehow still believable.
Super challenging.
However, back then when I wrote the fluid control paper, learning-based algorithms were
not nearly as developed, so it turns out, they can help us perform style transfer for
density fields, and also, animate them properly.
Again, the problem definition is very easy, in comes a smoke plume, we add an image for
style, and the style of this image is somehow applied to the density field to get these
incredible effects.
Just look at these marvelous results.
Fire textures, starry night, you name it.
It seems to be able to do anything!
One of the key ideas is that even though style transfer is challenging on highly detailed
density fields, but it becomes much easier if we first downsample the density field to
a coarser version, perform the style transfer there, and upsample this density field again
with already existing techniques.
Rinse and repeat.
The paper also describes a smoothing technique that ensures that the changes in the velocity
fields that guide our density fields change slowly over time to keep the animation believable.
There are a lot more new ideas in the paper, so make sure to have a look!
It also takes a while, the computation time is typically around 10 to 15 minutes per frame,
but who cares!
Today, with the ingenuity of research scientists and the power of machine learning algorithms,
even the impossible seems possible.
If it takes 15 minutes per frame, so be it.
What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next
time!