Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
A few episodes ago, we discussed a new research work that performs something that they call
differentiable rendering.
The problem formulation is the following: we specify a target image that is either rendered
by a computer program, or even better, a photo.
The input is a pitiful approximation of it, and now, because it progressively changes
the input materials, textures, and even the geometry of this input in a 3D modeler system,
it is able to match this photo.
At the end of the video, I noted that I am really looking forward for more differentiable
rendering and differentiable everything papers.
So, fortunately, here we go, this new paper introduces differentiable programming for
physical simulations.
So what does that mean exactly?
Let’s look at a few examples and find out together!
Imagine that we have this billiard game, where we would like to hit the white ball with just
the right amount of force and from the right direction, such that the blue ball ends up
close to the black spot.
Let’s try it.
Well, this example shows that this doesn’t happen by chance, and we have to engage in
a fair amount of trial and error to make this happen.
What this differentiable programming system does for us is that we can specify an end
state, which is the blue ball on the black dot, and it is able to compute the required
forces and angles to make this happen.
Very close.
But the key point here is that this system is general, and therefore can be applied to
many-many more problems.
We’ll have a look at a few that are much more challenging than this example.
For instance, it can also teach this gooey object to actuate itself in a way so that
it would start to walk properly within only 2 minutes.
The 3D version of this simulation learned so robustly, so that it can even withstand
a few extra particles in the way.
The next example is going to be obscenely powerful.
I’ll try to explain what this is to make sure we can properly appreciate it.
Many years ago, I was trying to solve a problem called fluid control, where we would try to
coerce a smoke plume or a piece of fluid to take a given shape.
Like a bunny, or a logo with letters.
You can see some footage of this project here.
The key difficulty of this problem is that this is not what typically happens in reality,
of course, a glass of spilled water is very unlikely to suddenly take the shape of a human
face, so we have to introduce changes to the simulation itself, but at the same time, it
still has to look as if it could happen in nature.
If you wish to know more about my work here, the full thesis and the source code is available
in the video description, and one of my kind students has even implemented it in Blender.
So, this problem is obscenely difficult.
So you can now guess what’s next for this differentiable technique…it starts out with
a piece of simulated ink with a checkerboard pattern, and it exerts just the appropriate
forces so that it forms exactly the Yin-Yang symbol shortly after.
I am shocked by how such a general system can perform something of this complexity.
Having worked on this problem for a while, I can tell you that this is immensely difficult.
Amazing.
And hold on to your papers, because it can do even more.
In this example, it adds carefully crafted ripples to the water, to make sure that it
ends up in a state that distorts the image of the squirrel in a way that a powerful and
well-known neural network sees it not as a squirrel, but as a goldfish.
This thing is basically a victory lap in the paper.
It is so powerful, it’s not even funny.
You can just make up some problems that sound completely impossible and it rips right through
them.
The full source code of this work is also available.
By the way, the first author of this paper is Yuanming Hu, his work was showcased several
times in this series, in one of the earlier videos, we showcased his amazing Jello simulation
that was implemented in so few lines of code, it almost fits on a business card.
I said it in a previous episode, and I will say it again.
I can’t wait to see more and more papers in differentiable rendering and simulations.
And as this work leaves plenty of room for creativity for novel problem definitions,
I’d love to hear what you think about it.
What else could this be used for?
Solving video games faster than other learning-based techniques?
Anything else?
Let me know in the comments below.
What a time to be alive!
This episode has been supported by Weights & Biases.
Here you see a beautiful final report on one of their projects on classifying parts of
street images, and see how these learning algorithms evolve over time.
Weights & Biases provides tools to track your experiments in your deep learning projects.
It can save you a ton of time and money in these projects and is being used by OpenAI,
Toyota Research, Stanford and Berkeley.
Make sure to visit them through wandb.com/papers or just click the link in the video description
and you can get a free demo today.
Our thanks to Weights & Biases for helping us make better videos for you.
This episode has been supported by Weights & Biases.
Weights & Biases provides tools to track your experiments in your deep learning projects.
It can save you a ton of time and money in these projects and is being used by OpenAI,
Toyota Research, Stanford and Berkeley.
It is really easy to set up, so much so that they have made an instrumentation for this
exact paper we have talked about in this episode.
Have a look here!
Make sure to visit them through wandb.com/papers or just click the link in the video description
and you can get a free demo today.
Our thanks to Weights & Biases for helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!