Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to see how this new AI is able to take a piece of text from us,
anything we wish, be it a squirrel dressed like the king of England,
a car made out of sushi, or a humanoid robot using a laptop, and, magically,
it creates not an image like previous techniques, but get this, a full 3D model of it. Wow.
This is absolutely amazing. An AI that can not only create images, but create 3D assets? Yes,
indeed, the result is a full 3D model that we can rotate around and even use in our
virtual worlds. So, let’s give it a really hard time and see together what it is capable of.
For instance, OpenAI’s earlier DALL-E text to image AI was capable
of looking at a bunch of images of koalas, and separately, a bunch of images of motorcycles,
and it started to understand the concept of both, and it was be able to combine the two
together into a completely new image. That is, a koala, riding a motorcycle.
So, let’s see if this new method is also capable of creating new concepts by building on previous
knowledge. Well, let’s see…oh yes! Here is a tiger wearing sunglasses and a leather jacket,
and most importantly, riding a motorcycle. Tigers and motorcycles are well understood concepts,
of course, the neural network had plenty of these to look at in its training set, but combining the
two concepts together, now that is a hint of creativity. Creativity in a machine. Loving it.
What I also loved about this work is that it makes it so easy to iterate on our ideas. For instance,
first, we can start experimenting with a real squirrel, or, if we did not like it,
we can quickly ask for a wooden carving, or even a metal sculpture of it. Then,
we can start dressing it up, and make it do anything we want.
And sometimes, the results are nearly good enough to be used as-is even in an animation movie
or in virtual worlds, or even in the worse cases, I think these could be used as a starting point
for an artist to continue from. That would save a ton of time and energy in a lot of projects!
And that is huge. Just consider all the miraculous things artists are using the
DALL-E 2 text to image AI and Stable Diffusion for, illustrating novels, texture synthesis,
product design, weaving multiple images together to create these crazy movies,
you name it. And now, I wonder what unexpected uses will arise from this being possible for
3D models? Do you have some ideas? Let me know in the comments below!
And just imagine what this will be capable of just a couple more papers down the line. For instance,
the original DALL-E AI was capable of this, and then, just a year later, this became possible.
So, how does this black magic work? Well, the cool thing is that this is also a diffusion-based
technique, which means that similarly to the text to image AIs, it starts out from a piece of noise,
and refines this noise over time to resemble our input text a little more.
But this time, this diffusion process is running in higher dimensions, thus,
the result is not a 2D image, but a full 3D model. So, from now on, the limit in creating
3D worlds is not our artistic skill, the limit is only our imagination. What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!