Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Dr. Papers.
Earlier we talked about two text to video systems, Google’s amazing Imagen Video, and also NVIDIA’s
videomaker AI. However, to the best of my knowledge, as of today, there are no easy ways
of trying them out. Now, before we start, just to make sure: tons of video AI-related techniques are
available right now, many of them for free, I’ll show you those, but their text to video system is
still in testing phase as of the making of this video. I will show you some results for that too,
however, good news, it is also expected to open to the public within a few weeks.
So this is Runway. They had a hand in the creation of the amazing free text to image AI tool,
Stable Diffusion, and they have an incredible creative suite for all kinds of AI-based video
magic. I will tell you about text to video in a moment. First, they introduced Gen-1,
this is their video to video system. Here you can take a video of yourself,
then choose a style, and bam! You have a completely new, stylized video. I think
just this one feature makes it worth trying this system as it has of uses,
but it has a ton of other really creative modes as well. We can mask out a dog, and only change
that part of the video, which will get tracked as this good boy moves around in the scene. Lovely.
It can also remove the background behind us in a video, and track both the foreground and
background over time. Also, if we have an image that we would like to expand, it can also perform
text to image with the appropriate constraints and it will expand it in a way that we often
can’t even tell that something happened here. We can also erase parts of an image,
and replace them with a text prompt as well. Now, wait a second, these two are image-related
operations. But we are talking about videos here. So what is going on? Well, one,
these photos can be used for Gen-1 as a style to a previously existing video, or even better,
get this. You can also create a video by adding a few photos that were taken a few seconds apart,
and like magic, it creates a video out of them. Yes, that means understanding what these photos
depict, and synthesizing every pixel of every image between these photos. Absolutely incredible.
And their suite is full of features we haven’t even discussed yet. Colorization
with or without using a text prompt, blurring faces while tracking them,
and so much more. Now, this is what you’ve been waiting for! Onwards to text to video.
Note once again that this is still in testing phase, and will hopefully be part of the main
suite within a few weeks. I was lucky enough to be able to try it and it did not disappoint.
Before you ask, the answer is yes, of course I generated a video of Fellow Scholars holding
on to their papers. Wow! I was told that it is not yet that good in generating humans,
and if this is the not good, I can only imagine how good the launch version will look like! This
is excellent. My first text to video experience ever. Loving it. What a time to be alive!
And here is my other favorite, fluid simulations, well, in this case, video footage of splashing
liquids which also worked really well. Some of the results were not as great,
so a little cherry picking was necessary here, but in the end I got some really tasty results.
I was also told that the system is improved every week, so by the time you are watching this video,
it will likely have improved a great deal already. The current system supports videos up to 1500x900
in resolution. Not bad! And that is easily good enough so that some folks are already
creating incredibly good movies with it that are several minutes long. Imagine that soon,
every creative person will have access to such an amazing system and can let their
imaginations run wild. All of us! This is the future of video making.
So if you wish to try this suite, note once again - text to video is coming later,
you can sign up for free on Runway’s website. The link is in the video description.
I will note that Runway is a previous sponsor of Two Minute Papers,
and had nothing to do with this video apart from letting my try it and answering my questions.
This is their first time seeing it too. So, what do you think? What will you make with
this? Let me know in the comments below, and for now, let the experiments begin!
Thanks for watching and for your generous support, and I'll see you next time!