Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going take a collection of photos like these, and magically, create a video
where we can fly through these photos. And we are going to do all this, with a twist.
So, how is this even possible? Especially that the input is only a handful of photos.
Well, typically, we give it to a learning algorithm
and ask it to synthesize a photorealistic video where we fly through the scene as we please.
Of course, that sounds impossible. Especially that some information is given about the scene,
but this is really not much. And as you see, this is not impossible at all - through the power of
learning-based techniques, this previous AI is already capable of pulling off this amazing trick.
And today, I am going to show you something even more incredible.
Now, did you notice that most of these were shot during the daytime, and these are all well lit
images. Every single one of them. So, our question today is, can we perform view synthesis in the
dark? And my initial answer would be a resounding no. Why? Well, in this case, we not only have to
deal with less detail in the images. It would also be very difficult to stitch new views together if
we have images like this one. Luckily, we have a choice. Instead, we can try something else,
and that is, using raw sensor data instead. It looks like this. We get more detail, but, uh-oh,
now we also have a problem. Do you see the problem here? Yes, that’s right. In the raw sensor data,
we have more detail, but also, much more noise that contaminates this data too.
We either have to choose from less detail and less noise or from more detail more noise. So,
I guess that means that we get no view synthesis in the dark, right?
Well, don’t despair, not everything is lost yet. There are image denoising techniques that we can
reach out to. Let’s see if this gets any better. Hmm! It definitely got a lot better,
but I have to be honest, this is not even close to the quality we need for view synthesis. This one
denoises a single image. However, yes, finally, there is an opening here! Remember, in the world
of NERFs, we are not using a single image, we are using a package of images. A package contains
much more information than just one image, and hopefully, it can be denoised better. So
this is what the previous method could do, and now, hold on to your papers, and let’s
look at this new technique, called RAWNerf! Can it pull this off? Wow! Seemingly, it can.
So, now, can we be greedy and hope that view synthesis works on this data? Let’s see.
My goodness. It really does! And, we are not done yet. In fact, we are just starting. It can do even
more! For instance, it can perform tone mapping on the underlying data to bring out even more
detail from these dark images, and here comes my favorite. Oh yes. We can also refocus these
images, and these highly sought after depth of field effects will start to appear. I love
it. And what I love even more is that we can even play with this in real time to refocus the scene.
This is a very impressive set of features, so
let’s take this out for a spin and marvel together at 5 amazing examples of what it can do.
Yes, once again, this is extremely noisy. For instance, can you read this street sign?
Not a chance right? And, what about now?
This looks like magic. I love it. Now let’s start the view synthesis part, and this looks really
good given the noisy inputs. The previous, original NERF technique could only produce
this, and this is not some ancient technique. Nuh-uh. No sir. This is from just 2 years ago,
and today, a couple papers down the line, and we get this. I can’t believe it. We can even see
the specular highlight moving around around the badge of the car here. Outstanding.
Two, actually, let’s have a closer look at specular highlights. Here is a noisy image,
the denoised version, and the view synthesis. And the specular highlights are once again,
excellent. These are very difficult to capture because they change a great deal as we move the
camera around, and the photos are spaced out relatively far from each other. This means a
huge challenge for the learning algorithm, and as you see, this one passes with flying colors.
Three, thin structures are always a problem. Look, an otherwise excellent previous technique
had a great deal of trouble with the fence here, even in a well lit scene. So,
let’s see. Are you kidding me? Doing the same with a bunch of nighttime photos? There is not
a chance that this will work. So, let’s see. Look at that! I am out of words.
Or you know what’s even better, let’s be really picky, and look here instead, these areas are
even more challenging. And even these work really well. Such improvement in so little time.
Four, as I am a light transport researcher by trade, I would love to look at it resolve
some more challenging specular highlights. For instance, you can see how the road reflects the
street lights here and the result looks not just passable, this looks flat out gorgeous.
Now, talking about gorgeous scenes. Let’s look at some more of those. Five, putting it all together.
This will be a stress test for the new technique. Let’s change the viewpoint, refocus the scene,
and play with the exposure at the same time. That is incredible. What a time to be alive!
And you are saying that it does all this from a collection of 25 to 200 photos? We can shoot these
in seconds! Now, clearly, not even this technique is perfect, we can see that this does not match
reality exactly, but going from a set of extremely noise raw images to this is truly a sight to
behold. The previous, two-year old technique couldn’t even get close to these results. Bravo!
And, this is an excellent place for us to apply the First Law Of Papers,
which says research is a process. Do not look at where we are, look at where we will be two
more papers down the line. So, what do you think? What will we be able to do
two more papers down the line? And what would you use this for? Let me know in the comments below!
Thanks for watching and for your generous support, and I'll see you next time!