- I'm here today at Peacemaker Filmworks
who have worked on small projects you might've heard of
like Deadpool 2 and the new Ghostbusters.
They're known for their crazy camera cars,
but today their space is being used
for possibly the craziest 3D capture setup in the world.
Created by Volumetric Camera Systems,
we've got over two hundred CPU cores,
one hundred terabytes of storage,
and if you follow me in here,
two hundred and thirty nine cameras,
capturing a staggering nine hundred and twenty k
image thirty times a second.
Which begs the question,
who needs to capture that much detail?
Oh yeah, it's Neill freaking Blomkamp,
the director of District 9, Elysium,
and the film we're looking at today, Demonic.
It dropped yesterday and the way they use 3D capture in it
is just mind bendingly cool.
And, terrifying.
And do you know what's also terrifying?
This segue to our sponsor.
Thanks to Origin PC for sponsoring this video.
Origin PC's can be customized with 11th Gen
Intel core processors,
and NVIDIA GeForce RTX 30 Series graphics
with Max-P design.
Backed by a 24/7 support team,
check out Origin PC at the link below.
(upbeat music)
This whole setup starts with the Yi camera.
A 4K action camera that's, well, inexpensive.
Volumetric Camera Systems or VCS
would've obviously preferred to have cinema cameras
all around here but,
when you have to buy two hundred of something plus spares,
it gets expensive fast and,
it's not like these are bad little cameras.
Each one has the Sony IMX377 sensor,
the same one as in the Google Pixel and,
when you have over two hundred of them
pointed at a subject,
you can get quite a bit of detail out of it.
Each camera has custom firmware
and an extra pin installed in the USB cable,
that allows for sync across all of the cameras.
This allows for the clocks to be synced
between every single camera so,
if I hit play on the master camera right here
which is conveniently marked by not having a front plate,
all two hundred and thirty nine cameras
start recording and then,
click it again and, they all turn off.
The special thing about this rig though,
is how all of that is turned into 3D data.
This is accomplished using feature detections,
so, all of the tape in the background
creates these sharp edges,
so AI can go in and figure out where each camera is
compared to all of the others in the scene.
Also every two cameras,
so like these two right here,
are used to create a stereo pair
and using some complicated trigonometry,
they're able to calculate the depth
of anything in front of them.
This is one of the keys to why the VCS camera system
is so sick.
By using this depth data in post,
they can just tell the software to discard
everything behind the subject,
removing the need for green screens
and allowing for complete three sixty capture
of the subject.
Again, using a bunch of complicated math and AI,
NVIDIA is actually a sponsor of VCS
and contributed a GV100 to help crunch
those huge equations,
they're able to create a point cloud of the subject
with all of the depth data.
On it's own, this is really cool,
it removes the need for a bunch of post processing time
because the animators don't have to worry about
things like cloth or hair physics, but,
the huge advantage over motion capture
is the ability to shoot full color textures
of the actors in full makeup and costume.
This does have it's own set of problems though.
The large five meter ring of cameras
surrounding me is great for filming
larger action sequences but,
when you consider each one of these cameras
has a wide angle one hundred and fifty five degree lens,
if you're standing even a meter or two away,
there won't be nearly as much texture data,
which is where the high resolution array comes in.
This was originally made using a trampoline
that was cut in half and held together with zip ties
but, they upgraded to this steel frame
for shooting with Neill.
Anyway, sequences will be planned out
so that the actor can do their larger movements
in the large array and then land in the high resolution
array for parts that require more detail.
Obviously the camera that is just a foot away
from my face is going to be getting a lot more detail
than the ones further out.
Even that though isn't quite enough detail for VCS,
like, they are going up against cinema cameras,
so they're up scaling all of the footage
from 4K to 8K for every single camera.
They custom made an algorithm
by capturing 4K footage on these cameras,
down scaled it to 1080p,
and then used the 4K footage to train the AI
for their best chance at up scaling it to 8K.
And with all that in place,
it still isn't quite the level of detail
that is needed for a full blownsies Hollywood film.
Unfortunately there isn't really any way
around the detail at this point,
but, that isn't really a problem.
For Demonic, they did a lot less post processing
than I expected,
and left in a lot of the artifacts that happen naturally
and it really adds to the horror of the film.
It's super cool how Neill and the crew
even used stuff like the point cloud data
to act as a transition between real life
and the simulation parts of the movie.
As you've probably guessed by now though,
running over two hundred cameras at once
uses a crap load of storage, like,
this thing is creating nine thousand five hundred
gigapixels per second,
and seven to fourteen million polygons per frame.
Which translates into about ten terabytes of data
for every twenty minutes of shooting.
The first day of shooting on Neill's movie,
poor Tobias and Scott of VCS
were here until six AM grabbing the data
with the two computers that they had, so,
clearly an upgrade was needed.
Surrounding the camera way,
we have twenty four computers from Oats Studio.
Each one is equipped with a ten Core i7-6950X,
nice, sixty four gigabytes of RAM,
and a GTX 1080 Ti.
They don't actually need the GPU's,
but they certainly need the CPU RAM and I/O
to copy the data from each stand of cameras.
Sorting all two hundred plus video files
was a huge problem for VCS,
so they created their own custom program
called Clippy Copy,
named after everyone's favorite paperclip.
That is based off of the Robocopy API
that automatically pulls and sorts
the insane number of files that are made
every time that they press play.
Once it gets to the post processing stage,
they create a global alignment of all the cameras.
This is basically looking at the features on the wall
to figure out where each camera is in 3D space.
After that, they create a depth map
of everything in the room
using the stereo pairs and, from there,
it's very simple to create a point cloud
of the entire scene.
Every point in the cloud will give you
not just the position in 3D space,
but also the RGB data of every point.
Every three points in the point cloud
is then Poisson mapped to create polygons
and, using some extra AI goodness,
using the camera data,
the textures are then improved again.
After all that we get what basically looks like
the best video game character model ever.
So, like right here we have a capture
of Tobias's nan just chilling in Unreal Engine,
and, I'm able to just sort of float around her
in, oh geez, Unreal Engine using WASD.
It's just so cool.
And she's just playing back in real time in 3D,
this is crazy.
This processing obviously requires
some balling computers that were provided
by MAINGEAR in HP so,
down here we actually have two tiny backpack rigs
rocking an RTX 2080.
With all of that, the director has an incredible amount
of flexibility in post, so,
they're able to pick camera angles,
import the character into any environment they want,
and relight the scene to their liking.
In order to relight the character though,
there can't be any shadows
which means they need a lot of light.
And, they've got a lot of light.
In our office we actually use these same ARRI SkyPanels,
we've got three of them,
and each one will run you about twenty grand,
so, naturally you're using twenty four of them here,
but, even that was not enough light.
While we were here today
they actually installed an additional eleven
ARRI M40's which are an additional forty grand each,
so, unsurprisingly, they rented the lights for this shoot.
It's also insanely hot in here right now.
Probably can't tell because my shirt's black,
Ittstore.com, but I'm drenched.
You don't necessarily need a balling studio setup
like this though to do volumetric capture.
Since they're effectively just action cameras
on a stand, VCS can record anywhere
and just, you know, footage goes locally on the cameras
and they have batteries built in.
This way they can set up in say like,
the jungle, at a farm, or at a trade show
and capture Jensen Huang and give us the data,
allowing us to have a little 3D Jensen
wherever we want.
For an extra portable setup,
they created this little box
that has an image sensor in each one of these holes.
And you can set up, say, five of them
around the subject and boom, you're good to go.
There are additional applications
outside of cinema 3D capture.
One of their biggest funders is actually Epic Games
who hope to use this technology
to do things like the Travis Scott concert,
but, without all the time and money
required to 3D model the whole thing,
like, he could just do a set surrounded by these cameras
and, bippity bop, he's straight into Fortnite.
They're also looking into medical
and military applications, so,
say there's a world renowned surgeon
who knows how to do a really obscure knee surgery.
That surgery can then be recorded in 3D
and played back from any angle for training purposes.
Future improvements are planned for both hardware
and software, so hopefully one day
cinema grade 3D models can be captured volumetrically.
Obviously getting better, higher resolution cameras
would allow for a better result,
but as more ground truth data is captured and imported,
they can train AI algorithms
to better interpret the footage, so,
hopefully we'll be able to get a similar level
of fidelity to these two hundred cameras
with just seventy or so.
So huge shout out to Tobias and Scott
for showing us their awesome tech,
and Neill Blomkamp and Chris Harvey
for sharing their incredible expertise.
Demonic is available now, so,
check it out at the link below,
and as you'd expect from one of Neill's films,
it's a fantastic technological showcase
and proper frightening.
And also, huge thanks to segue's
for bringing us to our sponsor, Ting.
Ting Mobile has new rates
that make it easier to see how much you can save
by switching.
You can get unlimited talk and text
for ten dollars, data plans that start at fifteen dollars,
and their new set twelve plan,
with twelve gigs of data for thirty five dollars,
and unlimited data for forty five dollars a month.
If you like their previous pay as you use plans,
it's still there with Ting Mobile's Flex plan
starting at just five dollars per gigabyte.
Data can also be shared if you have a family plan,
connect more phones, and save more.
You'll still get national and award winning coverage
and pretty much any phone will work with Ting Mobile.
Ting Mobile now does have the perfect plan for everyone,
no matter what your needs are.
So, check them out at linus.ting.com
and receive a twenty five dollar credit.
If you're looking for something else watch
while it isn't really similar to this video,
but, Porch Talk can review was loads of fun to shoot,
so, go watch that.