ANDREW JONES: Hi, my name is Andrew Jones
and I'm a production designer.
I've been working on The Mandalorian.
We've just about wrapped up season two.
But I just want to go back and talk a little bit
about season one and the process that we put together
for this project.
So it's a little bit an usual in its filming process.
We are using what we call a volume, which is--
on one stage we've got a 75-foot diameter cylinder with LEDs,
very high-resolution LED panels creating a round wall
and a flat ceiling over the whole surface at 7 meters high.
And on that LED wall and ceiling,
we project a 3D environment.
It's created in Unreal and it's projected onto the screens.
But it's not like a TransLite.
TransLite is just a flat image that you can film against
but if you were to move the camera,
you would realize that it's just flat.
This is actually a proper three-dimensional virtual
environment.
And the volume includes a motion-tracking system
so that as the camera moves, its movement
is tracked very accurately so that as it moves from position
to position, the content updates to reflect
the correct point of view that camera would see
in that virtual environment.
So no matter where you go, no matter where you look,
you're looking basically through the screens
into the virtual environment and it's very realistic.
And originally, Jon Favreau had been
kind of working with a lot of these ideas on Jungle Book.
There was LED panels used for interactive lighting.
The little boy in the middle of the stampede of water
buffaloes, there was a screen right
next to him of stampeding water buffaloes to create
the correct interactive light so we didn't have to actually
stampede buffaloes past him.
So that was kind of a little bit of it.
Jon had also tasked the team to create the virtual sets ahead
of time so that they can be pre-vizzed with and scouted
and developed.
Production designer Christopher Glass
worked with a virtual art department
to create virtual sets that were scoutable.
And before we even started shooting,
we were looking at what the environments would be like
and blocking action in there.
And Jon took that further in The Lion King.
Of course, there was no physical sets in that.
But he was developing tools with Rob Legato
that would create realistic camera
moves within the virtual world.
And scouting the virtual sets in VR.
So a lot of the tools that he was developing,
he brought it to The Mandalorian along
with us desire to create--
on Jungle Book, we created really beautiful
little miniature sets for the kid to interact with,
but everything else was blue.
And that's a nightmare.
So the hope would be here that we
could use LED walls with the real content, at least
to contextualize the scene, just so you
could see what was there.
In the best case, we would have content
on the screens that were so good that you can actually
include it in the shot as what's known as final pixel now.
So when we started working in Unreal,
we were aiming at getting the highest quality visual content
that, in essence, was photoreal.
So the idea of having a screen behind the action that
is filmable, that was our initial or our initial ask.
But once we started testing that, the DP, Greig Fraser,
realized that it's really not enough just
to have beautiful content behind the performance.
You need to immerse them in a three-dimensional environment.
Because it's all light and it's all
immersing the character in the lighting of that environment.
And in the case of The Mandalorian,
where the character is basically wearing a silver armor,
it's reflecting the environment as well.
So we kind of had no choice.
If we'd done that on a blue screen,
you would have had spill all over him--
a lot of work to do there.
So in a case, we didn't know if this was going to work.
But we figured that, best case, we would
be getting in-camera finals.
But at the very least, we were going to get great lighting
and be able to sort of perform the scene
and what everybody could see what the set was.
So this approach meant that we had
to really come up with a new pipeline that
was going to work for this.
We were creating, essentially, final content early--
much earlier than usual--
before we even shoot.
And that meant the virtual art department was
going to have to create content early enough
to hand it to ILM so they could ingest it into their pipeline
and create the final content that was going to go
on the screens way advanced.
So here's a flowchart that kind of
describes the pipeline from the art department's point of view.
And I'm sure each department is going to have
their own view of this process.
But this is the art department's.
You've got the Lucasfilm art department--
Doug Chiang's team up in San Francisco
working with the concept artists up there creating
amazing artwork.
And helpfully, also, they typically create 3D models
to do paint over.
So we're getting concept art and 3D models.
And the LA art department is taking
those and the script and the action
and starting to block out scenes.
And then typically, we'll just be bashing pieces
out in Maya and getting it into Unreal super quick,
just to start to look at what it's like.
And then we're also going to be sharing our environments
through Perforce with other departments--
tech-viz, pre-viz, visual effects.
And everybody is going to be analyzing those as we go along,
trying to figure out what we need to build,
how we're going to film it.
Visual effects is looking at what the requirements are going
to be and what they're going to need to do with the content
before it gets to the screen.
Perforce is great for that because we're
just all working collaboratively,
improving the set.
Every improvement gets propagated to everybody.
So we're very quickly working with the DP, lighting sets.
When that's going to pre-viz, they're
doing blocking on very high-quality scenes
and we're getting a lot more information.
And then when that's reached a certain level
and everybody's-- all the creatives have bought-off
on the look, that's going to allow them to prepare--
put through their pipeline to get it to be photo quality
content that will perform on the screen at frame rate
and all the bells and whistles.
So that's kind of the process there.
There's another flowchart I want to show that also kind of shows
the process broken down into stages--
design, concept.
But the interesting thing about this view of the pipeline
is that you see the virtual art department in the same zone
as construction department.
And that's essentially what it is.
That's the way I look at it.
The virtual art department is a construction department,
they're just using pixels.
You've got sculptors, painters, fabricators, props, costumes.
All of those trades that you have
in the traditional construction department,
you have in the virtual art department
and they're working concurrently.
So if somebody is doing a paint sample on a physical set
piece in the construction shop, they
may photograph that and give it to the virtual art department
to match, or vise versa.
If somebody working in Substance Painter
has got some great texture that's
looking great on this particular set, that gets rendered,
sent over to the paint department.
Likewise sculptors.
Sculptors are building in texture on a wall.
They're using ZBrush, whatever they're using.
That goes to the plaster.
And both of these tasks are happening simultaneously.
And both of these are also happening way
before we shoot it.
Set Dec.
Amanda Serino, set decorator, is creating a set decoration
that is going to go both in the physical world
and the virtual world.
If it's going to go in the virtual world,
she's got to create it early enough for it
to get scanned and transformed into a game asset,
put in the scene, and put there.
And a lot of times, you've got set pieces--
tables, dressing, whatever-- that
are going to appear in both.
So that's just a great way of looking at what the virtual art
department is.
Our virtual art department is made up of artists
from Happy Mushroom.
On season one we had some artists from Halon as well.
These are really good game artists.
And also mixed in people from visual effects
finals, which is really important.
Because we're not getting are our environments
to a certain level.
We have to go to photo level.
And even if the virtual art department
isn't creating that photo level environment,
we have to provide ILM with all of the elements
so that they can get there.
They can take it to the photographic level.
That's a big difference with this approach.
Pre-viz may get to a point where it looks really good.
But you can't to hand that over to Visual effects
and say in a few weeks you've got to turn that
into a photoreal asset.
We will take photogrammetry that is a very high quality.
Whether we resolve it to the highest level of detail or not,
we need to make sure that we've got that information that we
can give to ILM.
So if it's going to need to get to that level,
they've got the wherewithal to do that.
And the same with everything in the environment-- textures,
geometry.
It's all going to be heading towards
a photoreal final level.
One are the most interesting parts
of this was how we generated the content for these environments.
How we came up with the assets.
And I've got some slides here that demonstrate some of that.
This is a miniature that we made for a round meeting room
in one of the episodes.
To build this set physically would have
been tremendously expensive.
But we figured that we could do a third of it as a miniature.
Scan it in.
Make a game asset out of it.
Duplicate it around three times to make a full 360 degrees.
And then that became the basis for our environment.
And we've actually found that scaling up miniatures
works pretty well.
So this is the miniature and this
is the physical set that goes in the middle of it
where the performers need to interact
with the bar and the floor.
And when you bring them together,
you get a scene that is very compelling.
You can have interaction with smoke and light
from projectors.
But really, the whole thing came together very effectively
without us having to build a great deal.
This set-- this is a shot of the inside of Werner's office,
but on a day when we were doing a lighting test
with a sky in the background.
So these two don't go together, but it kind of
illustrates what the physical set actually is.
The physical set for that office was just some columns
and a floor and dressing.
And the interior of the room was actually
a scan of a warehouse in downtown LA
that we brought into Unreal and did some work on.
Did some painting and clean up.
We likewise brought in all of the set
dressing that would be in the virtual content
and put them together.
You can see the misalignment between the wall
and the ceiling here, which is because we're not viewing this
from the camera's point of view.
But here it is through the camera's point of view.
You can see it all marries up nicely.
We lit this a number of different ways
in Unreal with the DP for the different scenes
that play in here.
It was very effective because very little physical build.
We could build a set.
We put it in I think in a day and got it out overnight.
And then the next day we were ready to start another set.
So that's like the ideal scenario for the volume.
You don't want to be building a huge complicated set.
The point of this is you build the virtual set
and you've got it on a thumb drive
and you can bring it back up whenever you want.
You don't want to be doing a lot of physical building.
You want to build enough for the performers to interact with
and to create--
to receive the lighting from the virtual set.
But ideally, it's not a huge physical build.
Having said that, this one was a little bit more
work just because anything with all this much sand and geology
takes a little bit of time to get right.
But this is from a scan from Utah that we brought in.
Scanned it, processed it, brought it into Unreal,
decided where we wanted to shoot the action in VR scouting.
And then chopped a little section
of the physical rock out of the scan
and gave that to construction to build a physical rock that
matched it, which is what you see--
the physical rock on the right.
And then put them together.
And that's the shot from the final scene.
Which is, by the way, untouched.
It got a little bit of DI, I'm sure.
But that's what the camera saw on the day.
That's the cantina.
Now, Richard Bluff, the Visual Effects Supervisor from ILM,
went to him and said look, we can't afford to build a set.
We can build a bit of it.
Could we do this on the volume and have
the walls be on the content?
And he thought it was worth a try.
You know, it's a risky one.
But if it works, great.
If it didn't work, he was at least
going to have great lighting and a reflection
on all those reflective surfaces.
So he thought it was worth trying, and we did.
And it worked fine.
It was all in camera, the shots that we got.
This is the Roost, which was a very big environment.
And the complexity here was that the action was bigger
than the volume.
In other words, we had action here, and then off
in the distance there.
So we had multiple places that we had to move the volume to
within the virtual environment to capture the action.
You see this sort of ridge thing in the background
there, that yellow structure, that's in the content.
But the action took us down there.
And so when we got there, we had to have
a physical version of it.
And that's what this is.
Actually, we did not have the virtual content ahead of time.
We built this just a couple of weeks before we shot it.
So we had to get renders of the digital asset
and match that as closely as we could
in the physical, which was-- we've
got that process down now.
Here is it all come together.
And of course, you can see the misalignment
between the wall of the ceiling and the set,
which is because we're not looking at it from the camera's
point of view here.
So this process has been really intriguing we've
enjoyed working with Epic in the virtual art department
all through season one-- tremendous, generous
collaborators helping us figure out this process.
And we've kind of got to a place where
we are more confident about the sort of sets
that we can tackle with this, and we know the ones
that are more challenging.
But it's been an intriguing journey
and we're looking forward to continuing exploration
into what the possibilities of this process are.
So, anyway thank you very much.