♪♪♪
Voice Over: Sony believes the technology to instantly display high-definition,
photo-realistic computer graphics in real time will give creators more freedom than ever before.
Sony Innovation Studios was one of the first to use a game engine and LED wall for virtual production.
At CES 2020, a prototype was unveiled.
The LED display showed a photo-realistic 3D computer graphics set,
and the way the background synched with the camera movement suggested to many creators
the possibility of new shooting techniques.
In-camera visual effects use a game engine to display 3D computer graphics on a large LED wall.
This technique enables users to create a seamless integration
between foreground actors and virtual backgrounds.
The background image changes in real time according to the camera's positional information,
making it possible to shoot a 3D CG space as if shooting on an existing location.
Sony proposes the combination of its "Crystal LED" technology
which realistically projects space with rich color expression,
and its "VENICE" digital cinema camera, which can capture all light present,
as the ultimate virtual production shooting environment.
Ancient ruins, deserts, any location in the world and
and even fictional worlds can be filmed freely in the virtual production studio as if filming on site.
Creating virtual assets in high-definition and projecting in real time
greatly expands the possibilities for a studio shoot.
In October 2022, Sony Innovation Studios installed the world's largest stage for virtual production
using its “Crystal LED” technology.
We will expand virtual production workflows in various forms of entertainment
and create works that seamlessly integrate the real world and virtual spaces.
♪♪♪
♪ “Stardom” by King Gnu ♪
Shu Sasaki: Taking about the Music Video, this song, "Stardom",
has a huge range of sound image.
So I thought using existing locations in Japan
would never make it realistic.
But we still wanted to create something seamless for
the audience by combining computer graphics and live actions.
OSRIN: I think it all started with us talking about
whether we could create a visual that can convey the speed.
Which lead us to use a car as the main motif.
However, this will make filming really complicated because
we have to consider the actor's driving skills and
also have to avoid accidents.
Considering those factors,
we decided to construct a computer graphic world that
matches both the car we're going to use as well as our band image.
Shu Sasaki: What I expected from CG was
to be able to create a large field that is unthinkable in reality,
and to put band members in a space,
which is impossible in reality.
OSRIN: It showed us something between realistic and un-realistic.
We thought we were going to create
a very realistic tunnel for the scene,
but something more was added to it and
that was really exciting.
Mach was the main concept,
and we wanted to create something speedy
and everything fell in the right place.
What I had the most fun when we were shooting was,
when everything came together, for example the acting,
the art design and even each actor's personality,
when all those came together.
When what we shoot turns out few times
or even ten times better than what I had in my storyboard,
I almost cried.
Shu Sasaki: I had so much fun during the production.
Especially during Virtual Production shooting, me and my band members were able to see
what get captured on camera and talk things through while shooting.
So that made it feel like a regular live action shoot and was very fun.
OSRIN: I want to make sure the camera gets what's happening in front of me,
but I also want to make sure I see it with my own eyes.
Maybe I'm being greedy, but I want to take time to register
the scenery in front of me as a special view of that day
even when we're in production.
♪♪♪
Hidenori Ushiro: When we as SONY creates the background assets,
the most important thing to pay attention to is
definitely the absolute reality.
If we have a city as the background asset,
in this case it's Shibuya,
the ultimate purpose for that asset is to make it possible to shoot in that environment
without physically going to that location.
Satoru Higa: We created Shibuya Spain-zaka based on 3D scan data by FARO
and tons of pictures of a location.
We combined those huge photo data with the 3D scan data
to create a mesh model with realistic texture by using Photogrammetry technology.
What's important here is how it looks when we film what is projected on the LED wall,
and when I saw the result, it really looked like Shibuya Spain-zaka and I was stunned.
Tokihiko Tsukamoto: Real-time rendering takes absolutely shorter time.
It takes less time to render even a photo-real space and is able to output in real-time.
Satoru Higa: I'll get feedback for what I just did immediately, and that allows me for more try and error opportunity.
I believe that gives a creator much more freedom.
Tokihiko Tsukamoto: When we're working on the data in Virtual Production studio,
we could be like "hey let's fix this part a bit" and
we were able to reflect that change immediately.
It was not possible with pre-rendering.
Hidenori Ushiro: The huge advantage of Real-time rendering for creators is
that they do not have to wait for hours or days to see the composited image in real time on the set.
Takeshi Yoong: Whether we wanted to fix camera or lighting,
both elements would affect each other, rather than simply changing or adding one thing.
For example, if we want to have lighting in certain way,
we'd also have to change things in Virtual production asset data.
When dealing those issues, the lighting and CG specialist were able to talk things through
and make decisions on the go.
I think that kind of live interactions can only happen in Virtual Production.
When I was able to witness them creating the backdrop within 3 or 4 days,
I was shaking from excitement. It is so interesting.
And there is so much freedom.
Satoru Higa: You could try an idea that happen to come to your mind
or just try something that you usually wouldn’t even bother doing.
And that expands the possibility of creation or expression in so many degrees.
♪ “Clockwise” by Newspeak ♪
♪ electronic sound ♪