Even novice photographers and videographers who rely on their handheld devices to snap
photos or make videos often consider their subject's lighting.
Lighting is critical in filmmaking, gaming, and virtual/augmented reality environments
and can make or break the quality of a scene and the actors and performers in it.
Replicating realistic character lighting has remained a difficult challenge in computer
graphics and computer vision.
While significant progress has been made on volumetric capture systems, focusing on 3D
geometric reconstruction with high resolution textures, such as methods to achieve realistic
shapes and textures of the human face, much less work has been done to recover photometric
properties needed for relighting characters.
Results from such systems lack fine details and the subject's shading is prebaked into
the texture.
Computer scientists at Google are revolutionizing this area of volumetric capture technology
with a novel, comprehensive system that is able to capture full-body reflectance of 3D
human performances, and seamlessly blend them into the real world through AR or into digital
scenes in films, games, and more.
Google's this new system is called as "The Relightables"
There have been major advances in this realm of work that the industry calls 3D capture
systems.
Through these sophisticated systems, viewers have been able to experience digital characters
come to life on the big screen, for instance, in blockbusters such as Avatar and the Avengers
series and much more.
Indeed, the volumetric capture technology has reached a high level of quality, but many
of these reconstructions still lack true photorealism.
In particular, despite these systems using high-end studio setups with green screens,
they still struggle to capture high frequency details of humans and they only recover a
fixed illumination condition.
This makes these volumetric capture systems unsuitable for photorealistic rendering of
actors or performers in arbitrary scenes under different lighting conditions.
Google's Relightables system makes it possible to customize lighting on characters in real
time or re-light them in any given scene or environment.
They demonstrate this on subjects that are recorded inside a custom geodesic sphere outfitted
with 331 custom color LED lights (also called a Light Stage capture system), an array of
high-resolution cameras, and a set of custom high-resolution depth sensors.
The Relightables system captures about 65 GB per second of raw data from nearly 100
cameras and its computational framework enables processing the data effectively at this scale
Their system captures the reflectance information on a person--the way lighting interacts with
skin is a major factor in how realistic digital people appear.
Previous attempts used either flat lighting or required computer generated characters.
Not only are they able to capture reflectance information on a person, but they are able
to record while the person is moving freely within the volume.
As a result, they are able to relight their animation in arbitrary environments.
Historically, cameras record people from a single viewpoint and lighting condition.
This new system allows users to record someone then view them from any viewpoint and lighting
condition, removing the need for a green screen to create special effects and allowing for
more flexible lighting conditions.
The interactions of space, light, and shadow between a performer and their environment
play a critical role in creating a sense of presence.
Beyond just 'cutting-and-pasting' a 3D video capture, the system gives the ability to record
someone and then seamlessly place them into new environments--whether in their own space
for AR experiences--or in the world of a VR, film, or game experience.