As a car travels along a narrow city street, reflections off the glossy paint or side mirrors
of parked vehicles can help the driver glimpse things that would otherwise be hidden from
view, like a child playing on the sidewalk behind the parked cars.
Drawing on this idea, researchers from MIT and Rice University have created a computer
vision technique that leverages reflections to image the world.
Their method uses reflections to turn glossy objects into “cameras,” enabling a user
to see the world as if they were looking through the “lenses” of everyday objects like
a ceramic coffee mug or a metallic paper weight.
Using images of an object taken from different angles, the technique converts the surface
of that object into a virtual sensor which captures reflections.
The AI system maps these reflections in a way that enables it to estimate depth in the
scene and capture novel views that would only be visible from the object’s perspective.
One could use this technique to see around corners or beyond objects that block the observer’s
view.
This method could be especially useful in autonomous vehicles.
For instance, it could enable a self-driving car to use reflections from objects it passes,
like lamp posts or buildings, to see around a parked truck.
The heroes in crime television shows often “zoom and enhance” surveillance footage
to capture reflections — perhaps those caught in a suspect’s sunglasses — that help
them solve a crime.
In real life, exploiting these reflections is not as easy as just pushing an enhance
button.
Getting useful information out of these reflections is pretty hard because reflections give us
a distorted view of the world.
This distortion depends on the shape of the object and the world that object is reflecting,
both of which researchers may have incomplete information about.
In addition, the glossy object may have its own color and texture that mixes with reflections.
Plus, reflections are two-dimensional projections of a three-dimensional world, which makes
it hard to judge depth in reflected scenes.
The researchers found a way to overcome these challenges.
Their technique, known as ORCa (which stands for Objects as Radiance-Field Cameras), works
in three steps.
First, they take pictures of an object from many vantage points, capturing multiple reflections
on the glossy object.
Then, for each image from the real camera, ORCa uses machine learning to convert the
surface of the object into a virtual sensor that captures light and reflections that strike
each virtual pixel on the object’s surface.
Finally, the system uses virtual pixels on the object’s surface to model the 3D environment
from the point of view of the object.
Imaging the object from many angles enables ORCa to capture multiview reflections, which
the system uses to estimate depth between the glossy object and other objects in the
scene, in addition to estimating the shape of the glossy object.
ORCa models the scene as a 5D radiance field, which captures additional information about
the intensity and direction of light rays that emanate from and strike each point in
the scene.
The additional information contained in this 5D radiance field also helps ORCa accurately
estimate depth.
And because the scene is represented as a 5D radiance field, rather than a 2D image,
the user can see hidden features that would otherwise be blocked by corners or obstructions.
In fact, once ORCa has captured this 5D radiance field, the user can put a virtual camera anywhere
in the scene and synthesize what that camera would see.
The user could also insert virtual objects into the environment or change the appearance
of an object, such as from ceramic to metallic.
The researchers evaluated their technique by comparing it with other methods that model
reflections, which is a slightly different task than ORCa performs.
Their method performed well at separating out the true color of an object from the reflections,
and it outperformed the baselines by extracting more accurate object geometry and textures.
They compared the system’s depth estimations with simulated ground truth data on the actual
distance between objects in the scene and found ORCa’s predictions to be reliable.
Consistently, with ORCa, it not only estimates the environment accurately as a 5D image,
but to achieve that, in the intermediate steps, it also does a good job estimating the shape
of the object and separating the reflections from the object texture.
Building off of this proof-of-concept, the researchers want to apply this technique to
drone imaging.
ORCa could use faint reflections from objects a drone flies over to reconstruct a scene
from the ground.
They also want to enhance ORCa so it can utilize other cues, such as shadows, to reconstruct
hidden information, or combine reflections from two objects to image new parts of a scene.