Hello everyone, my name is Martin Misiak and on the behalf of my collegues Arnulph Fuhrmann
and Marc Erich Latoschik, I will be presenting our work on Impostor-based Rendering acceleration
for XR.
A central characteristic of XR’s immersive qualities is a believable depth perception.
The necessary depth cues, however can become very expensive to generate, as
new XR hardware constantly strives towards higher resolutions and refresh rates.
To achieve peak performance levels across a large variety of hardware, virtual objects
need to be reduced in their geometric and shading complexities.
A common approach for optimizing rendering performance is to employ various mesh-simplification
techniques.
One extreme form of simplification are impostors.
In its most common form, an impostor replaces the original 3D object with a rendered image
of itself.
Due to its simple geometry and pre-calculated radiance values, an impostor can be rendered
much faster than the original object.
Although impostors are only valid for a small viewing region, they are still a popular choice
in non-XR applications to replace distant or contextual objects.
The impostor illusion however does not work in XR anymore, as the underlying geometry
fails to reproduce the correct binocular disparity cues.
An impostor is simply perceived as a flat image, even when viewed from the correct viewpoint.
Motivated by the great speed-up of this technique, our work aims to adress this issue, as we
propose an impostor based rendering system for modern XR devices.
Over the years a number of different impostor techniques have been explored.
The majority relies on a pre-computed representation, which aims at completely replacing the original
object.
Our approach on the other hand dynamically creates impostors at run-time.
This allows us to better handle dynamic aspects of the scene such as changes in lighting or
geometry.
Most similar to our work is the technique of Schaufler.
He also presented a rendering system which relies heavily on dynamically generated impostors.
We follow the same core principle of reusing previously rendered images of objects, but
we extend this idea to be applicable to modern XR applications.
To create a three-dimensional perception, an impostor has to replicate the binocular
disparity between the left and right images of the substituted object.
We achieve this by displaying a separate impostor for each eye.
We call this image-pair a stereoscopic impostor.
Viewing such an impostor-pair from the same position as it was created, results in a correct
three-dimensional perception of the original object .
However, as the user moves away from the capture viewpoint, the perceived shape of the object
becomes increasingly distorted.
To solve this issue, the correct parallax movements of the depicted surface have to
be considered.
We achieve this by storing additional depth values for our impostors and performing a
raymarched parallax correction step - which looks like the following:
When rendering an impostor from a novel view, we first intersect the view ray with the object’s
bounding box.
The two intersection points are then transformed into the screen-space of the capture viewpoint.
Here, the two transformed points define the search interval for the ray marching step,
which is then performed to find the surface point visible from the novel view.
This correction step is separately performed for both stereoscopic impostors.
The search interval length depends on the distance between the novel view and the capture
position.
For small viewpoint changes, such as typical head sway movements or looking around, the
raymarching costs are kept minimal.
We experimented with different sampling strategies to increase the temporal stability of our
impostors.
While single sample raymarching is the fastest rendering method, it is also prone to silhouette
and inner surface flickering.
Sending out multiple raymarches and averaging their radiance values creates excellent results,
however, it becomes prohibitively expensive for a larger number of impostors.
So instead, we chose to perform only one raymarch, with a subsequent multi-sampling of the hit
point area.
To guarantee a hit point in the vicinity of silhouette edges, we dilate the depth values
of our impostors by a few pixels outward.
The results of this method are comparable to multiple raymarches, however, at a fraction
of the cost.
Our stereoscopic impostors are valid for the typical range of head movements. However,
as users walk around and explore the environment, our impostors can become increasingly invalid.
The two main factors are the revelation of disoccluded areas
and a resolution mismatch between an impostor and the real object.
The regeneration process consists of rendering the underlying object from a novel viewpoint
and storing it inside our main impostor atlas.
The first step is to determine which impostors are most in need of regeneration – This
happens based on our regeneration metric.
An impostor begins its regeneration process by computing its next view and projection
capture matrices.
The novel viewpoint for which the rendering will occur is derived from the current head
position and is then extrapolated a few frames into the future.
To create impostors of more complex objects without exceeding the frame budget, the rendering
process can be spread over multiple frames.
Each frame renders then a sub-set of the object´s triangles into a second atlas texture - The
accumulation atlas.
After the last triangle has been regenerated, the result is copied over to the main atlas,
from where it can be used during regular rendering.
As impostors are replaced by new ones, the resulting image can suffer from sudden popping
artifacts, hurting the overall temporal stability.
For this reason, we are performing a blending operation between the old and the new impostor
image.
As a new impostor is finished regenerating, the current impostor is copied over to a third
atlas - The history atlas – Where it can be used for
Interpolation towards the new impostor image.
As already mentioned, our main threats to impostor validity are disocclusions and resolution
mismatch. They also form the basis for our regeneration
metric.
We use a conservative parallax measure by Shade et al. for our disocclusion term.
The texture term is binary and is set to 1 if a better fitting tile is present in the
impostor atlas or 0 otherwise.
To better determine the regeneration priority when considering multiple impostors, we added
a distance and field of view weighting term.
Lets move on to the evaluation.
We implemented a VR version of our rendering technique, and evaluated it using a „Computer
Graphics Museum“ test scene.
This scene contains a total of 324 3D models with varying geometric and material properties.
The models have roughly 83 mil. vertices, and are spatially grouped together into 8
different stations.
We captured a stereoscopic image sequence of approx. 9000 frames, as seen here in the
video, and used it in our performance and image quality comparisons.
In the following evaluations, we will compare our impostor based rendering system against
a classic mesh based rendering approach.
Our rendering system replaces each of the 324 models in the scene with a stereoscopic
impostor and updates it as necessary.
In terms of performance, we can see that our approach is able to achieve a framerate of
over 90Hz for this complex scene, which outperforms the naive mesh rendering condition by a factor
of 3-6.
If we include Levels-of-Detail, the performance of the mesh rendering condition improves significantly.
However our technique is still faster by a factor of 2-3.
The only time where mesh rendering with LOD slightly outperformed our impostors was at
station 4, where a large number of very simple objects is rendered.
For image quality we compared our technique to mesh rendering using the SSIM and LPIPS
metrics.
Throughout the sequence, our impostors are very close to the reference as we achieve
SSIM scores between 0.98 – 0.99.
The lowest scores are achieved at stations 5 & 6, which showcase objects with highly
specular reflections and objects with a high disocclusion potential.
To evaluate the temporal stability of our approach we captured a sequence of 190 frames
exhibiting typical head sway movements.
We used the tPSNR-metric, which computes the Peek-Signal-To-Noise-Ratio between the difference
images of two subsequent frames.
Our impostor technique achieves a higher tPSNR score than a mesh rendering method with 4x
MSAA.
The improved temporal stability can also be seen in the two accompanying videos.
This concludes our presentation of the proposed impostor-based rendering system.
The system allows scenes with complex objects to be rendered at high framerates on modern
XR headsets.
In terms of limitations, our system is not designed to provide speed-ups for continuously
changing objects.
As constant regenerations would negate the performance benefit of the impostor representation.
In future work, we would like to perform a user study to validate the perceptual quality
of our solution.
We would also like to explore the use of raytracing for disocclusion mitigation, and even a selective
impostor regeneration on a per-pixel basis.
Finally, we would like to implement our technique into a popular game engine, as we believe
it could be a useful tool in the hands of other researchers and developers alike.
Thank you for your attention!