Hello.
My name is Gerlinde Emsenhuber
and I'm going to present the paper
eye-perspective view management for Optical See-through
head mounted displays on behalf of all authors.
View management for optical see through HMDs
requires knowledge of how the scene looks to the user
when viewed through the HMD.
This is necessary to avoid perceptual issues
such as poor legibility and decreased contrast.
For example,
when placing annotations in this scene,
one would look for areas
that are uniformly colored, dark and untextured.
Placing annotations in these areas ensures
that they stand out against the background and are legible.
However, the user's true view is unavailable.
Because of this, previous work has often
relied on the built-in HMD camera when making placement decisions.
For example, consider the Hololens 2.
There are noticeable offsets between the built-in camera,
the user's left eye and the user's right eye,
and as a result,
relying on the HMD camera leads
to incorrect assumptions about the user's true view.
Returning to our initial scenario,
the view that is shown here corresponds to the built-in camera view.
If we make layout decisions based on this view,
mid-air
labels will still suffer from poor legibility and contrast
when seen by the user.
This happens due to the parallax
between the HMD view and the user's eyes,
which causes a change in backgrounds behind the labels.
To avoid such issues,
eye-perspective rendering aims to reconstruct
the user's true left and right views.
The reconstructions can then be used as basis for placement decisions
which are consistent with what
the user really sees, avoiding the aforementioned perceptual issues
and leading to high quality
layouts.
Eye-perspective rendering
also opens up the possibility of stereo-based optimizations
since now two views are available
instead of just a single HMD camera view.
This can further improve legibility.
To explore the use of eye-perspective
rendering in the context of view management,
we developed a novel view management algorithm
that starts by reconstructing the user's left and right views
and then optimizes layouts in the reconstructions.
Labels move in a temporally coherent fashion,
but our algorithm aims to keep them in
front of a suitable background at all times.
We compute the left and right reconstructions via reproduction.
This method is fast and robust
and can immediately react to changes in the scene,
but it does produce disocclusion artifacts,
especially when the user is close to the scene.
In our implementation,
we avoided placing labels directly on top of disoccluded pixels.
In order to find good positions for label placement,
we analyzed the reconstructions
with respect to four different criteria.
Based on this analysis,
we classify pixels as either good or bad label placement candidates
and use vector fields to pull labels towards the good candidate pixels.
We use the following criteria.
Uniformity,
which stops labels
from overlapping several regions with different colors.
Lightness, which excludes areas that are too bright.
Texturedness, which excludes areas with many small details.
And finally,
stereo uniformity,
which makes sure that labels are placed in areas
with a similar color in both eyes.
The use of stereo uniformity as optimization criterion is only possible
if both the left and the right views are available.
We performed a study related to stereoscopic text annotations
that shows that placement in stereo
uniform regions does improve legibility.
To evaluate the effectiveness
of our eye-perspective view management
algorithm, we performed a within-subjects study
where participants wore a Project North Star HMD
with a Stereo ZED mini mounted on top of it.
The Stereo ZED mini was used for positional tracking
as well as for obtaining the RGB and depth data necessary
for reconstructing the user's eye perspective.
The participants’ task was to rate label layouts,
which were optimized with different view management algorithms.
The order of algorithms and layouts was balanced
using an orthogonal Latin squares table.
The three view management algorithms used in
the study were a baseline algorithm
which performs no background based optimization at all.
As a result,
the labels are always positioned slightly above their anchor points.
An HMD camera based algorithm,
which does optimize the labels but based on a built-in camera view.
Since there is only a single view available,
stereo vision uniformity cannot be evaluated
and only the other three criteria are used.
And finally, our eye-perspective view management algorithm,
which uses the reconstructed, left and right views and applies
all four criteria to optimize labels.
As for the results,
participants were asked to rate the legibility
of the optimized labels,
as well as the severity of background issues,
such as blurring and double vision.
The eye-perspective algorithm significantly outperformed
the other two algorithms with respect to both.
Participants were also asked to rate the labeled backgrounds
with respect to our optimization criteria.
This rating was performed
by estimating the percentage of the labels
background that fulfilled each criterion.
The eye-perspective algorithm again
significantly outperformed the other two algorithms.
Finally, we also asked participants to rank
the three algorithms in terms of preference
after they had finished all evaluations for a layout.
The eye-perspective algorithm was ranked first in 97% of all cases.
So to sum up our findings.
We find that the use of eye-perspective rendering does improve layouts,
as demonstrated by the fact
that our eye-perspective view management algorithm
significantly outperformed the other algorithms
with respect to legibility
and was also ranked as the preferred choice in nearly all cases.
Also, HMD camera based optimization can still be beneficial.
Although a single RGB
camera cannot be used to perform stereo based optimizations,
the HMD camera based algorithm was still ranked
as second best in two thirds of all cases.
Layouts produced by this algorithm were ranked
as significantly better than the baseline
with respect to uniformity and texturedness
Finally, the study showed that eye-perspective rendering
can replicate the user's views with sufficient precision,
even with unconstrained head movements.
In our study,
we allowed participants to move their heads freely,
unlike previous similar studies, where headrests were used.
Our reconstruction was precise
enough to be used as input for the optimizer,
and our algorithm produced good layouts
with respect to our optimization criteria.
However, we also identified some areas that could be improved upon.
First, regarding disocclusion artifacts
and the choice of reconstruction method.
We decided to use reprojection
due to its low computational costs and its robustness.
However, disocclusion artifacts need to be handled,
especially since disoccluded areas tend to become larger
when the user moves closer to the scene.
This might require the use of advanced inpainting methods
or a change to a different reconstruction technique
altogether, such as image-based rendering.
Second, regarding alternative types of optimizations.
In cases where reliable scene information is not available,
or if there is an insufficient number of good placement locations,
the view management algorithm could fall back
to alternative forms of optimizations,
such as using uniformly colored billboards as label backgrounds.
In cases where the scene is changing quickly,
sticking the labels
to real world geometry instead of placing them
mid-air, can improve temporal coherence.
And finally, regarding the placement optimization itself.
In the study, we forced labels into locations
where all criteria were fulfilled.
Such a strict interpretation of the optimization criteria
might lead to situations where the labels are placed
at large distance from the origin
or when those suitable can be found at all.
To avoid this,
the criteria could be integrated into a loss function
for a more lenient optimization.
In conclusion,
we have shown that eye-perspective view management
leads to improved label placements,
that it can be implemented on currently available HMDs
and that it works even with unconstrained head movements.
This marks the end of my presentation.
Thank you.