- Good morning everyone, today I'm going to talk about
how to use dynamic stereoscopic 3D parameter adjustment
for enhanced depth discriminations.
And I'm talking on behalf of both authors,
myself and doctor Joseph LaViola.
Generation of stereoscopic 3D requires two images,
one for the left eye and one for the right eye,
and to divert these images,
convergence and separation are really important parameters.
Separation is defined as the distance between the left
and right eye for two cameras,
and convergence is the distance where left eye and right eye
camera for instance intersect.
Usually, we see that in applications, these parameters
are kept fixed, for comfortable viewing experience
and it is optimizes that you see comfortable 3D
throughout the movie or whatever game you are playing.
But fixed parameter approach is effective for
depth perception of objects only within 25 meters,
and it works best for objects within 10 meters only.
But what happens when the distance is much larger
or if there is a large depth range in the scene,
so if you have a very close object in front of you
and other objects are too far away.
In those situations, this fixed parameter approach
doesn't work.
In real life, we continuously adjust the separation
between our eyeballs and we converge at different
distances, and this is the main motivation behind this work,
because the similar approach could work
for 3D applications as well.
So if you have applications where
the distances are changing, so if you have motion from
inside scene to an outdoor scene,
or if you have closer objects in situations like
if you have a first-person shooter game,
there is a closer object in front of you, a gun,
and other objects are far away, so you have to,
in case of fixed parameter scenario,
you have to optimize parameters for the gun,
otherwise it's going to look uncomfortable to look at,
which reduces depth perception in these scenarios.
A similar approach is for games where,
like a flying simulation games where you have a cockpit
in front of you, and other objects are too far away.
So in those situations, you are still needing depth.
So the goal of this research is to explore if dynamic
stereo parameters would be useful.
And there are three main requirements for this.
It has to be real time, the 3D viewing should be
comfortable, and as you are changing the parameters,
the change should not be perceived by the user.
There have been several approaches to reduce visual fatigue
in the past, one of them is to use a smaller separation
value, but it reduces depth perception.
The second one is to use a non-linear warping function
on the disparity map, but this approach requires
offline processing and we cannot use it for 3D applications,
also it's suitable for 3D movies.
Dynamic stereo parameters were first explored by Ware
in 1995, and they explored only the separation parameters,
and they did user testing in 1998,
but they did not include any depth judgment tasks,
and their approach doesn't work when
there are objects much closer to the camera.
There have been other work done regarding this area,
but no...
And in the past, the evaluation was limited
to more or less static scenes.
And in our case we have explored and doubled up
two dynamic stereo algorithms,
one is dynamic stereo which doesn't use eye tracking
and one which uses eye tracking.
And we have chosen several depth judgment tasks
to explore if these algorithms work.
So we have our algorithms analyze the scene and choose
parameters based on three aspects of the scene.
One is the scene displayed, display size and aspect ratio,
and inter-pupilarity distance of the actual user
using it, so we actually measure the IPD
of every user in the experiment.
So in our case, the maximum separation was defined as
the ratio of IPD to display width multiplied by
a scaling factor to transform from virtual space
to 3D space, actual real space.
And this is the formula we got from Nvidia's recommendation
for best 3D viewing experience.
In our algorithms we set the separation value
as the percentage of this maximum value.
So our first algorithm is dynamic stereo
without eye tracking.
It chooses ideal parameters based on the distance
of the closer object to the camera,
so we divide the scene into different zones
and we calculate the distance value from the camera
at the center of each zone and then choose
the minimum value to find the closest object.
This is a table we calculated based on several user studies,
and we found what value of separation and convergence
works for different distances
and how fast we should change it.
And in this case you can notice that when there are
closer objects, we should change the parameters
really fast, because if you have too much disparity
for closer object it makes it difficult to look at,
so we have to adjust to that really fast.
In case of eye-tracked dynamic stereo,
we optimize parameters based on where the user is looking at
and the distance of the object where the user is looking.
And we obtain the gaze data from an eye tracker,
and we choose the parameters from
the table I showed you last.
In both algorithms, the parameters should change gradually
at a certain way so that user is not perceiving these
changes as he is looking at the scene.
When the objects are much closer, in case of an eye-tracked
dynamic stereo, so if you have a much closer object,
and the user is looking at much farther distances,
then the disparity is much larger for the closer object,
so in that case, we make the closer object
partially transparent so that it is not
so uncomfortable to look at.
And there is no transparency when the user
again start looking at the closer object.
Something like this, when you look at the object,
it's visible, although the colors in this
projected over here so you cannot see
the two transparent objects here,
but I can see it on the laptop, so it...
(laughter)
It looks like they disappear.
So we did an user evaluation for both these algorithms
and compared it with a fixed parameter approach,
so in our test we had two independent variables,
stereo mode, where we have three guesses,
static, dynamic, and dynamic with eye tracking.
And object mode, who checked if these algorithms work
when there is a closer object or
if there is no closer object.
And the dependent variables include response time
and the task performance data for each of the tasks,
which I'm going to discuss.
The first task was depth ranking task in this case,
the user was supposed to identify what is the depth level
of this red object displayed in the center
based on the scale displayed on both the sides.
So they were just supposed to pick
one value based on the scale on the side,
and we recorded response time and error magnitude
for this object.
Second task was relative depth judgment task.
In this case there were two spheres and they were supposed
to judge which of them is closer to them.
And in this case we recorded accuracy and response time.
And our third task was path tracing task,
and in this case, two objects were highlighted
and they were supposed to answer yes or no
based on if there is a path of length two
between the two highlighted nodes.
And in this case also we recorded
response time and accuracy.
So we have five hypotheses, the first one was
both dynamic stereo algorithms will improve depth
discrimination ability, they will reduce the response time,
and eye-tracked and dynamic without eye tracking
will have similar performances when
there is no closer object.
But eye-tracked will perform better than dynamic
when there is a closer object.
And users will prefer these algorithms compared to
its experimenter approach.
So we had 36 participants, our setup includes
a BenQ 3D monitor, Tobii EyeX eye tracker,
Nvidia wired 3D glasses, we had to choose wired because
the wireless one was interfering with
the IR signal from the eye tracker.
We used a chinrest to fix the distance between
the display and the user, and a PC with GTX 780 graphics
card and Unity gaming engine.
And these are the results we have.
For depth ranking task, we found that the error magnitude
was much higher for a fixed parameter approach,
and the lowest one was for eye-tracked dynamic stereo.
But we did not notice any difference in the response time.
When we looked at based on the object mode,
we found that static stereo and dynamic stereo
perform similarly when there was a closer object,
but when there was no closer object, then dynamic stereo
and dynamic stereo with eye tracking
both had similar performances,
and eye-tracked dynamic stereo's performance didn't change
much whether there was a closer object or not.
There was no difference in the response time in this case.
In case of relative depth judgment task,
we had a similar performance, like the last case.
In this case, we were observing accuracy rather than error,
so in this case we found that the accuracy was much lower
for static stereo and much higher
for eye-tracked dynamic stereo.
There was no difference in response time.
When we divided it based on the closer object,
and still we had similar statistics, like static stereo
and dynamic stereo performed similarly when there was
a closer object, and eye-tracked dynamic stereo and
dynamic stereo were similar when there was no closer object.
There was no significant difference in response time.
In case of path tracing task, we found that the accuracy
was higher for eye-tracked dynamic stereo and less
for static stereo, but in this case, we found that
the response times were lower for dynamic stereo
and eye-tracked dynamic stereo compared to static stereo.
And when we divided it based on if there was a closer object
or not, the statistics were really
similar to the last experiments.
There was similarity between static and dynamic when
there was a closer object, and dynamic stereo and
eye-tracked dynamic stereo performed similarly
when there was no closer object.
And the response time was less when there was no closer
object and when there was a closer object,
then all the three performed similarly
in terms of response time.
So for our first hypothesis, we saw that there was
an improvement in the depth discrimination ability
and that allowed us to choose the right parameters
for different depths, so we were able to accept
this hypothesis.
In case of our second hypothesis, we found that in case of
depth ranking task and relative depth judgment task,
there was no difference in response time, however in case of
path tracing task, the response time was reduced
with dynamic stereo adjustments,
so we could not accept this hypothesis.
In case of hypothesis three, when there is no closer object
we found that dynamic stereo and eye-tracked dynamic stereo
perform similarly, and better than static stereo,
so we were able to accept this hypothesis.
When there is a closer object, then eye-tracked dynamic
stereo performed better, so we were able to accept this
hypothesis.
And in case of participant preference, when there is no
closer object, then participants prefer dynamic and
eye-tracked dynamic over static.
When there is a closer object, they prefer eye-tracked
dynamic stereo over the other two algorithms.
So we noticed that in case of closer objects
when we use fixed parameter approach, it chooses
the lower separation value, which leads to smaller depth.
And eye-tracking improved depth discrimination ability
in these scenarios.
We used transparency effect to avoid discomfort for
closer objects, but it was distracting to users
in some cases, especially when the other objects were
much closer to the closer objects.
The other option is to explore depth of field effect
which could be used, and it might help.
So we noticed that these results are promising
to our future 3D applications.
One could use dynamic stereo when eye tracking
is not available, and it works best when
there is no closer object.
And we could use eye-tracked dynamic stereo when
there is an eye-tracker available and it could be useful
especially for first-person shooter games
when you have a gun always closer to you.
Our stereo parameter estimation was based on several
parameters which were determined experimentally
and they may not work for other setups, so one has to
calculate these parameters based on their setup.
And the geometry of the scene changes as these parameters
are changing, so it's not a good idea to use dynamic
stereo when you require perfect depth judgment,
so it's not going to work in that scenario,
like industrial design applications where you need
to estimate the actual depth.
So in conclusion, we found that dynamic stereo parameters
provided improved, enhanced depth discrimination
compared to static stereo.
Participants preferred dynamic over static,
and in case of no closer object, dynamic
and eye-tracked dynamic stereo performed similarly,
and in case there was a closer object, eye-tracked
dynamic stereo performed better than the other case.
And in the future, we would like to explore what happens
when the user is moving around in the environment,
because in our case the user was static
and the scene was changing,
and one could explore depth of field blurring
along with eye tracking.
We would like to acknowledge NSF, doctor Fiore,
our members of the lab and anonymous reviewers for this,
and I'm open to questions.
(applause)
- [Man] Thank you very much.
Alright, we have time for one question.
If you have a question, would you come to the front
and say your name and your affiliation.
- [Man] Hello, Jerry Kemp from Korea University.
Can you just comment a little bit on the sickness,
you mentioned a little bit about--
- I don't think this is on.
Let's have a quick look.
There you go.
- In terms of thickness level we found that
the experiment duration wasn't that long,
it was like an hour or so, so we did not,
although we measured it in our experiment,
and we asked participants to rate their sickness level,
we did not find any, probably might happen when they
use it for longer periods of time, like three hours,
four hours, which I think is typical for any
stereo application.
- [Jerry] So, no particular difference between the fixed
setup and the dynamic adjustment case?
- We did not find any significant difference,
although there was a little,
depending upon the participants.
Some participants are too sensitive to a stereo,
but there were only a few of them,
so there was no significant difference.