hello, my name is David Goedicke and today I will present the paper XR-OOM
mixed reality driving simulation with real cars for research and design
the work was jointly done with Cornell tech and supported by the Toyota Research Institute
and done by myself Alexandra Bremers, Sam Lee, Frank Bu, Hiroshi Yasuda, and Wendy Ju
this work mostly tries to address some limitations of lab-based studies like the limited sense of
motion that often lab-based simulators have or the limited screen space that they have available
and also offer new avenues for research and design development. In particular
designing interfaces, new car interiors as a participant is driving in a vehicle,
autonomous vehicle behavior designs if for example, the participant is driving
as a passenger in a fake autonomous vehicle, and then experiencing various scenarios and leading
to the next point, unusual traffic scenarios give this message really gives us a way of
looking at these things from a much more sort of broader perspective and unbound by real-world
scenarios that frequently occur. Even though we're still driving in a real vehicle. We are
not the first ones to do this kind of work Volvo in 2020 published a video and a workshop paper
on using such a system where they demoed how and car interiors could be changed however for this
method really to become fully useful and gain a similar status as established in lab methods there
are a couple of additional aspects that we need to evaluate in particular safety how do you operate a
simulator safely and make sure that nothing happens where the participant will have you
know to drive away or just trying to drive while the headset shuts off. um Other aspects are but
there's been comfort after all the participants seeing the world through a mixability headset as
they're driving or as they're moving around that could be very naughty inducing so that's very
important and finally we want to understand what it actually means to measure driving performance
in this way. What are aspects that are similar and what are maybe kinds of conclusions or results
that we shouldn't draw from this kind of method? A quick technical overview we had we used the
Varjo XR-1 as our mixed reality headset we had a SmartTrack3 tracking this XR-1 that was in the car
as the participant was moving their heads and ZED2 tracking the vehicle itself. We
had a secondary system tracking the car motion as well which we mostly use as a way to compare
the performance of our virtual reality system. Before we get to driving tasks one other set of
tasks that we also evaluated were cockpit tasks they're almost equally as important
as they enable participants a good view of the surrounding and so on and so we wanted to make
sure to also include these steps in it. Many of these points were not a problem at all however
some taller participants had problems with bumping into the car as they were wearing the headset as
it is still relatively bulky. One point almost every participant had a difficult time with
was the very last one where they were where they were asked to verbally explain the indicator
lights that are visible and as we can see in the next clip this turned out to be a problem for
many participants due to high-dynamic-range issues of the camera so how it was trying to
auto expose as well as aliasing with the refresh rates between the display and the headsets cameras
See that I've engaged it and I've disengaged it verbally explain which indicator lights
are visible. indicator lighsts. I assume like up here I can faintly see zero miles per hour
I can see I can make out it to be fuel time is 228 fuel is full, I cannot read what's on the right
there, I cannot breed the small engine light or any of that on to the left of the miles per hour.
The second task then of our experiment was to evaluate the driving performance of participants
and in order to remain in a safe environment, we made sure that these were all tasks done at a low
speed on a parking lot separated from the from regular traffic when during in this parking lot
we laid out a couple of traffic cones as laid out on the pattern and then had the participants drive
through them in these four different steps. Sometimes these cones were physical calls and
sometimes those cones were virtual calls so we could compare between the different conditions.
In the following video we will show has been driving through one of these environments
very important to point out here is that with the camera setup as described in our paper and
visible here we can actually capture the entire environment in the top left corner for example you
can see the outside world as the parking lot where we were driving in, on the top right side from the
same 360 camera you can see the participant and how they are acting in the vehicle or operating
the vehicle, on the bottom left side you can see the participant's point of view exactly the
things that you were rendering to the headset, and then on the bottom right we have the output
from the tracking algorithm showing us where the car is moving relative to the virtual world
All right I've neat to back up a little.
In general, we could show that driving is actually not a problem and works
in these kinds of setups. uh however cockpit tasks were in particular some
were more difficult and could use some technical improvements additionally reversing and having
mirrors that also work with virtual objects were very challenging originally we wanted to have
the participants parallel park which we had to exclude due to the limitation of not being able to
see, reverse or back driving cameras uh the localization could be improved to allow for
higher speeds uh driving performance measures that were recorded could also be much more
analyzed like steering wheel angles and thermal input from the CAN-bus of the vehicle and finally
there are a few headset limitations like the weight of the headset and it's a relatively slow
auto exposure adjustment could be improved so that like the high dynamic range between switching
looking at the speed and looking at the real world could be improved, so that
once you look from the internal dashboard outside it's not all just white and blown out.
Finally I wanted to thank everyone I want to thank you for watching and listening I wanted
to thank the team for working on this project and the other members of the lab who also helped
if you want to get in contact feel free to send me an
email or try to contact me on GitHub and again thanks goes to Cornell Tech and of course the
Toyota Research Institute for supporting and funding this work Thank you so much.