Hi, I’m Jonathan Wieland.
Hi, and I’m Sebastian Hubenschmid, and we will present the paper "ReLive: Bridging In-Situ
and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies”.
ReLive is a joint effort between the University of Konstanz and the University of Maryland.
This work was done by me as well as Jonathan Wieland, Daniel Immanuel Fink, Andrea Batch,
Johannes Zagermann, Niklas Elmqvist, and Harald Reiterer.
So, conducting mixed reality studies often results in a lot of data: And while
there might be traditional measurements such as task completion times, there are also often
more complex metrics that for example characterize participants’ movements, their spatial arrangements,
or collaborative workflows.
This can lead to rather complex analysis procedures.
And in this regard, prior work has shown two different approaches:
So some works use a non-immersive approach and for example use a desktop for data analysis,
which often excels at an ex-situ analysis, as there is a rich ecosystem of well-integrated
and cross-compatible applications.
In contrast, recent works use an immersive approach, for example by visualizing data
with augmented or virtual reality.
This approach is often ideal for an in-situ analysis, as the immersion can be beneficial
for investigating the environmental context and also offer a better spatial understanding.
We argue that an analysis workflow for mixed reality user studies can benefit from both
approaches, depending on the current goal.
However, researchers have to reconstruct their workspace when switching between these two
approaches, preventing any kind of serendipitous findings that might occur if this transition
between these two approaches was seamless.
This makes the use of hybrid user interface or transitional interfaces rather important,
which can allow users to easily switch between these analysis approaches as needed.
So, to take full advantage of both an immersive and a non-immersive analysis approach, ReLive
combines an immersive VR view with a non-immersive desktop view.
Since we use these completely different devices with complementary characteristics, ReLive
represents an asynchronous hybrid user interface, where these devices are used in sequence,
rather than simultaneously.
And as a result, ReLive has a desktop interface suited for a holistic ex-situ analysis, where
users can investigate aggregated study data.
Alternatively, researchers can also put on a VR headset and relive an interactive spatial
recording of a study.
Both views are instantly synchronized, allowing researchers to switch between desktop and
VR as needed, allowing for a seamless analysis.
So let's first take a look at ReLive's desktop view.
Here, we combined concepts from a computational notebook with a video editor interface.
And similar to a computational notebook, we can create new configurable tools in a center
panel.
So for example, here we measure the distance between two handheld tablets that were used
during a study.
We can add different sessions from this particular study by dragging in more data from the left
panel to compare this metric across different participants.
So now, each line represents the distance over time between two tablets in one study
session.
Similar to a video editor, we can also look at media data and show video recordings from
the study or look into a 3D scene replication of the original study environment.
We can control the playback of this media data with a timeline interface in the bottom
panel.
Here, we can either play back the study as it originally happened, or scrub back and
forth for specific point in time.
Lastly, we can also explore the 3D scene replication by moving the camera around with our mouse
and keyboard.
Here, we can also use different perspectives, such as an isometric view from above or the
viewpoint of a specific device.
But of course, there are cases, in which viewing this non-stereoscopic 3D scene does not provide
the full picture.
And here our VR view comes into play.
So, we simply put on the headset and with that we switch to a fully immersive representation
of the original study scene.
This environment is automatically synced with the desktop counterpart and provides corresponding
in-situ visualizations like a 3D trail visualizing the participant’s movement or a measuring
tape showing the distance between two entities.
We also have access to the same features as on the desktop.
So for example, we can rewind, fast forward or also directly jump to a specific point
in time using the timeline panel.
And attached to our left controller we have another panel that allows us to add new visualizations
or also modify existing ones.
For example, here in the video a frustum visualizer is attached to both tablets to highlight their
field of view during the study.
And because of these 3D visualizations in stereoscopic 3D and the environmental context,
we think, that the VR view is especially helpful for an in-situ analysis to gather more insights
into participants' behavior.
So, depending on the task, we can either work in VR for an in-situ analysis, or use the
desktop view for an ex-situ analysis.
Since the application state is automatically synced between both views, this allows for
an easy transition.
So any change that is made in VR is also instantly visible in the desktop view and vice versa.
The 3D scene window of the desktop view further supports this transition, as it provides a
preview to the VR scene and allows for a quick peek into the in-situ analysis without having
to put on a VR headset.
To validate our findings, we evaluated ReLive in a two-step evaluation process.
So we first conducted an internal design walkthrough as a formative evaluation and then used the
results to further refine our initial prototype.
Next, we conducted an expert user study, where we invited 5 researchers from different labs.
All of them had prior experiences in conducting and evaluating mixed reality user studies.
With our study we focused on three different research objectives.
The first objective dealt with our chosen task allocation.
Here, all experts agreed with our intended distribution of using a desktop view for an
ex-situ overview of the data and a VR view for a detailed in-situ analysis in the environmental
context, so akin to an overview + detail approach.
For the second objective, we looked at the interplay between the in-situ and ex-situ
analysis.
Here, the synchronization was found to be essential for this kind of asynchronous hybrid
user interface.
However for some participants we also observed a loss of orientation when switching between
desktop and VR, which could, of course, be further investigated in future work.
As third and last objective, we looked at the general applicability of ReLive.
Here, all experts were quite positive about our ReLive framework, and also saw great potential
to use ReLive for a collaborative analysis of MR study data.
Finally, we gathered our findings as design insights and research implications, which
you can read in more detail in our paper.
And with this, we hope to inspire more research into the use of asynchronous hybrid user interfaces.
For more details on our work, have a look at our paper.
And you can also find ReLive as an open source project on GitHub.