- Okay, thanks for the introduction.
So, this is work done together with my colleagues
Patrick Reipschlager and Ricardo Langner,
as well as my supervisor Raimund Dachselt
at the Interactive Media Las Dresden.
So as you might know, 3D visualizations have
several challenges such as, misleading perspective
or occlusion.
You can see this in the terribly drawn
3D bar chart that I created in Excel there.
Also interactions, just imagine having to select a data item
in the 3D scatter plot down there.
So these are well-known issues but we believe
that mixed reality today has the potential
to address some of these issues and why is that?
Now, we believe that mixed reality being suitable
for 3D content can address these issues
for example, interaction.
So we have non-traditional devices,
new exciting interaction paradigms,
spacial interaction and so on.
We also sometimes have situated data
and semantic connection even to a physical location
so this is an interesting trend.
There's things like, for example immersive analytics.
Some of you might have known, might have heard about that,
so a recent trend in visualization community.
So one might be thinking about a revival,
so to speak, of 3 visualizations.
I wouldn't go that far because I don't believe
that we can address all the issues but,
at least try and look at what can be done here
and as I said, interaction is one of those things
that might benefit 3D visualization.
Spatial interaction might actually be
one of the key advantages of Mixed Reality solutions
compared to more traditional forms of input.
When I think about that or when we talk about that,
we specifically mean in this case with like tablets
or other handheld devices that the device position
is mapped to a virtual camera and you basically,
as you can see in the short video clip there,
freely move around, explore the visualization in this case,
its virtually located on top of that table
and use that to basically explore the data
and you can also see a picture to the left of that.
So this is what we're interested in,
but the question is of course is this really useful?
It might sound exciting, but maybe its not unusable or
even worse than traditional input so we don't know that yet.
How do we use spatial interaction in comparison
to other modalities?
What are the benefits, if any?
And of course, what could be possible limitations?
And yeah, that's what we would like to find out.
So what we did is a study where we compared
spatial interaction to touch input
for 3D data visualization on mobile devices.
We, during that study, identified some interaction patterns
that we really found exciting and we'll talk about that.
And I believe we are able to inform about some insides
into challenges of situated 3D visualizations.
So of course we're not the first to look into this
or to work in that area and there's a whole body of work
concerning for example, peephole windows.
So basically windows into 2D or even 3D information spaces,
such as work done by Fitzmaurice or
Spindler's work into tangible views for data exploration.
Then there's of course, several papers that use a
mobile device, a tablet or a smart phone as kind of a
3D controller for controlling 3D visualizations,
kind of decoupling the input and output space a little bit.
Then obviously, Mobile AR in itself and
reasoned technological advances lead to
new applications in that area and
we see the first visualization applications
using Mobile AR technologies.
And then, there are also specific studies on spatial input
such as, the work up there.
Benjamin (mumbles) presented that some weeks ago at
(mumbles).
Also, Martin Spindler's work on pinch-drag-flick
where basically he tested or he studied spatial interaction
for 2D navigation or navigation in 2D data sets compared
to touch and found that this can actually outperform touch.
So how does this work for 3D?
Our prototype, or set up for our study looks like this
so we have a tracked tablet.
We have a virtual information space
that we place on a table.
That table is not interactive.
Its just kind of a physical landmark to orient yourself
and we have some study PC in some.
As the baseline condition of touch,
we used Orbit camera model.
You can see that in this video.
So basically you use one or two finger drag gestures
and we support camera panning, rotation and zoom.
So its basically the same prototype that we also use
for the spatial interaction but with a touch camera.
Now we investigated 3 different visualizations and 3 tasks
and the first is a navigation task.
We have a 3D height map and the task was to locate
and select several markers, several targets
in this visualization.
So this was really heavy on navigation and we tried
to focus on that first and then we have two more tasks
that also, obviously have navigational components
but also a little bit more than that.
So the second task was a comparison task.
For this task we used 3D bar chart and as you can see,
the task was basically to compare the height
of 2 indicated bars and find out which one was higher.
And the third one, we call that
structural understanding task,
so basically you had a 3D scatter plot and
in this scatter plot there were several clusters of data
and users had to explore the scatter plot and
try to understand and memorize this and
then on an additional study PC they were presented with
several pictures of scatter plots.
One of them was the correct one and the other was similar,
but a little bit different and they had to find
the right one, basically.
So as far as study design we did a within subject design
with the three tasks I just presented and
the two current conditions of touch input and spatial input.
We collected lots of data obviously.
We recorded all the participants during the study.
We wrote down observations and we also,
that's going to be important later on,
tracked and recorded the camera position both
for the touch and for the spatial camera.
In the case of the spatial interaction this was
of course also the device position.
We also had questionnaires, so after each task we had
Likert scale questionnaires on task load
and user preferences and we also measured
task completion times and error rates where applicable.
So in the case of the navigation task,
there were no errors like that.
So we had 18 students and postdocs taking part in our study
and all of them completed the study successfully
so let's see, what do we find out.
Let's take a look at the results.
Maybe let's start with the task load
and the user preferences.
So in general, and I have to say that I only present some
of the results here.
There's much more in the paper.
In general we found out and this is not surprising,
the physical demand was higher
for the spatial interaction task because
you have much more physical movement of the table,
of the tablet in comparison to just holding it
and touching so this was not really surprising.
But we also had better control of the camera
so users reported that they felt more in control
and overall, spatial interaction was preferred
by most of the users.
We also have some significant results specifically
for the first task, the navigation task.
So their spatial interaction was rated to be more precise
and users also reported a lower mental demand
as well as stress level during doing the task
and they also had a higher perceived success rate.
Some of the comments we got and as I said,
there's lots more in the paper,
they said that interaction, spatial interaction
was more memorable than touch, more engaging and
also more intuitive but they noted the weight of the device
as one of the problems and also tracking
was sometimes criticized so we had some slight jitter
at the edges of our tracking volume sometimes.
So those users that really made use
of the whole interaction volume sometimes experienced that.
Now if you look at task completion times and error rates,
its interesting whether the preferred method
of interaction also performed better so as I said,
perceived level of success was higher
and we also found that for the navigation task
when looking at the data.
So spatial interaction was significantly faster
with quite a large effect size but on the other hand
we did not find any significant differences
between touch and spatial interaction for both
the comparison task and the structural understanding task.
And our idea here, our reasoning is that we believe
that navigational components, that is basically
what we do when we move the tablet around,
don't play as like of the role and
the more complicated tasks where you also had to focus
on points while comparing to bars in this bar chart
and you didn't really need to navigate that much,
or at least it didn't help you very much.
Now what's also interesting is that we observed
several distinct patterns of interaction during the tasks
and this was kind of helpful because we found
some exciting insights I guess.
So what we did is we observed those patterns
during the study and then we analyzed our video recordings
and tried to come up with a list of those patterns
and then we validated that by looking
at the tracked and recorded camera data.
So we did, or we created a custom 3D camera trajectory
plot visualization and I don't know if you can see that
in the video because its rather fine lines,
but you can see it maybe in the picture up there
or also on the screenshot.
Basically, what you can see there is these colored lines
showing where the camera moved during the task
and the color encodes the speed,
or the relative speed I should say,
of the device so red was slow movement
or being stationary while green is faster movement.
And I'm also going to talk about some
of the results we have here but again,
only the most important ones so.
One pattern that we identified was, we call it
bird's eye views and I hope that you can see it in the video
but I'll try to explain as well and you can see it
on the left side for touch and then again
for spatial in the middle.
Basically the idea that users tried to look at the data
from above trying to reduce the dimensionalities.
So if you look at a 3D height map from above you don't have
any occlusions so this is helpful and this is behavior
that we didn't train them, we didn't tell them to do that
so users and I have to say almost all of them,
learned that during the task in a very short amount of time
and it was easier for them to look at that.
And of course for spatial interaction you can't really hover
above the data, like physically impossible,
but still a similar pattern emerged.
And in this second one I wanted to briefly talk about this
using the principal directions.
For example, for the bar chart but also in the scatter plot
task users would look at the data from the sides,
from the top and from the front and you can see
those clusters basically in the camera movement data
and this is the same for spatial interaction as well.
So let me briefly talk about some of the limitations
of our study.
Each task, and you might have noticed already,
we only tested with one visualization so
this is definitely a problem and our touch camera,
while we tried to do it as best as we can
there is no accepted gold standard so you could argue
that maybe a different touch camera would perform better.
All the data, or most of the data used is artificial
and our participants weren't really visual and
analytics experts but, yeah students basically
so we don't know if this tranfers to other user groups.
However, we still found strong indication that navigation
at least, can benefit from spatial interaction and
navigation is kind of the basis for all other tasks
so this is kind of a good result for us, yeah.
Now, some insights that we found is that its important to
talk about the connection of the physical and virtual.
So this should be done in future work,
like the position and scale of the data,
how to use physical landmarks and we found the table
was really helpful for our users
but this should be investigated further.
Guidance and learnability is an issue so
these default strategies that some of our users used.
Could you try and tell them before that,
build this into the visualization maybe in some way?
So this should be looked into.
And also tool assistance because we had really simple tasks
and more complex tasks might need tools,
such as clipping planes and such,
so this is also some future work I guess.
So in conclusion, we believe that mixed reality
might benefit 3D visualization.
Spatial interaction is one but not the only of course,
promising paradigm in this area.
The results of our study, I think we believe underline
the potential and we also found some distinct behaviors
or patterns and yeah with that I can honestly say thank you.
My name is Wolfgang Buschel, thank you for your attention
and I am open to questions.
(applause)
- [Man] So, we have some time for questions.
- [Woman Interviewer] I'm Leila from Waterloo.
So you mentioned that the users reported that
with the spatial interaction they felt more in control
of the data, is that true?
I was wondering if you know why that is.
Did they feel like, more immersed using that technique?
- Yeah, that's an excellent question.
So the, more or less exact wording of the question was,
'Do you feel in control of the camera,'
and then of course they had to rate that on a Likert scale
so I wouldn't go as far as saying, as you said,
feeling in control of the data but rather the camera
to explore the data.
So I don't want to speculate whether or not
that's because of immersion.
I believe it was mainly being a very
intuitive interaction technique.
It was really fast so for example, we had
of course some short training period before the tasks
and we found that this is basically not needed
for spatial interaction so for touch they needed
a little bit of training to understand all the capabilities
like, do you need one finger, two fingers.
Whatever.
For spatial interaction it was (snaps)
instant basically so that might be a factor.
- [Man Inteviewer] I thank you very much.
Very nice work.
I am Miguel Nacenta from the University of St. Andrews.
How do you feel about hybrids?
It seems kind of like you navigate in using
the interaction (mumbles).
- The hybrids? Sorry.
- [Man Inteviewer] The hybrid techniques.
- Oh, so you mean combining spatial interaction
and touch or?
- [Man Inteviewer] Yes.
- Yes so we didn't really test this in our study
but I believe, or we believe I should say
that this is one of the advantages and
its also one of the reasons why we looked
into mobile devices instead of AR glasses or something else.
The advantage of having both the option
to use spatial interaction but also touch.
For example, for precise movements, for selection obviously.
So we didn't look into this in this study
but this is one of the things that I believe
are very promising.
For example also, using spatial interaction to position
the camera to navigate and then using touch
to control a clipping plane and
this is something we tried out in our prototype
but didn't do in the study. Thank you.
- [Man Inteviewer] Yeah I agree with you.
I think its an interesting direction for future work.
- Thank you.
- [Man] Okay.
Let's thank Wolfgang again.