We will start with Bridger Herman and he's currently working in the
interactive visualization lab in Waterloo Canada.
And usually, he's creating interfaces to bridge
the gap between Art and Science.
However, he's also looking at data physicalization and how you can
use tangibles to interact with the data.
So, are you there Bridger?
Here I am.
Hello.
All right, so please go ahead.
All right.
Well, hello everyone, I guess.
Good evening or
Whatever time it is where you're joining from.
I'm Bridger Herman.
My pronouns are he, him and his.
Today, I'd like to discuss multi-touch querying on data physicalizations
in immersive AR, which is a novel approach to hybrid virtual and
physical visualization.
Let me begin with an introduction to why to physicalization itself
and why it's important.
One of the key ideas behind physicalization is to leverage our human
perceptual abilities that go beyond the visual and into the Tactile.
It allows us to depict even more data than traditional visualization
by using the material properties of these physical objects.
Physicalizations, engage our senses and make huge steps toward
making data.
Approachable such as putting our data in a museum or even allowing
individuals to participate by placing physical tokens.
Can you all see my slides?
Okay, I just can't see the screen share.
So I'm just making sure, alrighty.
So yeah, so basically we can place these physical tokens, but what if
the user wishes to make Dynamic queries into the data to select,
filter,
sort, rearrange, or otherwise, interactively, explore their data.
Augmenting
physicalizations with digital content is one solution to this.
Interactivity challenge, these so-called hybrid visualizations,
combine physical and digital elements.
Into one cohesive interactive visualization, prior work in this area.
Indicates two primary challenges and requirements for these hybrid
virtual and physical systems.
First, we've got to be able to sense input from the physical world
interactions, like pinch-to-zoom touch to select and swipe to pan.
So for this, we need to have a high accuracy method to pick up a
user's touch, on this irregular surface and translate it into digital
space and then react to it with our digital content.
Also, when we move a physical object, any virtual objects that are
related to it should follow and for this alignment and registration of
virtual and physical spaces.
We need high accuracy and low latency tracking between these two
spaces.
So we're obviously not the first ones to encounter these challenges.
So on the left, we have tangible landscape and tangible cityscape,
which are two projection-based physicalizations that use a depth
camera for touch input, and only need to have one initial alignment
between this virtual and physical space.
Example two
shows cityscapes, which also qualifies as an active physicalization,
which is a shape-changing display where the columns move
up and down, actually, according to the data.
Example three here shows a window-based AR physicalization of a
molecule, using a video camera and image markers to continuously,
register the virtual and physical worlds.
Lastly, plan well
also gives us a continuous image-based tracking and a pen input
device to select points on an immersive, AR visualization.
So our approach to these two challenges is to sense, touch through the
physicalization
and to register the virtual
and physical world using image targets affixed to an optimized
tracking base.
For the remainder of this talk, I'll go into some detail on our
approach for addressing these requirements and I'll demonstrate the
approach as apply to exploratory visualization techniques as well as
sharing the results from a couple of evaluations.
To solve a registration challenge,
we use the few Vuforia image tracking tool kit and a custom
truncated pyramid truncated pyramid display.
So here we can see four image targets that are tracked.
Their axes are shown in red, green and blue.
This truncated pyramid design was informed by initial latency.
Trials of an oscillating image Target and we determined that the
latency increases for each added visible Target.
So the tracking base makes sure that there are only one or two visible
targets during the overview and details viewing of the visualization.
To address this touch, sensing challenge, we introduced touch through
the data, physicalization.
So first of all, we start with the original data.
And in this example, it's volumetric 3D time-varying data from
supercomputer simulations and remote sensing.
This portion of the data includes ice, water and terrain from the
Filchner–Ronne ice shelf in the Antarctic and contains variables like
temperature, salinity and ocean current velocity.
So once we have the original data, we transform it and split it into a
series of columns so that we can sense touch through the
physicalization
We experimented with several different prototypes here.
For example, this is an early iteration of our touch sensing through
the physicalization, which used to CM squares, But ultimately, we
arrived at this final design of one and a half, centimeter diameter
hexagons, which can be seen in our final physicalization here.
Once the physicalization is tiled and split.
Then,
we 3D print it and these columns are sliced in such a way that a
regular FDM printer using PLA filament can print them which is indeed
what we did for our model.
Once this is all said and done, we place this physicalization on top
of a force sensing tablet such as the Sensel Morph.
And this allows us to react, to the users touch in the digital world
to actually make changes to our AR visualization.
So putting it all together.
We have our image tracking base.
That registers the virtual and physical worlds.
Then we have the physicalization, that's been tiled and placed on top
of a force sensing tablet.
These together can sense.
The users touch relay them to the AR device so that the a our
visualization can react accordingly.
So now that we have a basis for this technical approach.
Let's apply it to some of the most common data exploration techniques,
for 3D spatial data.
In the first technique,
we have a cutting plane into volumetric data, which is defined by a
users two touch points and the up Vector pointing out of the
physicalization.
In practice, the user can touch two points, and a white preview plane
appears.
And once the fingers are released, the actual query is made into the
data.
If the user wants to move around the plane before, like it's not quite
in the right place.
They can just move their fingers around and then release when it's in
the right place.
We also implemented a stream rake for analyzing Vector Fields.
The stream rake takes a user's touchpoint and the up Vector and
Seed streamlines along this line and add Vex them
Along that Vector field.
In practice, the user can touch points on the physicalization, to see
the streamlines and we can see the preview is shown as a little white
stick.
If the user wants to pick different points along the Streamline.
All they need to do is just move their finger and then
release, when they found a point, that's interesting to them on the
physicalization.
To partially overcome, the limitations of a small physical model.
We have also implemented a physicalization as a world in miniature.
This not only allows multiscale views of the data but enables a user
to have the physicalization, as a haptic proxy for the much larger
AR visualization.
It's also important to note that the physicalization does actually
correctly occlude the virtual visualization, in particular, notice the
peaks along the right here, the cutting plane will disappear behind
them.
And then reappear on the other side of the actual physical object.
So what about the evaluation?
We compared user touch accuracy with and without the tiled
physical present after informed consent.
We asked each of our eight participants to touch a sequence of 30 dots
on each of the physical ization and the bear force-sensing tablet.
We found that the euclidean distance from the ground truth points for
both of these conditions was just over 3mm
and a wilcoxon signed rank.
Test was unable to find a significant difference between the two
conditions.
So the difference in error in these conditions is really subtle enough
that we need a much larger sample size here to detect a significant
difference.
We also asked a climate scientist studying ocean currents underneath
the antarctic ice sheets to use our system and as he was using it, he
remarks that this like the physical model really grounded him when he
was working with the AR visualization.
And he also thought that having a system like this with the physical
context would be really important for his work with other scientists
to have a common physical reference frame as they're working together
on the data.
He also had some suggestions for us such as having a much larger scale
model of the physicalization would be a lot more helpful than the
small one and also actually depicting more data on the
physicalization itself would have been helpful such as including the grounding
line where the terrain the water and the ice meet.
That would have been super helpful to include on the actual
physicalization itself.
So despite positive results from both of our preliminary evaluations.
There are still some limitations to our approach.
Firstly, the print size is limiting.
We are only able to sense the forces, like the touches in a 24 by
13.8 cm area, which, which definitely limits the size of the print
that you can.
sense touch on here.
The print time may also be limiting in some workflows such as this
Antarctic print that we've been using as an example, throughout the
rest of the talk.
It's only 6 cm tall, but it took about two days to print at a high
resolution layer height of 0.15 mm
And also touch through.
The physicalization works best for convex geometries that can be
adapted to have a flat bottom, which doesn't necessarily apply to
every data set.
It works really well in cases, like terrain, but if you had say
something inside of a sphere, it may not necessarily work.
so well. During our scientists interview,
The physicalization was used both as a querying mechanism and a
contextual piece.
This brings questions to mind
Like what role does the physicalization play in the hybrid virtual
and physical data analysis?
And how much data do we encode in the physicalization, versus how
much do we leave to the digital visualization?
That's overlaid on it?
Our approach also has implications for adapting new querying techniques
that use pressure or even the tactile context of the data such as
whether the touch is in a peek or in a valley or within a particular
volume.
In summary, we set out with this lofty goal of fusing the interactive,
queering animation and data, exploration techniques of digital
visualization with the immediate and relatable tangibility of data,
physicalizations.
But even with modern input Technologies and immersive displays
combining.
These two things is still a hard problem.
Through our process,
we found that the approach of touch through physicalizations is
viable as an input mechanism for
hybrid visualizations, but we also need to continue to design for the
future of hybrid virtual and physical visualization, including
adapting, current visualization software to a tangible context, and
continuing to develop hybrid visualizations, that lead to greater
clarity and communication of our data.
I just like to acknowledge my co-authors, in particular
Max Omdal, who was super instrumental in getting this whole thing
working and my funding from the National Science Foundation and the
E3SM project for the data, and I'd also like to thank all of you for
being here.
Any questions?
Alright.
I hope you could hear the Applause.
Thank you for the really nice presentation.
Also for showing the props because you can't be here, but now we still
can get a feeling for it, how it actually feels and behaves like and I
will also ask the chat to give us some questions if there are any and
of course, the people here in the room as well.
But I would start with one if you would have tasked me to build
something like that.
I would probably have used the Kinect or Leap Motion or any depth
camera of some sort.
So, why did you choose your approach?
Yeah, that's an excellent question.
So one of the reasons why we chose to go this route instead of
using a Kinect is because of the limited resolution, especially on
like The Limited depth resolution of these depth cameras.
So you never know exactly.
When you're touching, the physicalization, especially with
complex geometries like this, so,
In our case, it's actually physically sensing the force that you're
touching.
So it's a lot more a lot more direct, I guess, and less less false
positives for your touches.
All right.
Thank you, are there any questions?
Hi Bridger, thank you for the presentation, cool stuff.
But as you said yourself, you know combining digital visualization.
With the physical visualization is a challenge and I was wondering if
you could speak to about some of the challenges of scaling this and if
you know what I mean by scaling, I mean today, I'm looking at one
ground section of Antarctica but tomorrow.
I want to look at another one.
What technologies would you use to make this more
scalable?
Yeah, that's an excellent question.
So by scalability, I'm assuming a referring to like this print
time and iteration and stuff.
Okay.
Yeah. And any technologies you would use to make it scalable.
Yeah.
So one of the trade-offs that we always deal with in creating
these physicalizations from
um, digital fabrication is resolution versus time.
It takes to create.
So actually in our conversation with the climate scientist.
He he said that it actually would have been acceptable to have a much
lower resolution version of this physical object because he mostly use
that for context as he was as he was working with it.
So we can maybe even say laser cut a series of slices to put on top of
each other and that would be much faster.
And 3D printing something at a super high resolution and that laser
cutting also would solve the problem.
Partially.
I suppose, solve the problem of like the print size because of the
limitations of the print bed on the 3D printer.
Right.
So, you would incorporate the whole printing process into the
workflow.
I mean, you see that as part of the process, also in the future.
Yes.
Absolutely.
Okay.
Is there another question?
All right.
Bridger then once again, thank you for the presentation and hopefully
next time we can have you in person and try out as a demo.