Hello everyone, my name is Benjamin Lee. I will be presenting our work on an immersive
visualisation animation grammar called Deimos on behalf of my colleagues listed on screen.
One of the goals of Immersive Analytics is to use engaging embodied tools for data
understanding. Existing works have explored the use of embodied interaction techniques to
manipulate immersive data representations as though they were real objects.
For example, Tilt Map by Yang et al. clearly demonstrates how an embodied
interaction technique can directly be mapped to changes in a visualisation’s state. The
use of animation facilitates changes between the three visualisation types.
Traditional animation research has focused primarily on the design of the animation itself,
and not how it can be controlled or manipulated by the viewer.
Even so, animation research in Immersive Analytics has been sorely lacking. Many
works do not intrinsically focus on animation design,
instead using it as a by-product of some other form of action.
In our previous work at CHI 2022, we presented a design space for
how immersive visualisations can be transformed between 2D and 3D.
A core part of the design space was the user interactions used to perform
the transformation. However, it focused only on their design and not on how they can be created.
In this work, we continue by introducing Deimos, a declarative grammar for authoring dynamic,
embodied, immersive morphs. What you see on screen is a morph which
unstacks a 2D barchart into a side-by-side 3D barchart using an extrusion interaction.
Here is another example. As the user pushes a 3D barchart into a wall, it partitions into a
faceted 2D barchart and repositions onto the wall. Pulling it off the wall undoes this.
That barchart morph demonstrates two of our three design goals.
First, morphs are adaptable and dynamic. This 3D barchart was just a regular 3D barchart. However,
by pushing it into a wall, the morph is dynamically applied to it.
Second, morphs support embodied interaction. This type of hand
interaction is made possible with immersive technologies.
Our third design goal is simple: we want to support conventional
animated transitions as well, such as using familiar sliders.
These morphs demonstrate two contrasting approaches to animated transitions. The
conventional is data-driven. Animations are designed to showcase some particular aspect
of the data, with the viewer having little to no control.
Our new approach is interaction-driven. Animations are dynamically applied to
visualisations at runtime in response to user interactions, assuming the correct conditions
are met. The user can decide when and how to use the morph by performing the interaction.
These two approaches are facilitated by our grammar being dynamic. It can be set
up to work with specific interactions and visualisation configurations,
or be applied more generically to any 3D barchart regardless of variables and dataset.
Now let’s briefly go through how the grammar works. We begin with
a visualisation layer, which in this case is DXR by Sicat et al.
DXR uses JSON specifications linked with datasets to create visualisations in Unity.
Deimos sits on top of this as an animation layer. Deimos first loads all of the morph
JSON specifications. It then keeps track of each DXR visualisation as they update. Depending
on their state and user inputs, Deimos applies morphs to the DXR visualisations.
The question then becomes, so what are these morph specifications?
Morph specifications are written in JSON and follow the declarative Deimos grammar. Here
is an example of a very simple morph and its specification. It changes the colour
of any visualisation with spherical marks that are uncoloured to red.
This occurs whenever the left hand performs a “select” gesture, and not the right hand.
A morph specification consists of three components: states,
signals, and transitions. Let’s now go through these one by one.
First are the states of the morph. These are best thought of as animation
keyframes. Here we have one state for the base unhighlighted state,
and another for the highlighted state. Unlike regular keyframes however, they can be partially
defined. Remember that we apply these morphs dynamically to existing visualisations. Thus,
these states also function as a set of properties a visualisation must have to undergo the morph.
In this case, these two sample scatterplots are considered to be in the unhighlighted state,
as both have sphere marks with no colour encoding.
This in spite of the two using completely different datasets.
This barchart however does not match as it has cube marks with a colour encoding,
and thus cannot undergo the morph.
Note however that this highlighted state has a restrict: true parameter. This means that
a visualisation cannot match with and start at that state, but only transition into it.
Second are the signals of the morph. They are best thought of as streams
of data that emit values based on user input. The values from these signals can
then be referenced elsewhere in the morph specification. In this example we define
a signal called “leftHandSelect”. We specify its source as the hand with a left handedness.
We then choose the “select” value to be extracted from this hand, which is the pinch gesture that
you see. This signal will return true if the hand is pinching and false if it isn’t.
Third are the transitions of the morph. Transitions are what actually connect the
aforementioned states together to allow animations to occur.
We first specify which two states to connect together. In this case we declare the names
of both our aforementioned states, in the order of start state and end state.
We then optionally state the trigger for this transition. This is basically the condition
that must be true for the transition to actually begin. In this case we
use the signal that we declared to have the transition start when the left hand pinches.
Lastly we add some additional parameters to refine the transition’s behaviour. We specify a timing
value of 0.25 seconds. We also make the transition bidirectional, meaning it can occur in reverse.
When we put this all together, we can see that this forms a state
machine. The state machine diagram is colour-coded with the specification.
If a DXR visualisation has a sphere mark with no colour, it enters the unhighlighted state.
When the left hand starts pinching, it enters the transition for a time period
of 0.25 seconds and ends at the highlighted state.
When the left hand stops pinching, it re-enters the transition in the reverse
direction for another 0.25 seconds and ends at the unhighlighted state.
If at any point the DXR visualisation’s specification is modified outside of this
morph, such as through a GUI, it leaves the state machine.
It may then immediately re-enter if it matches the unhighlighted
state again. If it doesn’t match then it simply doesn’t re-enter.
And that is at its simplest is how the Deimos grammar works. It listens
for changes to visualisations, and applies morphs to visualisations based on user inputs.
Deimos itself can be found on an open-sourced GitHub repository. The link is included in
the paper. In the repository you can find other examples that you can try such as this
recreation of Tilt Map. Detailed walkthroughs on how to use the grammar are also included.
Lastly, the paper included an expert evaluation by six fellow Immersive Analytics researchers who you
see on screen. We got them to use Deimos for at least 2.5 hours and interviewed them afterwards.
They reported that Deimos did have a steep learning curve,
especially as they had to learn both a visualisation and animation grammar. They
did however see the potential in the grammar in expressing new animation and interaction designs.
Some proposed a GUI to simplify this authoring process, along with suggestions for how to do so.
Some others also saw potential in using Deimos for prototyping
their own research ideas. Of course, please see the paper for full results.
And thank you all for listening, and special
thanks to the six researchers who participated in our study.