Hello, my name is Ang Li, I am going to present our research titled "GestureExplorer: Immersive
Visualisation and Exploration of Gesture Data.
Our research is a result of the collaboration between Monash University, the Univerisity
of Queensland, as well as the University of Canterbury.
GestureExplorer is a virtual reality tool for exploring gesture data in 3D space.
It supports spatial understanding by representing gestures as 3D skeletons overlaid with motion trajectories.
Along with multiple 3D visualizations and coordinated 2D views, GestureExplorer turns
the surrounding immersive space into a part of the analytical process for a better understanding of gesture data.
GestureExplorer is primarily designed to support the analytic requirements of Gesture Elicitation Studies.
This type of studies collects and analyses gestures proposed by end users, extracting
common patterns to inform future gesture designs.
As a result, massive computational supports are required to identify potential gesture
patterns or clusters in the dataset.
Besides, it requires manual refinement of the identified patterns to improve the accuracy
of them.
Different views are also in demand for understanding gesture data.
Though there have been prior works that attempt to facilitate this type of study, what differentiates
our tool is our use of a spatial approach to better the understanding of gesture data.
As such, we raised extra rationales from an immersive analytic perspective.
First, we aim to exploit the surrounding space in the immersive environment to organise and
analyse gesture data.
Second, we provide multiple views and features that could be used in coordination for a better
understanding of the gesture data.
Lastly, we design GestureExplorer based on Munzner’s visual analytic framework as well
as Shneidermann’s mantra.
Here is an overview of the features we implemented in GesutureExplorer, each of them can be mapped
to an abstract visual analytic task as defined by Munzner.
Using a combination of them allows users to have not only an overview of the dataset,
but also a detailed understanding of data that is of interest when necessary.
I would like to demonstrate GesutreExplorer using a few scenarios in the context of a
typical gesture elicitation study.
In the start, users will need to have an overview of the entire dataset, to identify potential gesture patterns.
For this, we come up with spatial arrangements leveraging dimensional reduction algorithms
such as PCA and MDS to help users observe the overall distribution of data, and also clustering algorithms
such as k-means to generate the clusters.
In this scenario, the user simply turns on the overview map and observes the distribution
of gestures under spatial arrangement resulted from PCA.
He roughly identifies 3 groups of gestures and thereby sets up a clustering session based
on his observations.
And as a result, they receive the following clusters.
After the user identified the potential clusters, they will need to refine the clusters by finding
and excluding outliers from them.
To do this, we provide some other spatial arrangements such as the global, local and
the line-up arrangements.
These arrangements encode the similarity of a gesture to the average value of its corresponding
cluster into distance.
For example, in this scenario, the user is using the local arrangement and viewing the
distance between each gesture to the centre of the cluster.
The farther a gesture is from its cluster, the lower similarity it has with the other
values in the cluster.
The user can thereby mark them up as potential outliers.
Similarly, this could be achieved by the line-up arrangement, which organises gestures in a row based on their similarity.
row based on their similarity.
The user could expect to find the least similar gestures at the end of the line.
Now we have the outliers identified, the next thing to do is to make comparisons of them
in order to determine if they should be excluded from their current cluster.
GestureExplorer has multiple visualisations for users to identify the similarity and differences
between gestures.
The first way to do this is simply looking at the shape of their motion trajectories
for any significant differences.
The user can also pull out the small-multiples of the gestures for the difference in their
motion trends.
To gain more details, animation could be played for the gestures.
And if the animation is viewed together with the small-multiples, an overlaid animation
will be also applied to the small-multiples themselves.
Users can also stack gestures together and turn on the heatmap effect to highlight the
similarity and differences among the motion trajectories.
We blended the transparency of the trajectories, so the brighter an area is, the more frequent that area is visited by the trajectories.
We also provide a filter feature that allows people to focus on the motion of particular
body parts that are of interest.
Finally, after the outliers are confirmed, they could be simply excluded by initialising a new cluster.
Notice that our tool is dynamically updated, the position of the gestures in the new cluster
will be computed in real-time.
Lastly, I would like to highlight a very interesting feature we had in GestureExplorer, which is
the embodied search feature.
This feature allows people to propose their own gestures simply using the two VR controllers,
and using their proposal to fetch similar gestures in the dataset to inform the popularity of their proposals.
We evaluated GestureExplorer in two user studies.
The first study validated its usability.
We invited nine academics in the field of immersive analytics, data visualisation and
human-computer interaction.
They experienced every feature implemented in GestureExplorer and provided valuable feedback
through a semi-structured interview.
Overall we received quite positive feedback that the visualisations were intuitive and
immersive, and our tool was entertaining to use.
We were able to thereby improve existing features and introduce new ones based on their feedback.
I recommend referring to our paper for more details about this study.
The second study evaluated GestureExplorer’s practical ability to facilitate the analysis process
of gesture elicitation studies.
We invited ten participants and simulated a typical pattern analysis in the elicitation studies.
We purposely mixed up 90 gestures that belonged to 3 different gesture types, and asked our
participants to freely use our tool to identify the three patterns and thereby refine them.
After this, we gathered participants’ preferences via a UX questionnaire.
We had a very interesting observation that our participants were able to complete the
tasks using different analytic strategies.
The first bunch of participants relied more on the different spatial arrangements in GestureExplorer,
in order to identify outliers and exclude them correspondingly.
Another type of participants relied on more detailed visualisations such as the trajectories,
and the animation.
They made detailed comparisons between gestures and were also able to identify outliers and
refine the clusters eventually.
This confirms that GesutreExplorer is a promising tool as it allows different approaches of
analysis.
The approaches are different in the procedures, but are able to lead to the same outcome eventually.
To wrap up, we presented GestureExplorer.
This is to our knowledge the first visual analytic tool that enables complex and embodied
analytics in the immersive environment for gesture data.
We have gotten confirmations from domain experts and users that our tool is viable for the
exploration and analysis of gesture data.
For future works, we would like to make comparisons of GestureExplorer to its 2D counterparts
to gain further insights, as well as to explore other data types with GestureExplorer.
As there is a growing community in immersive analytics, we would like to share our tool
with the community to provide insights for future research.
Our tool is available via this QR code, and feel free to get a copy of it and try it yourself,
thank you very much.