Hello, I‘m Patrick Reipschläger, and I
want to present our work AvatAR, an immersive analysis environment for human motion data
combining interactive 3D avatars and trajectories.
This is work I did together with Frederik Brudy, Justin Matejka, George Fitzmaurice
and Fraser Anderson from Autodesk Research, as well as together with Raimund Dachselt
from the Technische Universität Dresden.
Avatar uses an Augmened Reality Head-Mounted
display to visualize virtual avatars reconstructed from 3D skeleton data directly in the environment
in which they have been captured.
We designed several techniques for visualizing not only the avatars themselves, but also
where a person walked, where they looked at and what objects in the environment they touched.
All visualization techniques can be accessed using the body of the virtual avatar itself
by mid air interaction.
Furthermore, we also integrated an interactive tablet, which provides alternative touch input
and acts as an overview device.
The visualization of human motion data has
been studied extensively.
A majority of these works are desktop-based solutions using a combination of 2D map-based
trajectories and heatmaps, sometimes accompanied by auxiliary visualization.
However, there is also research which uses immersive technologies, such as Virtual and
Augmented Reality to visualize motion data as 3D trajectories.
What differentiates AvatAR from previous work is the focus on immersive in-situ analysis
using humanoid avatars, complemented by 3D trajectories, as well as visualizations embedded
directly into the environment.
Thus, AvatAR provides a system of integrated techniques that have the potential to reveal
much more insights when analyzing motion data than just trajectories-based tools.
For a more detailed analysis of the related work, I would like to refer to our paper.
The core principles of AvatAR are as follows:
An analyst wears an AR Head mounted Display an can move freely to an enviornment to analyze
spatially registered human motion data.
The visualizations take the form of humanoid avatars which can be directly interacted with
using mid-air input.
These avatars are supported by other visualizations,
such as 3D trajectories, which can be accessed by interacting with the avatars Body parts.
This put a variety of techniques at the analyst's disposal, which can be utilized for gaining
insights into the bevaviour of persons directly in the same space in which the data has been
recorded.
To summarize, our contributions for avatar
are: The design of an in-situ analysis environment
for human motion data using a variety of techniques, accessing an manipulating those techniques
through the body pars of an avatar as well as a tablet interface,
and a prototype implementation of our concepts as well as three use case scenarios
In the following, I will present each of the
visualization techniques that toghether compose AvatAR and elaborate on how they can be useful
tools for the analysis of human motion data.
The central visualization technique are the
humanoid 3D avatars themselves, which provide a detailed representation of a person‘s
posture at a specific point in time.
They are color-coded to represent different persons and can be animated to perceive them
in motion.
Furthermore, each avatar provides access to
additional visualizations, like trajectories, by interacting with the various body parts
of the avatar itself.
In contrast to avatars, 3D trajectories provide
a good overview of multiple points in time.
Our system provides those trajctories not
only for the general position of a person, but for individual body parts like hands,
feet, and head as well.
However, trajectories offer much less detail
in comparison to avatars.
By combining the two, our system offers the benefits of both techniques.
Another technique that utilizes the combination
of trajectories and avatars is the Specter Visualization technique, where the humanoid
avatars are used in a trajectory like manner, to provide a detailed overview over multiple
points in time.
Another technique that tilizes the combination
of trajectories and avatars is the Specter visualization technique, where the humanoid
avatars are used in a trajecotry like manner, to provide a detailed overview over multiple
points in time.
Because AvatAR is an in-situ analysis framework,
visualizations can be directly embedded into the environment, for example by showing heatmaps
on the floor that gives a general overview of where people moved.
These visualization can even be occluded by real objects and thus become part of the environment
itself.
As an alternative to heatmaps, AvatAR also
provides a footprint visualization, that shows imprints at the exact positions where a persons
feet touched the floor.
This allows for more insights when analyzing the movement of a person, than the abstract
overview provided by a heatmap.
Another important aspect when analyzing a
persons behavior is where their visual focus was directed at.
For this, we present the gaze visualiuzation, which sends rays from a persons eyes into
the environment and visualizes where they collide with other objects.
This enables an analyst to perceive what a person could see from their current position
and which things were obstructed by other objects or their own body.
The last technique I want to present visualizes
touches of a person with the environment.
This can be useful for an analyst to understand, what objects a person touched and when.
The touch point are calculated by measuring when the distance between a person‘s finger
and an object in the environment falls below a certain threshold.
By selecting one of the visual imprints created by this technique, time is advanced to the
point where the touch occurred.
Thus, an analyst can quickly switch between different points and use the other techniques
to analyze the context of a touch for further insights on what a person was doing.
While our system utilizes mid-air input for
interacting directly with avatars and other visualizations, this can sometimes be impractical,
because objects might be far away or obstruted by other objects or the environment.
Therefore, AvatAR provides analysts with a tablet interface as well, which supports touch
interaction.
It allows to activate and deactivate all visualizations for each avatar and provides time controls
for manipulating them.
Furthermore, the tablet acts as on overview
device as well, providing a floor plan view which shows the current position and orientation
of the analyst as well as the position of avatars and 2D top down trajectories of their
overall movement.
To explore the technical feasibility of our
system, we developed a prototype using the Micrososft HoloLens 2 as AR device and a Microsft
Surface Pro 7 as accompanying tablet.
The applications for both devices were implemented with the Unity 3D engine and synchronize their
state over a central TCP server.
For datasets we used the CMU Panopic skeleton datasets, the JTA dataset and a custom dataset
using one of the authors home office environment.
To demonstrate the transferability of our
concepts to different applications domains, we developed three use case scenarios.
The first one is an office environment where we explored how AvatAR may be used to check
things like social distancing and identify areas that are frequently touched by several
people and therefore may require special hygiene considerations.
The second one is a remote analysis of dance
cheographies, where an instructor can remotly analyze the dance routine of client by using
a combination of AvatARs visualization.
This use case is based on CMU panoptic dataset.
The third and last use case is a shopping
scenario, where an employee uses AvatAR to analyze movement patterns of shoppers throughout
the store to identify bottlenecks and their typical routines when visiting the store.
This concludes my presentation of our work
AvatAR.
I would like refer to our paper for a dicussion of the limitations of AvatAR and it‘s relationship
to the underlying dataset.
We consider the next step for AvatAR to take our prototype to conduct a user study in a
real-world use case scenario and evalaute, how AvatAR can help to support the analysis
of human motion data.
We are confident that our work can provide a real benefit in this regard and help analysts
immerse themselves with the pesons bein analyzed.