Hi, I'm Pascal Janssen and I'm presenting AutoVis.
Autovis is an analysis framework enabling the immersive and interactive analysis of
in vehicle interactions, passenger behavior, and physiology,
combined with a replication of the vehicle environment and study context.
Today's automotive UI studies often
consider numerous factors such as interaction with external HMIs,
novel input and output modalities like speech and gesture, environmental context,
passenger physiology, and temporal sequence of interactions.
Consequently, practitioners get a large bodies of heterogeneous
spatial temporal data of movements, interactions, audio, and video recordings,
making human vehicle interaction analysis more complex.
Practitioners gather large bodies of heterogeneous spatial temporal data of
movements, interactions, audio, and video recordings, making human vehicle
interaction analysis more complex. their work has shown two different approaches.
Some works used a non immersive approach, providing holistic overview of the data,
enabling reproducible and flexible analysis.
However, this either focussed on a
specific research topic such as gaze and touch interactions, or only focused on a
manual driving and their visualizations are noninteractive.
In contrast, immersive approaches provide
visualizations in augmented or virtual reality.
The emotion can be beneficial to
understand spatial data and environmental context.
However, immersive approaches currently do
not support the automotive UI domain with its unique combination of a moving
interaction space and a dynamic environment with numerous road users.
Therefore, we present Auto V is.
The core of Auto Viz is the synchronized
interplay of a desktop and VR view, l erraging the best of both worlds.
The desktop view enables non immersive data analyzes.
The 2D panel provides a holistic overview of 2D data as line diagrams and events.
It hints at interesting sequences via
outlier detection and horizontal lines showing the me.
The 3D scene panel replicates the original study environment using a virtual ego
vehicle, other road users, passenger behavior, and environmental context.
The timeline controls tool wide play back of the audio 3D scene, video, and audio.
Additionally, analysts can select and annotate sections on the timeline.
Auto Viz considers four event types
interaction, emotion, driving, and activity.
These events highlight relevant sequences.
Before the examination of the data, the VR view enables immersive analysis.
Avatars replicate passenger movements from
prerecorded 3D skeleton data, and an integrated Avatar visualizes the average
positions and rotations of individual avatars.
Trajectories visualize hand and head movements for a selected time frame.
In addition, in vehicle gaze and touch
heat mapsprovide an overview of interactions with interior surfaces.
Environmental gaze and pointing heat maps
accurately map the replicated 3D meshes of buildings and other road users.
A virtual tab that displays study related metrics, the 2D panel, and events.
Analysts can quickly navigate through the scene via camera switches and a timeline.
To improve scene navigation, a Minimap provides a top down view of the analyzed
environment, and Auto Viz visualizes spatial temporal
events along the driving path, indicating their location and orientation.
To visualize the link between in vehicle interactions and environment,
context portals provide a glimpse of the reference context.
For example, a location or object by
showing a spatial temporal next to the robot's finger or head.
Analysts can place annotations to leave hints, descriptions, and opinions for
collaborators or when switching between desktop and VR.
Besides, analysts can use labels to annotate their data set.
For example, for deep learning applications.
In addition, desktop and VR
are pass through VR through leveraged rear vehicles, interior visuals, and haptics
for an immersive analyser specific to the automotive domain.
Autovis places the 3D visualizations at
intended positions within the rear vehicle and third from a data set.
Analysis can selectively activate the pass through for specific interior locations.
We also provide a companion tool that
preprocesses data according to the AutoVis data specification.
The companion converts low level signals such as pupil size into a high level
information such as cognitive load or drowsiness.
We use state of the art deep learning such as deep face, open pose, or yolo.
We demonstrate AutoVis using authentic
data from the use case of multi model in vehicle interaction.
For this, we simulated a ride in an
automated vehicle in a downtown San Francisco to represent a typical study.
Participants can freely interact with
multiple modalities, touch, gesture, gaze, and speech.
After loading the data and out of this,
analysts can start the analysis according to their research goals.
In our use case, the goal is to investigate the usability of multi model
interactions depending on the context, time difference, order, and their usage.
The heat maps and the avatars help to identify behaviors.
Some participants gaze collectively at the targeted and pointed close to the gaze
hit, while others pointed in mere directions with bent elbows.
The action events on the timeline
highlight a time difference between the interactions of 0.5 seconds.
The head movements indicate that participants mostly ignore the other
passengers, except during an additional conversation.
In addition, the heat maps show a widespread pattern indicating that
participants likely were curious about the unfamiliar setting and explored the
interior by looking around and touching it.
The environment heat maps also revealed
that participants interact with the unexpected cyclist.
In our second use case, we demonstrate how
Auto Vist converts and visualizes real world data sets.
For this, we leverage Drive and Act, which is a data set for driver activity
recognition, captured in manual and autonomous driving.
In Auto Vist, analysts can perceive the automatically recognized body positions
and movements during drive activities from any point of view.
In our third use case, we demonstrate how
Auto Vist integrates a real vehicle into the analysis.
For this, we recorded an exemplary data set in a BMW to analyze the novel UI
concept for an air conditioning control specific to the driver's seat from the
driver's point of view using Pass through VR.
The analysts discovered that participants primarily interacted with the center
console instead of the door where the novel UI was located.
Besides, they found a mismatch between the
door button layout of the real vehicle and the simulator vehicle used in the study.
To evaluate out of this, we conducted
heuristic walkthroughs according to Osen within these use cases.
Our use cases demonstrated that out of this meets most heuristics.
However, some heuristics are not met.
Autovis currently does not fully empower
novices as navigating an automated data set in VR can be overwhelming.
Besides, scalability is not met yet as
visualizations may overlap with increasing data set size, number of objects, for
example, in pedestrian crowds or traffic jams.
During implementation, we also get some insights.
The visualization is supposed to enhance
collaboration, could occlude the visualizations of individual users.
A safe analysis using the Pass through VR can only be performed as passenger, and
analysis during driving is only possible when driving the same study route.
Besides, imperfect real world data may
result in limited or missing visualizations.
In addition, automatic inferences in the
preprocessing could be erroneous and cause misinterpretations by users.
Finally, AutoVis can also serve as an analysis tool beyond the interior when a
vehicle is equipped with the right sensors.
With AutoVis, we aim to facilitate the analysis of human vehicle interaction.
For more details, see our paper and the open source code available on our demo.