Hi. Welcome to the presentation of our paper "Is it my turn?
Assessing Teamwork and Taskwork in Collaborative Immersive Analytics."
My name is Michaela Benk.
I will be presenting this work today together with my coauthor Raphael Weibel.
Collaborative Immersive Analytics is an emerging research field
that has the potential to facilitate and transform collaborative work.
However,
there are a number of research challenges that the field is currently facing.
They include defining appropriate application scenarios,
designing systems that also integrate traditional tools and interfaces,
as well as assessing all the dimensions of collaborative work.
A promising application scenario
for collaborative immersive analytics is machine learning interpretability,
which refers to the development of methods to validate ML models,
explain their inner workings and their predictions.
Such methods include visualizations of model
outcomes or counterfactual explanations.
Existing
interpretability tools are usually displayed on 2D screens, comprised
multiple interactive functionalities using keyboard and mouse input
and involve single users.
It remains an open challenge how to design systems for industry
relevant and realistic CIA applications that focus on ML modeling and promote
collaboration among the members of interdisciplinary teams.
In our work, we tackle this gap
by introducing an AR based system for collaborative, immersive analytics
that is designed to support ML modeling and interdisciplinary teams.
We conducted a user study to examine how collaboration unfolds when users
with different professional backgrounds and levels of machine
learning knowledge interact and solving different machine learning tasks.
Finally, we make recommendations how immersive systems
should be designed to elicit sustained collaboration and ML modeling.
Next, we'll give an overview of the system as a whole and
then of the augmented reality application that we used for our user study.
Here you see an overview of our system.
In the center, you see two users, for example, a
data scientist and a risk officer.
Both wear a Microsoft HoloLens 2 on
which our augmented reality application is running.
The application allows both users to interact and manipulate
with the visualizations shown at the shared workspace.
Both HoloLens are connected to a backend server
such that the visualizations are synchronized across both HoloLens.
In addition to a shared workspace, each user as a personal workspace,
which is only visible to that specific user.
Additionally, each user also has their own
personal computer on which a web application is running.
The web application allows to share information
with participants through email or other web pages.
To give you an idea of what our augmented reality application looked like,
let us show you some of the possible interactions.
The first interaction
you see here is a user interacting with a data cloud by
setting the desired variables on the axis and selecting a specific data point
to receive more information.
Second, you see an interaction
with a counterfactual explanation of a selected data point.
Using the system, we set up a user study.
For our user study we used a realistic industry setting.
More specifically, participants were told that they work
in the credit lending department of a bank.
Each participant then took the role of either
a data scientist or a risk officer at this bank.
And based on their assigned roles, participants were
then asked to solve a number of tasks which allowed them to assess
the reputational risk of the algorithmic credit lending system.
As we have these two roles that assume a different background,
we also paired participants with two types of backgrounds
industry professionals and university students.
The industry professionals were asked to adopt the role of a risk officer.
The students were asked to adopt the role of a machine learning engineer.
Here an overview of the procedure.
Once everything was set up,
participants received instructions on their personal computers
to start exploring the data by themselves at their personal workspace.
After this first task, participants
were asked to collaborate for the remainder of the session.
Together, they were asked to further explore the data, build new models,
and validate those new models using our system.
During the user study, we had two main dimensions of measurement.
First, we evaluated the teamwork using the analytics method by Hernandez
and colleagues,
which is suited for collaborative settings with two roles
such as the ones we have in our study.
Based on video
and audio transcriptions of participants interaction,
we focused on three main variables coordination of attention
pauses in joint actions and navigation of joint actions.
These variables were then used to evaluate the teamwork
along six dimensions of collaboration. Namely, pair dynamics,
division and allocation of work, common ground and awarenes, references
and deixis, incentives and engagement, and consensus
and decision-making. In addition to the six dimensions
we also computed the times participants interacted with the visualizations.
The second main dimension was the task work,
which was measured using questionnaires after task two, three and four.
The questionnaires assessed
participants understanding and confidence in different models.
In addition to the two main dimensions, they also recorded participants
thoughts on the usability and user experience of our system.
Next, we present
selected results of our study.
Our findings confirm that pair analytics can be an effective method to elicit
collaboration for ML tasks in co-located, synchronous, immersive settings.
As mentioned previously, the use of two roles one subject matter
expert and one visual analytics expert are central to the method
putting participants
in the roles of risk manager and data scientist match those roles well.
The role play was intuitive and allowed users to share
their mental models without interruption.
We found that
clear role division promotes effectiveness in solving ML tasks.
This was shown by one participant taking on the role of managing
the navigation of ML modeling tasks and the other participant
taking on the role of driving and executing the tasks.
We also observed that the visualizations could be misleading.
For instance, in Task four, participants indicated to be more confident
in a new models performance,
even though the accuracy of the model remained the same as the original model.
Designers of CIA systems should thus explore ways
to encourage critical thinking to avoid such overconfidence.
Our results indicate that the use of multiple interfaces
does not impede collaborative efficiency when solving ML tasks.
Users could communicate and switch back and forth to different interfaces
easily.
In several sessions, participants
asked each other for consent before interacting with the visualization.
We found that such handovers are a dynamic of
successful collaborations.
Common
ground between individuals could be interrupted due to difficulties
such as uncertainty about the synchronicity of the visualizations
or the content of the tasks or difficulties in air tapping.
It is important to consider mechanisms that allow to reestablish common ground
when encountering such difficulties.
Overall participants were engaged and immersed
in the visualization, with the exception of a few technical challenges.
The different interfaces did not reduce participants engagement.
In fact, the use of immersive interfaces, along with traditional input tools
such as keyboard and mouse, may have supported the realism of the study.
Additionally, participants user experience
ratings were in an acceptable range.
To conclude collaboration
and machine learning modeling is characterized by several challenges.
They include the complex nature of ML tasks and the interdisciplinary approach
in real world ML applications, where domain experts
and ML experts with different skills have to collaborate.
Our CIA system is a first step towards solving these challenges.
It can elicit sustained collaboration among users during ML modeling.
We expect our findings to be especially relevant
for interdisciplinary teams and real world applications from industry.
As such CIA systems may help reduce the complexity
of ML modeling among the broader workforce.
Thank you for your attention.