Hi, I'm Carolina Nobre.
This paper is work I did with Dylan Wooton, Lane Harrison and Alexander Lex while I was
a student at the University of Utah.
Today I will talk about Evaluating Interactive Multivariate Network Visualization techniques
using a validated design and crowdsourcing approach.
This work has two orthogonal contributions.
First, we ran an evaluation on the pros and cons of interactive node link diagrams and
adjacency matrices for visualizing multivariate networks.
Second, we introduce a methodology for evaluating complex, interactive techniques designed for
experts in a crowdsourced environment.
So let�s get started.
Let�s start off by defining multivariate networks.
When we think about networks or graphs, we often think of its structure, or how things
are connected.
In the case of a social network, nodes represent people, and edges connect people who have
some relationship to each other However, real world networks almost always have attributes
associated to the nodes and edges.
So in this case, we might have information on each person such as their name, their age
and what they do for work.
We can also have attributes on the edge, such as how to people know each other or how long
they�ve known each other for.
So we can define a multivariate network as one where we have information about the structure
of the network, or it�s topology, as well as attributes of the nodes and edges.
Our study compares the two most common forms of visualizing multivariate networks: the
node-link diagram and the adjacency matrix whose interactive design you can see here.
There are a series of works that compare node-link and adjacency matrices, starting with a study
in 2005, and a few follow up studies that both replicate and extend on the tasks/network
types.
However, these studies only covers topology tasks, and at most encode a topology derived
attribute (clusters) as color on the node/node labels.
What distinguishes our work here is that this is the first study to incorporate attributes
(both in the representation and in the tasks) when comparing these two graph representations.
What this study aims to answer, is which of these techniques is better for performing
different tasks with multivariate networks?
In order to address this, we must first ask, can we even use empirical studies to evaluate
these complex visualizations?
If we consider the spectrum of visualizations that are amenable to empirical studies, it
ranges from very controlled perceptual studies all the way to complex visualization systems.
In the middle, we have studies using static visualization or controlled interactions.
While the first three types are often used in empirical studies, complex visualization
systems are much more rare due to a series of challenges.
The first and perhaps biggest challenge is how can we make sure that what we're testing
is what we care about?
When a participant does poorly on a task, was it because we didn't select the right
colors?
Was it because we didn�t use optimum coding for the task?
Or is it because that technique inherently does not support that task?
To address this, we developed our designs by leveraging existing guidelines and knowledge
on what the best encodings are for our tasks.
For tasks where we only need to show two attributes, we use size for the quantitative attribute
and color for the categorical attribute.
For more attributes, we use nested bar charts, such as this one.
the bars represent quantitative attributes, and small glyphs are used for categorical
ones
Here�s what a small network looks like with the nested bars encoding.
We also consider edge attributes, where we use color to show the type of an edge, and
width to show a quantitative attribute, such as how often that interaction has happened.
Each visualization affords a certain set of interactions such as dragging nodes or highlighting
neighbors.
Searching for a node highlights the node in the graph.
Users can toggle the highlight on and off by clicking on that node again.
Highlighting a node also highlights all of its neighbors.
User can also drag nodes around the screen.
Hovering over a node shows a tooltip with exact values for the encoded attributes.
Let�s move on to the adjacency matrix, which represents each node as both a row and column
in a matrix.
An edge is a filled in cell at the intersection of the source and target node.
We use opacity to show quantitative edge attributes such as count , and color indicates the type
of the edge.
Quantitative node attributes are represented as bars in an adjacent table.
We use colored binary squares for categorical node attributes.
In the adjacency matrix, a node can be unselected by clicking on the intersection of its row
and column.
To highlight neighbors, users can simply click on the node column label.
They can further group all neighbors, by clicking on the sort icon under the column label.
We then move on to attributes in the table, including the ability to sort them by clicking
on the column headers.
We can also sort the network by clusters, which reveals the structure in the matrix.
Users can hover on an edge, and click to select it.
Another approach we use to address the problem of confounders is validating our designs using
external experts.
We use heuristic evaluations with 69 questions asking for design suggestions and criticism,
which we implemented in our final design.
Another challenge with using crowdsourced empirical studies for complex interactive
visualization is that we are using novice users, who are not graph visualization experts.
So the question is, can we train them enough to use these complex systems so that we can
generalize the results to people who actually use multivariate network visualizations?
What we found was a combination of both active and passive training can in fact give users
the expertise necessary to complete these tasks.
The average accuracy in this study was over 75%, with users who had never seen these multivariate
graph representations before.
The study had 4 sections and took participants, on average, 40 minutes to complete.
The training component had both a passive and an active section.
In passive training, a video introduced users to the network, and to the visualization approach
they had been assigned.
In this example, the video is teaching the user what an adjacency matrix is.
After the video, users engage in active training with a hands on guided tour to learn how to
interact with the visualization.
Before starting the study, users had to answer two trial questions correctly.
In order to understand the effect of extensive training for novice users, we ran multiple
pilots and tracked detailed provenance.
This allowed us to see things like How many incorrect answers are submitted before getting
the trials right?
Are they using all of the relevant interactions?
Here we have an example of one of the participants where we can see how long they took on the
training component and their timing throughout the study.
We also had to consider how to best motivate our users to complete a study that took almost
an hour.
We did this by making tasks that applied to a relatable twitter network, and offering
above average compensation, more than double what the average crowdsourced study usually
pays.
As a results, we recruited all participants for our study in just under 2hrs.
Participants also reported that they found the study interesting compared to what they
usually encounter.
We did this by making tasks that applied to a relatable twitter network, and offering
above average compensation, more than double what the average crowdsourced study usually
pays.
As a results, we recruited all participants for our study in just under 2hrs.
Participants also reported that they found the study interesting compared to what they
usually encounter.
Let�s take a look at how we designed the study itself.
To run the study, we use a crowdsourcing platform called Prolific, on which we recruited 300
participants, 150 for each condition.
We applied our study tasks to a twitter network, which contains tweets during EuroVis 2019.
We subsample the original network to create a large variant with 75 nodes, and a smaller
network of 25.
The study itself had 16 tasks, the first 15 of which were assigned in random order.
The last task, shown here, was an open ended exploration task where we asked users if there
was anything interesting or particularly surprising about this network.
The study had a between subjects-design, so half of our participants saw the adjacency
matrix, and the other half saw the node-link diagram.
We designed our tasks to explore several different hypothesis, such as the expectation that distractors,
which are node attributes that are visualized but that are not necessary to the task. will
have a negative effect in terms of performance and time in the NL.
In the AM however, we expected accuracy and time to not be influenced by the number of
distractors.
For each task, we collected information on participants accuracy, speed, as well as their
confidence in their answer and the perceived difficulty of the task.
We analyzed the results using bootstrapped confidence intervals and effects sizes, and
also performed qualitative coding to extract insight types for each visualization.
Our results both confirm previous work on studies that considered only the topology
of the network, as well as found new differences that related specifically to attributes.
For example, similar to previous work, our results show that NL diagrams still outperform
AM for path based tasks.
For attribute specific findings, we saw that when we included distractors, or attributes
that were not essential to the task, the AM performed better than the NL diagram.
Both in accuracy as well as in time.
We also used our study to better understand the type of insights that each visualization
naturally supports.
The most common type of insight for the AM were attribute overviews.
These are those that, as the name implies, give a summary of one or more attributes of
the entire network.
As an example, one participant noted that "Institutions have much fewer tweets in general
than person accounts�
The second most common insight type for the AM were ranked attributes.
In these insights, the ranking of one or more attributes is specifically mentioned, such
as this participant who noted that one user has the 2 youngest account age and very few
tweets, but still had the 4th largest follower count of the network.
Insights for the node-link diagram were mostly topology driven, meaning that they related
to the structure of the network but did not mention any attributes.
In this example, the user notices how several people only have on connection in the network
and they are all with the same person.
The second most common node-link insight type includes information on the topology of the
network as well as at least one attribute.
This participant mentions that they are surprised by the low attribute value of tweets, considering
how well connected certain people are in this network.
Another common insight type with node-link diagrams are those the relate to comparisons
of attributes of a single node.
This participant notes that despite few tweets, this person had several followers and a long
account age.
The study also has a few limitations.
First off, our comparison of two complex systems makes it impossible
to identify individual factors that contribute to the performance
of the techniques.
For example, we do not know whether it is interactive sorting or the encoding in aligned
bars that leads to attribute-related insights in the AM.
Another limitation is that we were not able to compare multiple different network types
in terms of size and density.
Instead, we focused on testing a broad set of tasks on networks with varying attribute
configurations.
As a results, we cannot make claims as to whether our results would
generalize to networks with significantly different topological characteristics.
To recap, this study makes two main contributions.
We evaluate the performance of the two most common types of MVNV techniques, and find
that the AM outperforms the NL for clusters, as well as where non-task essential attributes
were encoded.
Node-link diagrams however are best suited for paths and tasks that rely on within node
attribute comparisons.
We also contribute a new methodology for evaluating interactive techniques, and highlight the
importance of active training for novice users, as well as proper compensation to ensure that
user are motivated to properly complete longer studies.
The view of the visualization community on which kinds of studies can be run on a crowdsourcing
platform might be too narrow, and that a broader range of systems is amenable to crowdsourced
experiments.
Thank you for watching this talk and we hope you�ll check out the paper, available along
with all the supplementary material at this url.