Understanding visual
investigation patterns through digital field observations.
The amount of data being generated by businesses today
has grown to a point where video analysis is no longer feasible.
Many users
are therefore turning
to a growing variety of visual analytics
products to help them explore and derive value from their data.
To identify ways of improving these products.
We need to have an understanding of what users
are trying to do, their objectives, and how they use visual
analytics tools to achieve their goals.
However, the variety of data sets users operate on
and the breadth of functionality
these tools provide can make it
difficult to reason about user goals and actions across domains.
For example, do improvements that help a university researcher
find repeating patterns in genetic sequences
also translate
to helping a business analyst predict customer acquisitions?
A large body of past research has grappled with this problem
by developing conceptual spaces, frameworks, models, typologies,
and taxonomies to categorize user objectives and actions at various
levels of abstraction.
However, most
of this prior work has been subject to several limitations.
First, a majority use synthetic or non domain data sets
that participants had no prior expertize or familiarity with.
These studies have also been with data of a limited size,
on the order of tens
to hundreds of megabytes, substantially smaller than what is
encountered in large scale business or enterprise settings today.
Because many of these studies have been with experimental tooling
or on noncommercial systems,
participants have usually needed training
with the product prior to or during the study,
which may have affected their strategies and behaviors.
Last, these studies have generally been performed in isolated
and controlled environments.
However, in enterprise
and business settings, users generally work in a larger ecosystem
where they can leverage other tools, resources or people.
They often work with disparate data sets
that are inconsistently available between tools,
and they usually have
multiple responsibilities that reduce their ability
to conduct open-ended explorations without a motivating goal or problem.
These limitations lead us to the following research question
How does our current understanding of user objectives, actions,
and how to support them generalize to an enterprise setting
where a larger ecosystem of people and products might be utilized?
To answer this question, we conducted two studies.
The first was a digital field observation
where we remotely observed people
doing their day to day work on PowerDrill,
a visual analytics product, via video conferencing.
We then analyzed usage logs of a different visual analytics product,
which will call Data Explorer, of over 2500 user sessions to test
whether the findings from the first study were product specific.
Next, I'll briefly explain
our method, followed by how we analyzed
the data and our results in the form of objectives, patterns
and emergent themes.
We conducted 90 minute sessions with a 15 minute interview
at the beginning to understand their objectives and 15 minute
retrospective interview at the end.
We had nine full time Google employees
who regularly used PowerDrill
for at least 90 minutes a week participate in the study with five
using it for more than 20 hours a week.
These participants came from eight different teams performing
visual analyzes on domains ranging from product production
code to sales and accounts to travel to A-B experiments.
The data they were working with ranged in size
from tens of megabytes to hundreds of terabytes with schemas that went
from tens to thousands of fields.
Next, I'll share a brief overview of how we analyzed our data
and highlights of our results.
If you'd like more detail, we encourage you to read the paper.
For objectives, we transcribed
what people described working on and paraphrased it
to various levels of abstraction to find commonalities.
Using this technique, we identified four common objectives
across the domains our participants worked in.
Some of these mapped directly to prior work.
For example, investigating
where and why data was not what was expected was very similar
to the previously identified goal of identifying a main cause,
whereas determining if data signals match known patterns
had not been previously identified.
We found participants only did this when they had extremely specialized
domain knowledge and when their objective in analysis
something they had performed repeatedly,
something you may not observe in a controlled lab setting.
Next, let's take a look at how users achieve these objectives.
To determine this, we manually created sequence models
of all the keystrokes, mouse clicks, and other atomic actions.
We then grouped these into 263 codes that fell into 20 categories.
Examples of these
categories might be visual analytics actions, such as filtering or zoom.
or use of other tools like opening an email
or looking up a person in the company directory.
We then use these categories to visualize
each investigation, thenlooked for common patterns.
From these visualizations, we identified
six patterns of investigation.
We found three that supported patterns found in prior work.
The first was the Drilldown, a series of progressive filtering,
restricting or excluding data to narrow the investigation space.
The second was what we called the Sawtooth, a repeated adding
or removal of
either the same filter or progressing down a list of filters.
The third was the side by side
comparison, which is the same as the compare pattern found in prior work
We also found three patterns
that had not previously been seen in prior work.
The first is the sidestep, where after drilling down,
a user would remove almost all of their prior filters
except for the most recent to see if data patterns
look the same, with only the last filters applied.
The next was the fan out, when a participant would open
multiple new branches of investigation
simultaneously, often by creating new tabs.
They would later iterate through each tab, often performing
the same analytic steps
in each before closing branches that weren't promising.
And finally, the plug and play, which illustrates when participants
loaded a previous investigation, either their own or someone else's
from a template to save time,
where the visualizations needed for the investigation
were largely already there,
and they just needed to change out a variable.
Next, I'll highlight
five unique themes that we identified related to the way participants
collaborated and interacted with their larger product ecosystem.
Unlike many prior studies that occurred in isolation,
we were able to observe that a majority of analyses
actually leveraged prior work, both as a starting point
and as a reference during investigations.
We found that real time collaboration was a crucial part
of some investigations,
with three participants
pausing or switching to a parallel work stream
while waiting for help from someone else.
We also found that five of the investigations involved
direct review of another person's
analysis to verify that they would reach the same conclusions
As far as ecosystem themes, all of our participants
described an intention to act on their findings or perform
follow up actions during the session, highlighting
that Insight generation was not an objective in and of itself.
We also found that
products outside of the visual analytics tool were crucial.
We observed that participants leveraged 53 products
outside of PowerDrill to do things like orient themselves
to their investigation, document
and share findings, or look up and ask others about business logic.
Next Feng will talk about how we analyzed logs from Data Explorer,
a different visual analytic product, to validate our observed patterns.
Thanks, to validate how frequently patterns were used
in each investigation, we conducted a study with over
2500 user sessions from Googlers and used an algorithm
named closed FAST to mine frequent patterns.
Let's look at the findings.
We used heuristic rules to identify patterns.
For example, for drilldown we checked to see
if there were repeated actions and the figure shows that we had
a 12 actions of filtering with support of 1209.
We also found other four patterns by the algorithm, including plug
and play, sawtooth, side-by-side, and fanout.
Sidestep was manually identified as its support was low.
As a summary, we found that five of the six patterns
were validated, while sidestep was manually identified.
We described a field observation of nine Google employees
from eight different teams doing their analysis in PowerDrill
to identify four objectives.
six recurring patterns and five emergent themes.
We also described an analysis of data explorer logs to validate
that the patterns we found weren't product specific.
Our work had several limitations.
The products and participants
were all within the Google ecosystem, so whether the results
are generalizable to other ecosystems or populations is an open question.
Our manual sequence modeling may have led to some inaccuracies,
and some details
of the study were proprietary and had to be omitted from this work.
However, we believe our findings may be applied to future efforts
to predict user behaviors or guide users to their objectives
more quickly.
They also suggest the need to consider
how real world analysis behaviors within a larger ecosystem
may differ from what's observed in isolated study conditions.
With that, we'd like to thank you for listening
and thank you for your attention.