We present an interview study on how domain experts create conceptual diagrams
and its implications for diagramming tool design
I'm Nimo, from Carnegie Mellon University
Conceptual diagrams, also known as explanatory diagrams, concept visualization
describe a class of diagrams that provide a high-level, graphical overview of a certain conceptual model
whether it’d be the structure of an LSTM network
how an encryption algorithm works
or, how to order things at a local burger joint
People use them to explain and communicate concepts efficiently in all kinds of media
on web pages, in academic papers, in books, and on YouTube channels.
Conceptual diagrams tend to be abstract and topological
in that, the physical placement of shapes and labels don’t matter as much
as long as it illustrates the same concept
Consider a graph showing how some nodes connect
the change the node positions doesn’t necessarily change the meaning of the diagram
The relationships are topological.
In addition to these nice properties, people also found that diagrams improve students’ learning and make teaching easier
They are positively correlated with the impact of scientific publications
And, let’s be honest, nice diagrams are a pleasure to look at.
However, much is unknown about how people create these pretty diagrams.
One reasonable guess is that people start with some informal sketches, and later translate them to better looking, final diagrams.
But how do these diagrams evolve from sketches to their digital forms?
At what point does the diagrammer decide to switch tools?
How do diagrammers manage the evolution of diagrams in general?
What tool, if not multiple tools, are chosen? For what purposes?
Are there any barriers when using them?
What kinds of affordances do they provide?
We had a lot of questions.
So we decided to conduct semi-structured interviews with 18 diagrammers from 12 different domains
ranging from architects, to K-12 educators, to mathematicians.
Following the interviews, we performed thematic analysis on the verbal data
to extract high-level themes as well as insights to designing the next generation diagramming tools
We grouped the codes from our thematic analysis into 3 high-level themes.
The first one I’ll introduce is representation finding.
By representation, we are referring to visual representation, how abstract information is represented pictorially.
To illustrate this concept, let me show you an example from a seminal paper on diagramming by Jill Larkin and Herb Simon.
Consider a truck move rightward on the screen
and think about what forces are exerted onto cart B,
and how you would visualize it.
A standard representation is free body diagrams
where the truck becomes a mass point, and forces are arrows with magnitudes.
You can also represent them as circles
or... if you are hungry, chocolate bars.
But there are problems with the last two.
The circle diagram loses the directionality of forces, and the chocolate bars might be too weird for everyone to understand.
The main message is:
appropriate representations distill complex information by boiling them down to the essence
and not all are equally effective in different contexts.
Since representations are so important, our participants reported the process of representation finding
by seeking existing standards in their domains.
For instance, in computer graphics, the view angle, or the camera, in rendering diagrams tend to be represented as an eye icon
and the light source as a little sun.
In the absense of existing standards, some diagrammers create their own ones
like these diagrams showing the structure of certain mathematical groups created by one of our participants
In this process of finding visual representations
sketching using pen and paper becomes an important tool for exploring design alternatives and evolving their visual representations
like how one participant evolved the design of a projecter callibration system
by inventing differents kinds of arrows to denote more complex configurations
The representation finding process is creative and rewarding. Like P1 stated:
Why didn’t anybody draw that picture before?
I have been taking something that almost seems completely confusing or unimportant
and having a picture that makes you know what’s going on... is truly satisfying.
Next, we’ll talk about the tools that our participants use and why they made their choices.
In general, the diagramming tools they use fall into one of two categories
differentiated by their dominant mode of interaction.
Direct manipulation, or WYSIWYG, tools have graphical interfaces that allow you to directly create and manipulate shapes.
The programming language based tools have textual interfaces
and often involves typing snippets of code into the editor and compiling the program to view the diagram.
Examples of language based tools include D3, TikZ in TeX, and matplotlib in Python.
For direct manipulation tools, our participants mentioned tools like Adobe illustrator and Figma.
For programming languages, most of their users reported better support for abstraction and automation of workflow.
They also suggested that textual inputs gave them better control locally
like how to precisely place a shape
Common drawbacks of PL tools include a steeper learning curve
and a higher upfront cost for constructing diagrams
due to the compile-and-run cycles required and the complexity of languages.
For direct manipulation tools, our participants liked their faster feedback cycles
and the ability to see and control how shapes relate to each other globally
The cost of this is that the drawing process tends to be highly manual
and the diagramming interface can be very viscous
In other words, once drawn, it’s tedious to make a change
because many other shapes need to be manipulated manually to accomodate the new change.
The pros and cons of these two kinds of diagramming tools lead to interesting patterns of automation and reuse.
We saw from some partcipants an interesting pattern of backtracking.
One example is by literally keep the whole history of the diagram as separate artboards in illustrator
not just linearly, but also with branches.
Other diagrammers maintain the consistency across their work by keeping a text file with low-level parameters
like color codes and line weights because their tools don’t always remember them.
To keep track of their visual representations, some participants also curate a library of many diagrams
such that they can build on top of those examples when they create similar diagrams.
In addition to learning a great deal about how people diagram and appreciating their beautiful work
we also saw some gaps in their diagramming workflows
Our participants search, adopt, create, and iterate on representations outside their digital diagramming tools
but prefer lower-stake medium like pen and paper.
This is in part due to the limitations of both direct manipulation and language based tools.
For instance, languages are reusable and powerful but require more time and effort.
On the other hand, GUI tools are flexible but require you to commit to a representation upfront
without the ability to change them easily.
All of the above lead to various ad hoc workarounds to maintain and reuse diagram elements
without strong support from the tools themselves.
The net result is that diagrammers are often sadly deviated from the creative core of diagramming
because their tools are limited.
Can we fix this?
How can we design a tool that provides a more natural experience just like pen and paper
but with the additional benefits of the power of computing?
Inspired by these observations and extensive review of related literature both in and outside of HCI
we present some opportunities to guide future designs of diagramming tools.
We collectively call them a natural diagramming experience.
We adopt the term “natural” from natural programming
meaning that the diagrammer can express their ideas in the same way they think about them.
We describe natural diagramming in four dimensions in our paper
but In this video, we will only go into two of them here.
but I invite you to read our paper for more content.
Given the importance of representation finding and reuse in the results
we argue that a natural diagramming tool should treat representations as first class entities
potentially by explicitly encoding the concepts being drawn, and the mappings from concepts to concrete geometries.
Just like how cameras are represented as eyes
or functions as curves or flows between axes
Visual grammars have been very successful in the domain of data visualization.
Many popular datavis tools like D3 and Vega-Lite build on a visual grammar called the grammar of graphics
which lets people encode, transform, and map data attributes to visual properties.
Other systems like Penrose propose grammars for domains outside of dataviz.
For math diagramming, Penrose lets users directly write down abstract concepts as a declarative program
and the visual mappings as another [program].
Different from Vegalite, Penrose maps math notations to constraints instead of data.
This allows the system to generate a whole space of possible diagrams with the same meaning, or semantics.
The power of these systems come from a salient encoding of visual representations
which opens the possibility for reusable representations for different contents
and therefore faster, or even automated exploration of design alternatives
Penrose and Vega-Lite encode representations by introducing a higher level of abstractions, but it comes with a cost.
As suggested by our participants, the sense of agency or engagement is deeply valued for diagramming
as well as the ability to have immediate visual feedback
The extra layer of indirection introduced in Penrose and Vega-Lite
prevents users from freely editing the diagrams and in part take away the sense of engagement in the diagramming process.
Efforts to bring programmability to direct manipulation and liveness and engagement to programming like prodirect manipulation attempt to bridge this gap.
Prodirect systems attempt to propagate changes between textual and graphical views bidirectionally.
Closely related, live programming is a new programming paradigm where the output updates in real time as the programmer interact with the program itself
often in novel ways that go beyond traditional GUI or textual interactions.
Similarly, programmable brushes such as dynamic brushes allow users to program the behavior of brush strokes
for procedural and generative arts.
These techniques, although currently limited to their domains
provide promising directions towards bridging the gap between programming and direct manipulation for conceptual diagramming tools.
For more discussion on related work and other natural diagramming opportunities
Please refer to our paper by following the link
Thank you for your attention