Hello, my name is Parzival Borlinghaus and together with Stephan Huber I am presenting
our explorative work on the automation of affinity diagrams.
Affinity Diagrams are a method commonly used in user research to cluster qualitative data.
Observations and user statements are written on individual notes and hierarchically clustered
bottom-up.
The resulting diagram builds the „Bridge to design“ as Holtzblatt and Beyer put it
– and allow to derive insights about user needs as well as inspire design ideas.
However, manually creating the affinity diagram is extremely time consuming and can keep a
team busy over several days.
So our research question was how we can make affinity diagramming more efficient, without
losing its benefits for the design process (that is allowing the designer to dive into
the data and learn about the users‘ needs)?
In this work, we restrict clustering to the lowest level of the hierarchy, where the clustering
process starts and most time is spent.
Higher level headings should be generated by designers themselves or be addressed in
future work.
I’ll use the next couple of slides to introduce you the data we used and the models that we
took into account.
Affinity diagrams often consist of hundreds, sometimes thousands of affinity notes.
Affinity notes share some simplifying characteristics such as their similar length, acceptable spelling
and rare usage of negation.
Affinity notes are written in story language and from the users point of view.
Exemplaric notes from a bycicle project are “I am annoyed when the police stop me without
a reason” or “I like to ride through the park, because there is not much going on and
I am faster.”
Since this is only a ten minute talk, I cannot go too much into detail.
However, I refer to the paper where all models are briefly explained.
The models were compiled with diversity in mind.
Some of the models utilize modern context-predicting approaches, some are rather mathematical and
are based on the decomposition of word-count matrices.
Later models must be trained and make use of training data such as the wikipedia corpus.
All of them can be used to find document embeddings in the semantic vector space.
That is a space where similar documents are represented by similar vectors.
Please note that similarity can be measured in multiple ways.
Due to the high dimensionality (300 and more are common) the usage of cosine similarity
is widespread.
The most suitable model was determined by a human similarity task.
Ten native speakers labeled about 300 sentence pairs from an open-ended survey for relatedness
in meaning.
A model that is suitable to auto-generate affinity diagrams should be able to reflect
the human sense of semantic relatedness.
The degree of similarity was measured with Spearman’s rank correlation.
fastText turned out to be the best performing model with a rank correlation of 0.44.
FastText was published by Bojanowski and colleagues and falls in the class of context predicting
semantic models.
As the name suggests, it’s fast to train since it processes sub-words.
Sub-words model the internal structure of words like prefixes and suffixes which is
beneficial to morphologically rich languages like German.
Bojanowski and colleagues chose all subwords of size three to six that occur in the corpus.
Another beneficial side effect of sub-words is the unlikeliness of out-of-vocabulary errors.
The sub-words’ embeddings are subsequently summed and form the document’s embedding.
However, fastText is not capable of word-sense disambiguation.
That is, fastText is unable to distinguish between words of the same spelling but different
meanings.
This slide shows the performance of fastText in the wild.
Applying fastText to six affinity diagrams and comparing the results to their human generated
counterparts lead to promising results.
The clusters were compared using overlap and jaccard indices.
This resulted in an average overlap index of about 0.7 and an avagered jaccard index
of 0.3.
For comparison: Two random affinity diagrams with 250 notes and 69 clusters result in a
Jaccard index of less than 0.008.
We can conclude that fastText is in principle a good choice for the automation of affinity
diagrams.
We therefore conducted a study to determine the helpfulness of fastText pre-clustered
affinity diagrams in real-world applications.
We set up an evaluation study to test, whether algorithmically supported affinity diagramming
is more efficient without losing its value for the design process.
In the first part, we focused on efficiency, and hypothesized that design teams working
with pre-clustered affinity notes would be faster in building an affinity diagram than
design teams with randomized affinity notes In the second part, we interviewed the design
teams, while they explored and evaluated a fastText generated diagram of notes they were
familiar with.
Finally, experts rated the quality of diagrams that design teams had clustered in the first
part of the study, compared to a reference version created by a design team without any
time limit, and a reference version entirely clustered by fastText.
Opposing our expectations, the group working with random notes progressed faster without
a significant difference in workload.
A reason for this may be, that the teams in the fastText condition spent much of their
time discussing if the suggested groups were a good match or not, while the teams in the
random condition just started from scratch.
Key insights from the second part of our study were, that participants suspected a change
of task that came with the algorithmic support.
Instead of „thinking with their hands while manually clustering data“ they were afraid
they had to merely correct the already pre-clustered fastText output of a seemingly ready Affinity
Diagram with too much information at once.
They saw a potential benefit when working with large amounts of data, especially if
they were familiar with the data already.
Experts graded the reference version of the affinity diagram best and the entirely fast
text generated affinity diagram worst.
They judged the fastText clustered version as seemingly following a „mere sorting by
similar words with a lack of semantic“.
At the end of the study we disclosed the origin of the diagrams to the experts.
One expert remarked that they had experienced similar quality from student design teams
before, and would have expected from the computer to do better.
Summarizing our contributions, we examined seven textmining models and found fastText
to be most appropriate for pre-clustering short qualitative notes.
Even though there was no measurable objective benefit, algorithmic support has much potential
and the concept was generally accepted by participants in our studies.
For future work we recommend to shift the point of algorithmic support in the process,
so that designers do not have to rearrange pre-clustered data but are supported as they
cluster the data themselves.
This could be achieved for example by highlighting notes that could match the content of a cluster
the designer is momentarily working on.
Conveying confidence of suggestions would be a great challenge here to promote the designers‘
decision making process and acceptance of the algorithmic support.
Thank you very much for your attention in this short overview.
You will find more details in the paper.
Now we are happy to take your questions!