Hi, I am Yunlong, from the NUS Ubicomp Lab.
On behalf of my co-authors, it’s my pleasure to present our paper —
Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation.
Crowdsourcing can easily collect many ideas from many people, for applications such as
creative story writing, graphic design, and ideating motivational messages.
However, with many ideators, several of them are bound to conceive identical or similar ideas.
This leads to the problem of redundancy in crowd ideation.
How can we reduce this redundancy and foster diversity?
Let's see with a concrete use case.
Many people lead sedentary lives, but they could be encouraged to be more physically active if they receive motivational messages.
We employ crowd ideation to obtain these motivational messages, by prompting each crowd worker with phrases to stimulate his creativity in writing the messages.
This is the method we proposed in our CHI ’21 paper last year, which is called Directed Diversity.
Since our current work is based on Directed Diversity, let’s briefly review it.
To direct ideators towards different directions to avoid redundancy, this approach includes three steps:
1. Curate prior ideations.
We show each ideation as a vector in the embedding space.
A larger distance between two vectors means the two ideations are more semantically different.
2. Prepare diverse prompts.
3. Send prompts to ideators.
However, in Directed Diversity, there’s no support for ideators to iteratively revise their ideations.
We believe that ideators could improve their ideations iteratively.
But without proper feedback, the iteration might be inefficient.
Therefore, we focus on feedback for iterative ideation in this work.
First, we provide ideation scores.
Second, we show explanations of the scores as actionable feedback.
Once the ideation is finished, it is added to the set of existing ideations, which will be used to compute feedback for the next ideation.
We call this approach Interpretable Directed Diversity.
Next, let’s walk through the iterative ideation process and see how we generate the feedback for ideators.
Imagine we have some prior ideations.
Now an ideator receives a prompt, “did not regularly exercise”, then produces a motivational message to encourage exercise.
Without any feedback, the ideator could only revise the ideation intuitively.
Here we can see the ideator changes the wording between attempts.
To provide the ideator a clear goal, we first predict the ideation score, informing if the revision is better or not.
We have scores for both ideation diversity and quality in our system, but only show the diversity score in this presentation for demonstration.
With only scores, ideators may still have no idea how to increase the score.
Here come our designed explanations based on XAI as more actionable feedback.
The first one is Attribution Explanation.
The attribution of each word in a message is the change of the ideation score when ablating that word.
In this case, word C is more influential for the ideation score as it has a higher attribution than A and D.
Overall, the sum of attribution scores eques to the ideation score.
In the feedback UI, we highlight the top-3 words that contribute the least to the ideation score, which should be prioritized for revision.
Second, we provide Contrastive Attribution Explanation in terms of the score change between iterations, which further explains why the iteration is better or not.
In this example, word B and C are deleted, and M is inserted.
We use the ablation method again to calculate the attribution score of each edited word.
Here we can see clearly which edit is good or bad in the feedback UI.
Third, we provide Counterfactual Suggestions to replace certain words in the ideation.
Word C is problematic in this message.
So, we search related words from the ConceptNet knowledge graph as alternatives to replace C.
In this example, we see three suggestions with the potential effect on the ideation score when replacing the target word.
We expect ideators to get inspiration from the suggestions.
Now let’s see how we evaluated these different types of feedback.
We first conducted a formative ideation study to investigate how users interpret and use the feedback, and identify any usability issues.
Then we conducted a summative ideation study to collect ideations under different feedback conditions.
We focus on the ideation quality, diversity, and the ease of use of the feedback.
For Ideation Quality, in addition to the embedding-based quality score, we collected Individual Quality Ratings with a validation study.
For Ideation Diversity, we applied Computational Diversity Metrics based on ideation embeddings and Pairwise Dissimilarity Ratings from another validation study.
Do explanations improve Ideation Quality?
No. Although providing scores already improved the ideation quality, the extra explanations did not enhance the effect.
Do explanations improve Ideation Diversity?
Yes. The results of embedding-based diversity metrics consistently suggested explanations improved ideation diversity.
For instance, this figure shows how much the new ideations are different from the prior ones in the embedding space.
Feedback conditions with two or more explanation types significantly outperformed other conditions.
This figure shows validators’ ratings on message-pair dissimilarity.
The highlighted results correspond to the conditions with explanations in feedback, which are significantly better than the conditions of no feedback or feedback with only prediction scores.
In summary, we described Attribution, Contrastive Attribution, and Counterfactual Suggestion Explanations as feedback
to direct crowd ideators to iteratively generate more collectively creative ideas.
We call this approach Interpretable Directed Diversity and evaluated this approach in the crowd ideation task of writing motivational messages to encourage physical activity.
Our approach opens opportunities for XAI towards scalable and rich feedback for iterative crowd ideation in the field of creativity support tools.
Please read our paper for much more details of our approach and experiments. Thank you.