Hi, I am Yunlong Wang, from the NUS Ubicomp Lab.
On behalf of my co-authors, it is my pleasure to present our paper,
Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation.
Crowdsourcing can easily collect many ideas from many people, for applications such as
creative story writing, graphic design, and ideating motivational messages.
The goal is for each ideator to conceive one or more creative ideas.
However, with many ideators, several of them are bound to conceive identical or similar ideas.
This leads to the problem of redundancy in crowd ideation.
How can we reduce this redundancy and foster diversity?
Let's see with a concrete use case.
Many people lead sedentary lifestyles.
But they could be encouraged to be more physically active if they receive motivational messages.
We employ crowd ideation to obtain these motivational messages,
by prompting each crowd worker with phrases to stimulate his creativity in writing the messages.
The key to obtaining diverse messages is to select diverse prompts to provide different ones for different workers.
In our work, we propose an automatic diverse prompt selection method to do this: Directed Diversity.
Directed Diversity involves three steps.
First, we extract phrases from a corpus of documents, such as online articles and discussion forums.
Second, we obtain a vector representation or embedding of each phrase by applying the Universal Sentence Encoder, a deep learning language model.
This allows us to calculate diversity metrics and select phrases based on their vector distances.
Third, we select the diverse subset of phrases using our proposed selection mechanism described next.
For Phrase Selection, we developed a diversity maximization algorithm to
select diverse phrase embeddings in the language vector space. It involves four steps.
Having mapped each phrase to an embedding vector,
we build a minimum spanning tree using agglomerative hierarchical clustering with single-linkage,
and cluster the points based on the number of prompts specified.
For each cluster, we select the point with largest minimum pairwise distance.
These points are the most diverse prompts.
This method for Directed Diversity phrase selection is the first contribution of our work.
To evaluate if and how well Directed Diversity works, we conducted several simulation and user studies.
We were interested in three research questions.
How does Prompt Selection affect Computed Diversity of Prompts & Ideas?
How does Prompt Diversity affect Ideation Effort for ideators?
How does Prompt Selection affect Perceived and Thematic Diversity of Ideations?
For all research questions, we compared prompt selection techniques as the main independent variable:
None, Randomly selected prompts, and Directed Diversity selected prompts.
To answer different research questions, we identified various metrics and methods,
and formalize them with the Diversity Prompting Evaluation Framework.
This is the second contribution of our work.
This Framework is anchored by an Ideation Chain which we define with four steps:
the prompt selection technique will influence the creativity of selected prompts,
the ideation process of the ideators, and the creativity of their ideation.
On manipulating the prompt independent variables, we measure various metrics
to track how well ideators are prompted to generate creative ideas, namely,
embedding-based distance and diversity metrics,
ideator ratings about their ideation effort,
validator ratings about ideation diversity and quality,
and diversity measures from thematic analyses.
We applied this framework in a simulation study and user studies with 540 participants from Amazon Mechanical Turk.
So, back to our research question. Does Directed Diversity improve diversity for ideating motivational messages?
Yes. Directed Diversity increased the minimum pairwise distance between ideations compared to Random and None prompts.
They are also most evenly spaced.
Ideators rated its prompts as more unexpected.
Validators rated its ideations as more unrepetitive and more different.
And a thematic analysis found more original ideas from Directed Diversity.
In summary, we presented Directed Diversity to mitigate redundancy in crowd ideation.
It is a corpus-driven, automatic approach that leverages embedding distances based on a language model to select diverse phrases.
We further proposed the Diversity Prompting Evaluation Framework to evaluate the efficacy of diversity prompting along an ideation chain.
Applying the evaluation framework in a series of experiments,
we found that Directed Diversity improves ideation diversity across multiple measures.
Please read our paper for much more details of our approach and experiments.