Hello everyone, I'm Mina.
And today, I'm excited to present our recent work, CoAuthor: Designing a Human-AI Collaborative
Writing Dataset for Exploring Language Model Capabilities.
First of all, why are we interested in language models?
Because we use them every day.
When you search on the Internet or write an email, notice that systems are actively trying
to predict what you are going to type next.
These technologies are powered by language models, which predict the next word given
a context.
Going beyond autocomplete systems, we hear more and more anecdotes of people using language
models as writing assistants.
For instance, in 2018, one writer created an autocomplete system with a language model
to help him write his novel.
Then, language models got better and better over time.
This year, another writer wrote a book with a language model in 24 hours and published
it, while enlisting the language model as a coauthor.
Now, there is a list of books officially written with language models.
So, I think it is fair to say that we’ve seen enough evidence that language models
have potential to help our writing process.
However, it is difficult to understand these language model capabilities in a holistic
manner, for at least two reasons.
First, language models have highly context-dependent capabilities.
That is, depending on the types of writing, topics, and writers, a model may exhibit a
varying degree of performance.
Second, even if we look at one specific piece of text, it can be subjectively interpreted.
There are many viable definitions of “good” collaboration.
Some people may want creative, new ideas, whereas others may want more coherent, predictable
autocompletions from the system.
Based on these different goals, the assessment of the same text could vary.
These challenges led us to ask this question: How can we better understand language model
capabilities in a holistic manner?
Our approach is to curate and analyze large interaction datasets.
Here, we propose four desiderata for designing large interaction datasets.
Then, exemplifying this approach, we design a dataset based on the desiderata.
Here is how we designed CoAuthor.
First of all, CoAuthor covers diverse contexts, including multiple types of writing, writing
prompts, and writers.
Second, CoAuthor provides various measurements for designers to analyze writing outcome,
writer perception, and writer interaction in multiple perspectives and goals.
Third, CoAuthor preserves details of rich interaction during writing sessions.
Every text insertion and deletion as well as cursor movement is recorded as an event
at a keystroke level.
Lastly, we designed CoAuthor to be reusable and easily expandable in the future, by making
writing sessions modular, and using GPT-3 out of the box, rather than fine-tuning it
to a specific domain.
For data collection, we used a text editor where writers can press the tab key at any
time to get suggestions.
For the language model, we used GPT-3 with either high or low randomness.
Here are example suggestions from GPT-3 under the two randomness settings.
As a result, we collected rich, replayable interactions between 63 writers and GPT-3
in around 1500 writing sessions.
Now, we will share part of observations from our initial analysis.
For more analyses and details, please refer to the paper.
Our first analysis studied whether GPT-3 can generate fluent text in interactive settings.
For grammaticality, we observe that the text written by both a writer and GPT-3 has fewer
spelling and grammatical errors compared to the text written by the writer alone or GPT-3
alone.
For the diversity of vocabulary, we also observe that the text written by both a writer and
GPT-3 has the most diverse vocabulary.
This may imply that the use of suggestions from GPT-3 helps writers expand their vocabulary.
In another analysis, we looked at instances of writers taking new ideas from suggestions
and reusing them in the subsequent writing, by identifying named entities.
In creative writing, we observe that 13% of suggestions contained new named entities,
among which 20% were reused by writers in subsequent writing.
Lastly, we show that designers can use CoAuthor to find preliminary evidence for formulating
hypotheses under various definitions of good collaboration.
For example, a designer who wants to increase writers’ productivity may find that a number
of factors in CoAuthor positively correlate with the amount of text the writers and GPT-3
end up producing.
Likewise, a designer who wants to increase writers’ feeling of ownership may use ownership
scores in CoAuthor’s survey to assess potential correlation between the scores and other factors,
such as the proportion of text written by writers and the number of edits writers made.
To summarize, in this work, we identified a need for holistic understanding of language
model capabilities that are highly context-dependent and often subjectively interpreted.
Then, we argued that datasets can help designers understand such capabilities.
As a case study, we curated CoAuthor and showed that it can reveal GPT-3's capabilities under
diverse contexts.
Our dataset and interface for replaying the writing sessions are publicly available.
Thank you.