Hi, and welcome! My name is Mikke Tavast, I'm a PhD student at Aalto University and
I'm presenting our paper titled Evaluating Large Language Models in Generating Synthetic
HCI Research Data: a Case Study co-authored with Perttu Hämäläinen and Anton Kunnari
Our work stems from the observation that much of HCI research is conducted using
verbal or textual data and given the recent success of large language models, LLMs,
in this work we sought to evaluate LLMs as a synthetic data source for HCI research.
There could be a number of benefits of synthetic data, for example,
in initial pilot testing, setting up analysis pipelines, and generating new research ideas.
From a broader perspective we argue that with LLMs computational user
modeling could be expanded to new avenues such as to user motivation and emotion
In this work we use openAI's GPT-3 model to generate synthetic data and compare this to real
research data from a study by Bopp and colleagues concerning art experiences in video games.
We chose this data set for two reasons. First, experiencing art is a topic that should be
challenging from an AI user modeling perspective, and also, the data set is recent enough for not to
be included in the training set of the original GPT-3 models. We conducted three experiments,
but on this presentation, I'm going to focus only on the experiments one and three.
Here you can see GPT-3 in action. So, I have given it a prompt and GPT-3
then continues the text by continuously predicting the next token in the sequence.
Here you can see different prompts used in the experiments, each prompt ends with a
question that is directly adapted from the open questions in the Bopp and colleagues study,
and this allows meaningful comparisons to be made between the real and synthetic datasets. The first
question asked to remember an instance where a digital game was experienced as art, the second
what is the title of the game, and the third why the participant considered this experience as art
The first experiment is a quantitative study on how distinguishable GPT-3 responses are from
real human responses, and in this experiment we used the answers to the first question
in the previous slide. For this experiment we recruited participants from Prolific and the task
for the participants was relatively simple, shown here on the right side of the screen.
They were asked to read a text and then decide whether they thought it was written
by a human or generated by an AI, and before the experiment the participants had been
informed that half of the text passages they will see were written by humans and
half generated by an AI system called GPT-3, but that the presentation order is randomized.
With methods from Signal Detection Theory, we analyzed whether the participants were more
likely to identify human written text correctly or to answer that GPT-3 generated texts were
written by humans, and the results showed that actually, in this experiment GPT-3 responses
were significantly more often evaluated as human written than human written texts.
On the figure on the right, you can see the average proportion of "written by
human participant" answers for each stimulus in the experiment
Then let's move to Experiment 3, where we investigate the content similarities
between the generated synthetic data and the human data of Bopp and colleagues.
The main motivation behind this study is that for synthetic data to be useful, it needs not to be
only superficially human-like, but also reflect real properties of user behavior and experiences.
For this experiment we generate responses to all these three prompts shown earlier,
although in this presentation, I will mainly focus on the prompt three, what exactly made
you consider this experience as art. We analyze these responses using automatic qualitative coding
So what do I mean by that. The automatic qualitative coding
approach combines GPT-3 powered coding of the responses and topic modeling methods.
First we use GPT-3 to generate codes that summarizes the content of the answers. The
motivation behind this coding is that it distills the essence of the answers and reduces noise.
Concretely, the automatic coding is achieved with the prompt designed to guide GPT-3 to
generate summarizing codes for the answers, here you can see one answer being coded.
The few shot examples in the prompt guides GPT-3 to generate codes: memorable experience,
emotional response, graphics, music, and strategy, summarizing the content of the response.
Once we have extracted the codes, the next three steps are similar to the BertTopic
topic modeling approach. First, the codes are transformed into vector representations using
text embedding models, the dimensionality of the embeddings is reduced using UMAP,
and then they are clustered using HDBSCAN, and at the end of this process, we have combined
similar codes into larger groups or topics. And by counting the frequency of the clustered code
groups this allows us to compare the prevalence of topics between the human and GPT-3 data.
As a side note we are planning, in near future, to release a dedicated python package implementing
this approach, so if you're interested follow Perttu's GitHub profile to keep you updated.
And to the results so this figure is a bit complicated, but let's go through it step by step.
On the left, there are the human data groups or code clusters and on the right are the GPT-3 code
clusters, and they are sorted from top, from the most frequent, to bottom, to the least frequent
code groups. And the lines connect each human data code group to the most similar code group
in the GPT-3 data, and if we zoom on the top of the figure we can see the most common code
groups in both datasets. For example, the most frequent group in both datasets relate
to aspects of story and storytelling, the second most frequent to aspects of music and soundtrack,
the third most common code group in the human data relates to experiences of beauty
and similar kind of code group can be found as the fourth most common in the GPT-3 data.
So there seems to be at least for the most frequent code groups quite a lot of similarities.
And here's another set of results from the experiment. I'm not going to go into details here,
but I will just highlight that we also looked at the performance of different GPT-3 models,
and we analyzed the performance in terms of similarity of the distributions of the embedding
vectors for synthetic and real data, and generally the quality increased as the model size increased,
but curiously the newer text-davinci-002 model that is finetuned using human feedback had
high quality data, but it covered less of the distribution of the real human data compared to
the older davinci model, the original 175 billion parameter GPT-3 model. So this suggests that
the finetuning has increased the quality of the answers but reduced the diversity of the responses
And to conclude we find these results promising and intriguing. We feel the synthetic data could
be useful in exploring possible research ideas or hypothesis, especially if combined with automatic
data analysis similar to our experiment 3. That said, as LLMs are well known to
hallucinate facts and generate false information, to state the obvious, confirming hypotheses on
what people really think, feel or need should be only based on real data. And unfortunately
quality requirements for synthetic data might be lower for misuse than for actual research.
One possible implication of our results is that if malicious users start using LLMs in online
crowdsourcing services to conduct studies faster, this might make crowdsourced self-report data
fundamentally unreliable, as these responses would be hard to distinguish from real data.
I thank you for your attention.