Welcome and Thank you for your interest in our paper Cody:
An Ai-based system to semi-automate coding for qualitative research.
My name is Tim Rietz from the Karlsruhe Institute of Technology
and together with my co-author Alexander Maedche our research focuses on building human AI teams
to enable the data collection and data analysis from wide audiences.
And this also inspired this particular research,
as we did another project where we automatically conducted interviews with about 250 participants
and ended up with a data set that was too large for us to manually analyze.
So we looked into the qualitative coding environment, particularly into current
research and available commercial tools, and found that there are three primary problems.
Qualitative coding is time-consuming and error-prone, due to the amount of,
well, intensive focused work that's involved.
It can be a painstaking, repetitive process, especially with large data sets
which can force a trade-off between data size and data depth in the studies that we do.
The prevalent techniques that are presented to tackle these problems
is natural language processing commonly in the form of ex-post defined code rules,
which shows promise but has yet to be evaluated with end-users or qualitative researchers.
And on the other hand we have semi-supervised, supervised or unsupervised machine learning
which commonly faces issues with regards to the integration into coding processes
and the trust in the provided suggestions.
And to learn more about these issues and how we can fix them we ask two research questions.
Firstly, how do qualitative researchers interact
with and would they trust an interactive machine learning system supporting coding?
And secondly, how does our system compare to
the established qualitative data analysis system MAXQDA.
Our system Cody has three primary features.
Firstly, it allows the end user to define a code rule in a search query style
as well as providing initial rule suggestions.
Secondly, it uses the available coded data to make supervised ml-based code suggestions.
And thirdly, it uses counterfactual explanations to
help the user understand why a specific suggestion was made.
Here's a quick example that showcases the first feature, code rules.
The user can make any annotation, create a new label, for example,
integrity, and is provided with an exemplary code rule.
At any point, the user can go back, for example, and adjust the code rule if there's new
information, and changes will then be reflected in a change to the suggestions provided by the system
In our study, we present the evaluation of our system in two settings.
Firstly we started with two rounds of formative evaluations
where we collected information to improve our artifact, with two rounds of three participants.
And therefore, we applied a contextual inquiry including the think-aloud-method
and semi-structured interviews.
After we finished the development of our prototype
we started with a summative evaluation to investigate the artifice impacts.
We invited 11 participants to code a publicly available data sets in two treatments.
One was Cody versus MAXQDA and after they finished coding, we conducted individual
semi-structured interviews and analyzed the log data, if available.
The findings that we had include, regarding the impact of automated suggestions on coding,
that the code rules provided structure and transparency for participants.
And that, while checking the suggestions for quality is an extra step in the coding process,
it was perceived as beneficial to better understand the data set by participants
Further, implications for designing AI-based coding
support include that explanations are desired but rarely used in practice.
And that suggestions should be provided at an appropriate level of detail,
to not tempt participants to just accept suggestions.
Also, our results have implications for our future work and for future work in general.
Firstly, we saw that the combination of rules and machine learning can help
to increase coding quality.
However, these findings are limited by the specific setting of our evaluation.
And therefore, we are excited about evaluating such an interactive machine learning system in a
field setting where researchers actually use this data for their own work or to publish a study.
And secondly, we saw that Cody improved the intercoder reliability, however without relevant
speed gains in this particular setting where we focused on individual coding.
However coding usually is a collaborative
activity and thus we're excited to investigate collaborative settings as well.
And with that, I thank you for your attention and I'm looking forward to answering your questions.