I'm Hannah Cohoon and in this video, I present a case study from some recent fieldwork I
conducted under the mentorship of James Howison.
This video discusses grounded theory methods, focusing on an often mystified aspect of this
methodology: theoretical sampling.
We share our experience with theoretical sampling, drawing attention to how useful it can be
for overcoming challenges during data collection.
And we provide recommendations for how you can practice theoretical sampling yourself.
Though a grounded theory approach, researchers produce an explanation of some phenomenon.
But publications rarely describe in detail how that explanation or theory is created.
Results and discussion sections might be rich with detail and analysis, but grounded theory
methods are often described only briefly.
For newcomers to HCI or grounded theory, this makes learning difficult.
And glossing over methodology runs counter to the growing demand for transparency in
science.
Our case of grounded theory methods and theoretical sampling comes from my dissertation work.
To better understand how a platform for open science, OSF, is built and used, in this study
I collected several forms of qualitative data and used a grounded theory approach to analyze
it.
OSF allows researchers to openly share their work through OSF projects.
Projects can contain datasets, papers, study materials'97whatever the researcher wants.
The platform is also useful for registering research, an open science practice that involves
disclosing methods and hypotheses often in advance of data collection.
I conducted a grounded theory study of this digital field site, OSF'97its users, its developers,
and its nonusers too.
Grounded theory analysis involves iteratively collecting and analyzing data to construct
a theory that explains the observed phenomenon.
This theory can then be generalized to other contexts.
The iterative processes involved in grounded theory are referred to as constant comparison
and theoretical sampling.
Together, these processes are an abductive approach that produces a theory that is '93integrated,
consistent, plausible, close to the data.'94 Rather than deduced through hypothesis testing,
the theory is grounded in the data and internally validated.
This work can be informed by sensitizing concepts that provide '93initial but tentative ideas
to pursue'94 during analysis.
Sensitizing concepts for the study of OSF were derived through a literature review.
For instance, we were sensitized to possible conflicts between the values of OSF developers
and researchers meant to use OSF.
We were also sensitized to routines as a useful source of insight about actors' primary concerns
and challenges.
There was no formal hypothesis, but the data collection strategy was designed to detect
differences in values and routine practices.
I sought to create a theoretical sample that could do this.
So interviews over Zoom addressed scholarly values.
Observation was planned to capture OSF developers' routine behavior.
And I planned to gather users' data from OSF logs as evidence of their routines as well.
Constant comparison and theoretical sampling prompted adjustments to this data collection
plan along the way.
Constant comparison is the fundamental process of grounded theory analysis.
And it means that the researcher is continuously making comparisons between data points so
that they can find patterns.
Comparison might be made in written memos or by coding data and comparing coded examples.
Data might be recoded later and old codes might get combined like here in Atlas.ti.
Codes get tested and made more consistent through additional constant comparison.
In the study of OSF, constant comparison also showed things like where recruitment efforts
needed to be concentrated.
For example, by comparing coded data about registration it was clear that I needed more
data from users who had registered a study on OSF.
There was plenty of data on OSF developers and their understanding of registration, but
little in-depth detail on users' experiences.
I saw this through comparison.
but following through on that insight and adapting recruitment to capture more about
users is an example of theoretical sampling.
Theoretical sampling is when the researcher collects new data because of insights they
derived from old data.
As new data are collected, they may prompt the researcher to pursue new directions or
dig deeper into ongoing lines of analysis.
In the example I just gave, I was prompted to dig deeper into the practice of registration.
Theoretical sampling should be conducted with respect to the study's goals.
Fathers of the grounded theory method, Glaser and Strauss say that the basic question in
theoretical sampling is where do we turn to next in data collection?
And for what theoretical purpose?
Researchers shouldn't wander toward available data, they should seek out data that informs
their research questions and emergent findings.
Theoretical sampling helped address a major challenge during the study of OSF.
The original plan to collect users' OSF logs proved troublesome.
The logs like the ones shown here showed the actions (like uploading files) that OSF users
had taken to build their projects.
The data was meant to triangulate findings from interviews & observation and provide
evidence of routines.
But, I failed multiple times over many weeks to gather activity logs through the OSF API
and web scraping tools, so we decided a new theoretical sample was needed.
But where should we turn to and for what theoretical purpose?
Rather than just collect any additional data, I needed to create a theoretical sample that
would best inform my ongoing analysis.
So when faced with the challenge of finding a new data source fairly late in my study,
we identified one that would bring greater depth to themes I had already recognized as
important in my data.
This would help clarify my emerging grounded theory.
We also reasoned that it was still key to collect behavioral data on users' routines'97this
connected with the sensitizing concepts that had inspired the study and this additional
form of data allowed for triangulation.
I opted to create a theoretical sample to meet these needs by doing what I called project
mapping.
I used Miro to visualize the organizational hierarchy and contents of users' OSF projects.
When compared with my interview and observation data, these project maps, like the one I'm
building here, were further evidence of established findings: I saw that some users considered
their research to be registered when they uploaded files that explained their study
methods.
But, OSF developers considered registration to be a native feature of the platform that
involves creating an archived version of the user's project.
These findings were evident in my interviews, but project maps showed the extent of users'
confusion'97some users leveraged the native feature, some users uploaded files, some users
did both within the same project.
This insight about conflicting forms of registration resonated with the sensitizing concepts established
through my literature review, but it wasn't anticipated or hypothesized.
Instead, it was cultivated through a series of decisions about what data to collect or
which theoretical samples to construct.
When constructing a theoretical sample yourself, we suggest answering a few questions to help
guide and document your decision making:
Why are these data the right ones for the study?
How will these data speak to the research questions, sensitizing concepts, or emergent
findings?
Which findings prompted changes in recruitment strategies?
What aspects of recruitment does theoretical sampling influence?
What ways does it impact research materials like interview protocols or collected traces?
Journal articles might not have room for your answers to those questions, but they are excellent
memo topics.
And, you can commit in advance of data collection to answering these questions'97Qualitative
research might not be traditionally thought of as compatible with open science practices
like registration, but by committing in advance to answer questions like these, you can promise
post-hoc transparency and make yourself accountable to critically engaging with your data'97the
real goal of grounded theory methods.
We hope you've learned a little bit about what theoretical sampling looks like in action
and have a better feel for how you might implement it in your work.
Thank you very much!