Hi! My name is Sonu Mishra and today I'm going to talk to you about this Late-Breaking Work,
"Theory is in the Eye of the Beholder: exploring difficulties
with validating intervention mechanisms." When we create behavior change interventions,
we make them in the expectation that they'll work a certain way,
and that's called the mechanism of change. Different models of behavior change, like the
theory of planned behavior and the health belief model, describe different mechanisms of change.
And understanding what mechanism an intervention works through is important because the knowledge
we gain about designing for specific mechanisms is what enables us to accumulate knowledge about
designing for behavior change. And for this reason,
there have been increasing calls in the HCI community for articulating and measuring our
intended mechanisms in behavior change research. But how do we validate that the intervention
works the way we intend it to? In particular, how do we know that the end user responds to
the intervention the way researchers expect? What if the researcher thinks it's operating
one way, but in-- for the user it's actually operating through a different mechanism?
We need to understand whether user responses to interventions align with expert predictions
about how interventions work. Otherwise we run the risk of
not understanding the full picture of how an intervention is succeeding or failing.
So we looked at this question in the context of a larger study exploring how to encourage
hospitalized patients to speak up to clinicians when they have concerns about their care.
We sought to create low fidelity prototypes encouraging patients to speak up based off the
Integrated Behavioral Model, and explore patient responses to those prototypes.
So to do this we did three studies, and I'll describe each of them in turn.
Study 1 was a survey study asking seven experts--defined as people with
advanced degrees in HCI, behavioral science, or related fields--to
map the low-fidelity prototypes we created onto mechanisms in the Integrated Behavioral Model
that influenced behavioral intention. And our goal was to validate that our
prototypes would relate to specific mechanisms by achieving expert consensus on what prototypes
acted through which mechanism. Sounds simple, right?
What we actually found was really low consensus amongst experts, which we hadn't expected.
And we also heard some feedback from participants
that the survey had been really hard. So we just wanted to find out what
was going on and make sure that our validation method was valid.
So we did Study 2. We interviewed some of our expert participants to understand
what was going on and what their thought process had been in completing the survey.
We found that experts sometimes mapped prototypes onto more than one mechanism when the prototype
was too complex to operate through just one. So for example, a prototype designed to build
self-efficacy by showing examples of other patients speaking up
also operated through influencing people's beliefs about what other patients did--but
that's a different mechanism from self-efficacy in the Integrated Behavioral Model.
In addition, we found that experts often had assumptions about the
interrelations of different mechanisms. So for instance, feeling that people's
beliefs about what other patients do would influence self-efficacy.
So those things together made expert consensus very difficult to achieve.
We eventually assembled a set of prototypes using a simple majority instead of
statistical agreement and did Study 3. And that was taking these prototypes
to hospitalized patients to explore what kinds of systems might be valuable to them.
So we interviewed patients at two different hospital sites including both adult and
pediatric patients and including the parents or caregivers of some of the pediatric patients
and we coded the transcripts using a codebook derived from the Integrated Behavioral Model.
We found that experts--like experts, patients didn't always interpret the prototypes in the same
way--they didn't always have consensus either. And moreover, patients and experts didn't always
align in their interpretation of the prototypes. So for some prototypes that experts had agreed
acted through one mechanism, patient responses illustrated that it could operate through another.
So what does this mean? The question is which
mechanism should be evaluated--the mechanism that researchers who create the behavior
change intervention intend to target? Or the mechanism that data from end
users suggest may actually be in effect? Our findings show that these mechanisms
may not be the same, and in addition, interventions with complex designs may
unintentionally leverage more than one mechanism, or those mechanisms may actually be interrelated.
So maybe instead of asking, "Which mechanism should we evaluate?" researchers should ask,
"What possible mechanisms should we evaluate?" And to do this, researchers should look at the
whole of the theoretical model their intervention is based on.
So for instance, if a researcher creates an intervention using the Integrated Behavioral
Model and decides to target self-efficacy, the researchers should also see if the intervention
affects other mechanisms in the model such as perceived control, beliefs about norms,
and attitudes towards the behavior. There are obviously some limitations to this work.
We had a small sample. We only studied one model,
which means it's hard to say if our findings are just due to problems in the model.
But the findings still reveal something critical, which is that the way researchers
think their intervention is operating may not line up with how it actually impacts users.
To account for that, researchers should think about measuring outcomes using the
whole theoretical model they're basing their intervention off of, not just the
mechanism they're specifically targeting. Thanks to AHRQ for funding this work,
our research sites and participants, and my wonderful collaborators.