- [Lorin] All right.
Is the volume okay?
Thanks everybody for coming.
This paper is inspired by research
on Search-and-Rescue HRI
that I conducted at Stanford
under the late Clift Nas.
And, it some ways it can be considered
a retrospective on that work.
The broader applicability of the ideas presented here
became clear after I began working and discussing validity
issues with Wendy Ju,
whose projects often feature simulations.
A little overview where we're headed.
I'll begin by going over some definitions
in the paper's claim
and then its contribution.
Next, in the bulk of the presentation,
I'll talk about the possibility of justifying
our claim empirically,
and deliver on our theoretical justification for it.
And finally, I'm going to close
by reviewing measures, examples, and guidelines
that are related to, or follow from our argument.
So, some definitions.
Ecological validity is the degree
to which results obtained in controlled conditions
are related to those obtained in naturalistic environments
and this is frequently confused with external validity
which refers to how applicable a study's
results are to populations and situations
other than the ones studied without specific regard to any
correspondence or lack thereof between the laboratory and
the real world.
Traditionally, measuring ecological validity
required correlating data collected in the lab with
data collected in the real world.
For example, the ecological validity of tests in
executive functioning, which measure things like working
memory, attention, and inhibitory control
can be estimated by correlating participants' performance
on them with their ability to perform everyday tasks.
Fields like Search-and-Rescue HRI face serious obstacles
to the strategy.
One of them is impracticality.
Between 2001 and 2014, ground robots were deployed in
at least 23 different disasters worldwide, but no
live dependents were ever encountered
and to my knowledge, the situation has not changed.
The second challenge is clinical equipoise- the idea
that researchers are only able to treat participants
differently so long as no evidence exists of one treatment's
being superior to another.
Once such evidence exists, you're harming some participants.
And then there's the issue of informed consent.
People in life-threatening or otherwise traumatic situations
are by definition unable to consent to participate in
research because they're under duress.
Face validity is the extent to which a study or measure
seems to get out what it claims to get out regardless if
whether it actually does.
And our claim is that for researchers who are unable to
estimate ecological validity by collecting data both in the
lab and in the real world, face validity,
measured in terms of presence, can be a useful proxy for
ecological validity.
It's important to talk about the scope of our argument.
It's most relevant to fields that have things in common
with Search-and-Rescue HRI, like the well-being of
participants being at stake.
So it feels like military and police training,
space exploration, surgery all have that in common.
It's less applicable to areas like assistive robotics and
education, which can have negative consequences for
participants, but those might be less chronic.
But for any study, the onus is on researchers,
ethics committees, and peer reviewers to determine
whether any purported obstacles the real world
data collection are truly strong enough to justify
a reliance of its claim.
Our contribution is three-fold:
first, we describe and clarify complex theoretical problem
that may be unfamiliar to you.
To our knowledge, the only paper or workshop to have touched
on anything like this topic was one by Bennett and
and colleagues in 2006.
The second contribution is the explication of relevant
concepts and providing a glossary of key terms,
and we do so with reference to literatures on
social psychology and virtual reality.
And third, we provide a point of reference for discussion
and perhaps resolution in certain validity issues.
There's a couple of ways to empirically justify the
idea of face validity as a proxy for ecological validity.
One is to predict ecological validity with face validity
and the only study that at first glance seems to attempt
this is one by Deniaud and colleagues in 2015,
but it didn't capture data from (mumbles) driving,
only driving simulators. So it's out.
Another strategy is correlate ecological and face validity
and Hayes has made an analysis on the effectiveness of
flight simulator training provides mixed evidence here.
On the one hand, they found the combination of real aircraft
and simulator training led to better flight performance
than real aircraft training alone,
so that's consistent with our idea.
But on the other hand, whether a simulator included
motion cues had no effect in flight performance,
which seems counter intuitive.
So now we turn to some useful concepts in our theoretical
justification of the claim.
Transposition describes the maintenance of key structural
conditions when moving from one situation to another.
For example, in Milgram's study of obedience.
For those unfamiliar, the study involves participants
playing the role of a teacher and a confederate playing
the role of the learner.
The experimenter would instruct the teacher to deliver
progressively intense electric shocks to the learner
and of interest for the points in which the teacher
protested or refused to continue.
Milgram transposed factors like hierarchical titles,
teacher and learner, spatial separation by putting the
teacher and learner in separate rooms, associated objects
by attaching an electric shock generator or giving that
to the teacher and attaching electrodes to the learner, and
ideological commitments so that the teacher was
told that science was an end in itself, for example.
Another concept is emergence.
When a pattern or concept seems to emerge from a particular
set of stimuli, then it is said to be an emergent property
of those stimuli.
So consider what happens when one is asked to determine
whether democracy exists in a room full of congress people.
One might observe the positions of the chairs, which are
arranged such that people face one another instead of a
single podium.
One might listen to the way discussions are conducted,
which seems fair and reasonable.
Neither these observations alone will determine the presence
or absence of democracy, but the overall pattern that
emerges might prompt one to declare that democracy can there
be found.
The more likely people are to make this same declaration,
that is the more democracy tends to emerge in a situation's
details in the minds of different people, the more
face valid the presence of democracy becomes.
As this discussion suggests, there is a positive correlation
between a simulation's realism and its face validity.
However, there is a difference between the kind of realism
that leads participants to take a simulation seriously,
and that which denotes a strong resemblance between
the laboratory and the real world.
The Handbook of Social Psychology calls these experimental
and mundane realism, respectively.
Experimental realism matters because people take real life
seriously.
If participants fail to take a simulation seriously,
then it can hardly be claimed that the study's result
apply to the real world.
Mundane realism matters because similarity to the real world
is often pre-requisite to a situation being taken seriously.
The two kinds of realism are far from independent.
Throughout the 1970s, social psychologists,
including Milgrim, privileged experimental realism over
mundane realism.
And this raises and interesting question.
Why create experiments that have no strong connections to
the real world?
And the reason is that experimental realism allows us to
study psychological processes that matter a great deal
despite the fact that they are fictional or unlikely to
happen in real life.
For example, Jones and Sigall introduced what's known as the
bogus pipeline technique, in which participants were
hooked up to a lie detector, convinced that it works,
and then questioned about their attitudes.
The fact that lie detectors do not work and few people had
ever encountered one in real life did not prevent the
researchers in discovering how people behave when they
do not believe that it is possible to lie.
Experimental realism is beneficial to researchers in
Search-and-Rescue HRI for the same reason.
It is impossible to create a disaster scenario that has
perfect mundane realism because to do so would cause harm
to participants.
Structural conditions to elicit sheer terror cannot be
ethically transposed into the lab.
The opposite is true of other structural conditions
like sensory deprivation, uncertainty, and
perceived dependence on a third party.
These can introduce enough mundane realism that participants
begin to suspend disbelief in a simulation, and
if they do then experimental realism has been achieved.
The upshot is even in the absence of any real danger,
important conclusions can be drawn about how people think
and behave when they are sensorially deprived and
uncertain, and feel dependent on outside help.
In other words, one can draw conclusions that ought to
apply where ever exists the structural conditions
that were transposed.
Now I wanna talk about
The field of virtual reality has developed measures that are
useful in operationalizing realism.
These are immersion and presence.
Immersion, just like mundane realism, is a property of
simulations.
VR technologies that provide a wide field of vision,
accurately and quickly track people in three-dimensional
space, and offer stereoscopic vision are more immersive than
the ones that lack these features.
Presence, just like experimental realism, is a property of
user psychology of users said to be present in VR
when they subjectively experience being inside the
simulation as opposed to the real world.
The more immersive the VR, the more present users tend to
feel inside it.
The same relationship holds for fields like
Search-and-Rescue HRI, although the simulations are
physical rather than virtual.
Instead of field of vision, user tracking, and stereoscopic
vision, physical simulations are made immersive through
the use of props, trained actors, and deception ---
techniques far from the artist's stage craft.
The more immersive the physical simulation, the more present
study participants tend to feel inside it.
It's important to note that measures of presence get to the
heart of what it means for participant to take a
simulation seriously.
If a participant reports having experienced a virtual body
as their own, that means their avatar's actions and
appearance have real consequences for their thoughts and
feelings about themself and others.
If researchers observe someone use fewer paper towels after
they've been educated through VR about deforestation,
that means they took it seriously enough for it to
affect their behavior.
And if a participant sweats and experiences rapid heartbeat
while also claiming to have been present in a simulation,
that ought to increase our belief in the idea that they took
the simulation seriously.
To sum up, measures of presence tell us about the
experimental realism of the simulation, about how face valid
participants found it.
And they are, therefore, how we ought to operationalize
our proxies for ecological validity when a real world data
cannot be collected.
From 2011 to 2015, three studies were conducted with the
goal of understanding how autonomous a robot should be in
order to maximize positive outcomes for dependence.
The variables used in each study were often different,
but in each one, participants were instructed to imagine
that an earthquake had just occurred and interact with a
robot for about 13 minutes while awaited rescue.
To measure presence, we paid attention to whether
participants complied with basic instructions
such as not to touch the robot, and how seriously they
interacted with it.
Responding sarcastically, for example, indicated lower
presence.
Here I'm calling this measure compliance.
Details of the three studies can be found in their
respective papers, but suffice it to say that the
complexity of each simulation increased drastically across
all three studies, as did compliance across
studies two and three.
We went from a rudimentary physical simulation with a
styrofoam enclosure to an elaborate simulation
featuring a compelling script, realistic sound effects,
and well trained confederates.
And so the number of participants showing these breaks in
presence dropped from around 50 percent to only five
percent.
We don't know whether the results of any these studies
will apply to real world disaster scenarios.
But the reltively high face validity of study three makes
us more confident in the ecological validity of its results
than we are in the results of study two.
I couldn't find images of the simulations from
studies one and two, but this is what participants saw when
taking cover under a table during study three.
The image here is dimly lit, but participants experienced it
in under complete darkness,
save from some intermittent flickering from the lamp that
you see in the background.
In the foreground, you can see the corners of rubble and
binders that appear to have fallen from the walls on the
book shelf. The intensity of the simulation is not
well-captured in this single image.
Before arriving in this room and taking cover,
participants have been blindfolded and led in a
meandering path to disorient them.
They were also given a handheld radio that we had hatched
to play a realistic radio announcement recorded with help
from the campus radio station about the
extent of the earthquake that had occurred.
So, in conclusion, here are five steps for you to take
when creating immersive simulations.
First, you should check to see whether you can collect
real world data.
If you can, congratulations,
you don't have to worry about most of this.
Identify
Identify key structural conditions from the real world
situation of interest is the second step.
For example, if you're running a lab on how people interact
with a home robot, then hominess would be an important
structural condition.
Third, you wanna transpose the structural condition into
the lab.
In the same example, you can have each participant bring
any framed photo of a family member to set nearby while
go through the study.
Fourth, you wanna measure participants' presence.
Nobody seems to do this currently unless they're
studying VR, but it's the upshot of our entire argument.
And fifth, you need to repeat.
This entire process is iterative and you must constantly
devise and test new strategies for improvement.
Thanks.
(applause)
Happy to take questions.
- [Host] So I have a question. So do you think
about how the human emotional response could be simulated
in this set up. Like post the disaster, they're probably
gonna be confused, disoriented and...
- Yeah, exactly. So when we were
Are you talking about study three in particular?
- Yes.
- Yeah, yeah. So we worked with researchers from
Texas A&M University Robin Murphy CRASAR
Center for Robot-Assisted Search and Rescue
who interact a lot with people who do this work
on a day-to-day basis
and they told us a lot of different things
that were helpful for setting up the study.
They told us that the most important dependent variable
to look after was how hopeful somebody was that they'll be
rescued soon, they told us the different challenges that
people face when they are trapped and
we did everything we could to take those conditions they
told us about and create laboratory versions of them.
There's a couple of questions.
- [Male Audience Member] Hi. Thanks for the talk.
I am (mumbles).
I've been conducting lots of simulations in
the medical domain with mannequins.
Full scale simulations in the medical domain where you
interact in a physical world and to me it seems like
you were talking about VR a lot?
In the medical domain, it's always important to look at
what the actual aim is of that simulation and
tailor this then to what you actually need to simulate.
If the aim is training, then you don't have to have all the
touch and feel of the real world,
but rather provide the tools so that the experts can
kind of show their expertise in the training and
then learn.
And I was wondering what were the aims of your studies, like
should they learn how to behave in those simulations, or
learn how to interact with technology in that interactions?
- Yeah, that's a good question. Thanks.
So we were interested in the features that a search and
rescue robot should have in order to make dependence
the people that are awaiting rescue feel more hopeful that
they're gonna be rescued soon.
And other things, like how much they trust the robot and
the rescuers, that sort of thing.
- More questions?
- [Female Audience Member] I have a bit of a philosophical
question.
- Uh-hm.
- While you can create more immersiveness by following the
strategies you've mentioned, do you have any concerns
related to ethics, like around deception.
You can create more immersiveness if you deceive your
participants well, but that may raise some other concerns
for researchers.
- Yeah, absolutely. Deception is a major issue, and it's one
that we responded to by having thorough debriefs after every
session and making sure that all of participants' concerns
whether immediately after they participated or weeks or
months later will be addressed.
We opened lines of communication and made sure we're always
there.
If somebody ever had a negative response during the study,
which never happened.
Once there was a negative response prior to actually
putting the blindfold on in study three,
but we would make those participants aware of campus
resources or talking about the experience or
yeah how to get help.
- If there are no more questions, let's thank all
our speakers.
(applause)