This is "Pulling Back the Curtain on the Wizards of Oz,"
by Martin Porcheron from Swansea University,
and Joel Fischer and Stuart Reeves
from the University of Nottingham.
This paper offers an ethnographic insight
into the use of the Wizard of Oz method.
WOz, as it's often shortened to, consists
of an intelligent system being presented
to users, where this intelligence is
a deception managed by human operators
pulling metaphorical levers.
It's often used in iterative design processes
or in usability testing.
And obviously, it has a long tradition in HCI,
spanning some 30 years or so.
Recent news has tended to involve assimilation
of some sort of machine learning component, often
natural language interfaces, or in robot control systems.
And this is probably due to behind development time
and high monetary costs involved in building such a system,
especially if it's just for an experiment
or you're partway through a design process.
Literature tends to gloss over the details
of the interactional organizational work involved
in running such a study.
So in this paper, we examine our use of the method
by drawing upon one of our experiments as a case study.
So we were conducting a study, which
was an elicitation study with a natural language robot vacuum
cleaner.
We wanted to understand what language people would
use to instruct this vacuum cleaner to clean
different areas of a workplace.
And in particular, we chose a food chemistry laboratory.
Participants were given five scenarios to complete,
and the researcher would stand with a participant
while another researcher would be working elsewhere
to control the robot in secret.
I don't have time to present much from this today,
but I will be presenting one video clip very shortly
of a participant issuing a command to a robot
and the robot responding.
And let's just say you can get a flavor of what
we're looking at in the paper.
So this is the laboratory where our study took place.
On the left of the screen, you can see me,
and you can see me from two different angles.
And here, I'm working as a wizard sitting at a laptop,
controlling the voice of the robot.
We then have a participant standing
with a researcher who's reading from a piece of paper.
Over here on the right, we have the two vacuum cleaners.
Now we're only using one vacuum cleaner in each study.
The second is a hot spare should we have any technical issues.
Finally, we have three piles of debris.
They're not very clear in this video clip,
so you'll just have to trust me that that's where they are.
So let's watch.
[VIDEO PLAYBACK]
So the first one is a simple cleaning scenario.
And in this lab there are three areas that need to be cleaned,
and you have to choose an area to clean and instruct
the Robovac to clean the dirt or the dust.
You have to call them over to the mess, Robovac.
Robovac?
Yes.
OK.
So if you can do it.
Robovac, move forward and clean.
OK.
I will clean forward now.
[END VIDEO PLAYBACK]
OK.
So I'm going to stop there.
So this is just a 35-second snippet of some data.
But we're going to start to see this deception unfolding.
And it's an interactional outcome
of the work of the researchers throughout the entire study.
It's not just a given.
And they're using various resources
to accomplish this as well.
So things like the setting, the narrative,
also the tangibility of a robot, all help to pull this off.
But also, the researcher who is introducing the study
is presenting a fiction from the agreed protocol.
They're reading from this.
And the wizard also has a copy of it on the screen.
So you can see a copy here on the right of the screenshot.
On the left is the system they used to control the voice.
So the wizard is enacting the fiction being
presented by the researcher.
So you can start to see elements of collaboration going on.
The researcher has to treat the actions of the robot as
per the fiction, so they have to performatively
account for each robot action throughout the scenarios.
And this means, whatever a wizard
chooses to do in resort to a participant request,
a researcher has to be able to attribute this to an autonomous
system of sorts.
The wizard, conversely, must be responding
to participant requests in a way consistent with this fiction
being presented by the researcher.
So both of these people have to work together,
but they can't communicate.
There is also a challenge here for the wizard.
We have to interact with the robot
the way the NLU/NLP might were the system actually
implemented, and they have to do this quickly and consistently.
So that's the time pressure for them.
A further pressure, of course, is the contingent nature
of all interaction means that a wizard must
be able to respond to any form of request,
no matter what the participant says.
And that also introduces new methodological obligations
of how to ensure consistency between and within
participants.
We talk about that in the paper as well.
That's my five minutes.
So thank you very much.
Please do read the paper.
If you have any questions, I'm happy to take them.