All right. Hi everyone, I'm Ben Lafreniere from FRL Research, and I'm going to present our paper,
False Positives vs False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error
Preference. So, input recognition systems can make two main types of error -- false negatives,
where the user intentionally performs some input, such as a gesture, but the system fails
to recognize it... and false positives, where the user is not intentionally performing input
but the system recognizes input anyway. And existing work has looked at how to optimize the
trade-off between these error types for particular applications, but the typical assumption is that
there's one optimal trade-off. But if you imagine a gesture that works sort of like a mouse click,
where it could activate many different commands, then a fixed trade-off probably doesn't make
a lot of sense. A false positive could be really bad for a 'send message' action but,
maybe not so bad for a 'compose email' action. And false negatives are often not that bad because you
can just try the gesture again, but they could be really bad if your computer is going to restart to
apply updates if you don't do a 'cancel' action. So, we wanted to investigate how error-type
preference changes with the relative cost of false positive versus false negative errors,
to lay the groundwork for cost-of-error adaptive recognition techniques that can dynamically adjust
this threshold. Our research questions were, first, how does the time cost of recovering from
false positive versus false negative errors influence error type preference? Second,
are there hidden cognitive costs associated with these error types when time cost has been
accounted for? And, third, how does the task the user is performing influence these costs?
OK, so how did we do this? Well, our overall approach was to manipulate the relative time cost
of false positive versus false negative errors, and then measure error type preference. But
to do this, we needed a task where the recovery actions are consistent across the two error types,
and we can manipulate the time it takes to recover from each error type.
To address this, we developed the tile search task that you see here. The goal of this task
is to search a grid of tiles for target items and then select them with a click.
The system can inject a false negative error when the user finds a target item and goes to click --
the system acts as though no click has occurred. To recover, the user must reopen the tile
and then click to select it. And the time it takes to reopen the tile during this recovery
can be manipulated. The system can inject a false positive when the user finds a non-target item
and goes to move on -- the system acts as though a click has been performed and the tile is selected.
To recover, the user must, again, reopen the tile
and then click to de-select it. And, once again, we can manipulate the time it takes to reopen
the tile during this recovery. So, the recovery actions for both error types are the same, and
we can independently manipulate the time cost of each error type through the tile reopening times.
And we can also precisely control the number of each error type that is injected in the task.
Participants performed blocks were over about four minutes they had to open around 40 tiles
and six errors of each type were randomly injected.
Then they were asked for their error-type preference, through this interface here.
And they rated the strength of their preference. And we used the strength
response and the preference response to generate a strength-weighted error preference for each block.
And then finally participants provided a frustration rating for the block.
OK, so our first study followed a within-subject's design with each participant doing 13 blocks,
which tested a range of differences in time cost between the two error types.
And this study was deployed on Mechanical Turk. OK, so what did we find? Well,
this graph shows the strength of preference for false positive errors on the y-axis,
ranging between false negatives strongly preferred to false positive strongly preferred at the top;
by the difference in tile reopening time between the error types on the x-axis, from
false positives were much cheaper, to, on the right false negatives are much cheaper than false
positives. The first thing you can notice is that the fit line slopes down as it goes to the right,
so, as false positive errors become more costly relative to false negatives,
participants prefer them less. But what's really interesting is where the line crosses
the x-axis -- if all participants cared about was the time to recover, we'd expect this to happen
closer to the center of the graph, where the time cost between the two errors is equal, but instead
it crosses way over to the left, and this suggests a strong bias against false positive errors that's
not explained by the time cost alone. It's equivalent to around
one and a half seconds of added time cost... So why might this be?
Well, at the end of the study participants were asked to comment on the error types,
and a common theme was that there was this cost of paying attention to notice false positive errors.
In our next study we found further evidence for this added attentional cost.
We conducted a second study to understand the effects of high level task on error-type
preference. We tested the same task as in Study 1, but we also tested two variants of the task.
In the Time-Pressure variant the user had to do the standard task,
but you're also on the clock -- there's this bar counting down
and if the bar empties completely the cartoon cat eats a slice of pie, and you have to pay
a four second penalty at the end of the round for each slice of pie that's eaten.
And a Split-Attention variant, where the user has to do the standard task but they also have to
monitor a changing word at the top of the screen and hit a keyboard key when they see animal words.
And, again, for each word that you miss there's a four second penalty at the end of the round.
This study followed a mixed design with tasks as a between subjects factor,
and the cost conditions as a within subjects factor.
OK, so what did we find? Well, the Standard task -- the red line that you see here -- was about the
same as in Study 1... again we see a bias against false positive errors, though it's a bit smaller
in this case, on the order of about half a second here. The Time-Pressure variant -- the
green line -- is not significantly different in terms of bias against false positives,
but the slope is steeper suggesting a greater sensitivity to the time cost of recovery... and
that kind of makes sense given that it was sort of like a game you're trying to beat the clock.
And for the Split-Attention variant we see kind of a flat response...
and this prompted us to do a follow-up analysis where we subdivided participants
on this task into two groups based on how well they did on the word monitoring secondary task.
Here's the same chart with Split-Attention divided into into two groups. For the participants with a
lot of penalties on the word monitoring task -- that's the purple line -- we see kind of a flat
response and we believe that these participants were just overwhelmed by trying to do the two
tasks at the same time. But for the low penalty participants, the ones that were able to manage
the secondary task, we see a really high bias against false positive errors, equivalent to about
2.7 seconds of reopening time... so this provides more evidence that attentional cost is a driver of
bias against false positive errors. OK, so just wrapping up... we found that error type preference
can be driven by differences in the temporal cost of false positive and false negative errors.
We found that users exhibit a bias against false positive errors, which can be equivalent to one
and a half seconds or more of added recovery time. And we found that false positive errors impose
greater attentional demands on users as compared to false negative errors, which may partially
explain this bias. There's a lot of other interesting stuff that I've left out for time so
i encourage you to check out the paper. But, with that I'll end, and thank you all for listening.