Hello and welcome, I'm Brianna Tomlinson, I’m a PhD student at Georgia Tech in Human-centered computing.
Today I will present our paper, Review of Quantitative Empirical Evaluations of
Technology for People with Visual Impairments, a research project conducted with
Emeline Brulé, Oussama Metatla, Christophe Jouffrais and Marcos Serrano.
This paper received an honorable mention award and had really passionate and dedicated reviewers: if you are watching: many thanks!
This paper started at CHI 2018 during a workshop on technologies for visually impaired people, during which we discussed the many
issues encountered during evaluation.
The problem could be summarized as follows: for assistive technologies to be adopted, or to increase the uptake of assistive tech,
we need robust quantitative evaluations of assistive tech.
This is a gold standard in policy - and there’s long been a concern that assistive tech is not adequately evaluated.
For instance, in their 2011 literature review, Kelly and Smith found only two papers with evaluations reaching quantitative standards in
educational technology publications, over two decades of research.
First off, this review and evaluation practices can not be separated from concerns about quantitative empirical evaluations in the entire
HCI community - in particular, the concern about the validity of
experiments and statistical validation, can be broadly summarized as a replication crisis.
Often, there’s a concern that lab-based user studies, because of their non-realistic settings, are not really a good indication of use in
practice, and don’t represent use over a long period of time.
There are also concerns regarding the evaluation of accessible technologies, especially with visually impaired participants:
We need to be careful when using usability scales because it appears visually impaired participants rate their experience higher.
Representing the diversity of users in a study is difficult, and many scholars in the disability community have called to end evaluations with
blindfolded participants, because it’s disrespectful, and because it doesn’t realistically represent their experiences.
Finally, Kelly and Smith argue that our usual experimental designs might not be adequate, and that to represent this diversity, we could
instead focus on single subject experiments.
However, we don't have a clear picture of how quantitative empirical evaluations are used and conducted in HCI for this population,
Nor how it’s changed over time.
In this work, we investigated three questions:
1st: Are evaluation practices consistent? If so, how are they designed and run?
2nd: Are participants representative of the target population?
3rd: W hat are the difficulties reported?
We'll step through each of these individually, but first we'll cover some background on our search.
We completed a systematic literature review using a scopus search
on publications from CHI, TOCHI, ASSETS, and TACCESS.
We included both artifact and empirical research contributions.
We included 178 papers from 1988 to 2019.
We tagged for things like mode of interaction, application area, and number of reported quantitative evaluations.
The corpus is available in the supplemental materials for this paper, in the ACM digital library.
Please use it to investigate your own research questions.
As for the tagging process --
It’s not that easy to decide what counts as a quantitative empirical evaluation --
we based our work on Barkhuus and Rode's 2007 paper,
which defines evaluation between the type of measures (qualitative or quantitative)
and the type of design (empirical or analytical).
where the former involves users as participants and the latter could rely on experts).
So let's jump into that first question:
Are evaluation practices consistent? If so, how are they designed and run?
The most common paper type, about 80 papers or 44.9%
were those presenting a new system followed by a summative quantitative empirical evaluation.
Think of these as your prototypical lab-based studies measuring performance using statistical tests.
The other 55% included a large variety of approaches,
and had formative studies, and leveraged more user-centered and iterative evaluations.
On to the 2nd question: Are participants representative of the target population?
According to the World Health Organization,
blind and legally blind people make up about 7% of all visually impaired people.
But 65% of the evaluations we examined involved only blind participants.
Additionally, 65% of visually impaired people are over 50 years old,
while the median reported participant age was 38.5
As far as children go, about half of children born with vision impairments in the global north countries have other disabilities,
but only 2.4% of studies reported participants with additional impairments.
And finally, we want to acknowledge that this issue of inconsistent reporting harms replicability and our ability to identify promising
leads for future work.
Across the corpus, there was no consistency in how authors described visual impairments.
Reporting things like age of onset or cause of impairment would help understand the impacts toward cognition or technology use.
Lack of consistency in additional information makes it difficult to make connections between study results and generalizability toward others
within this group.
And finally, the 3rd question we investigated was: what are the difficulties reported across this literature?
17.3% of the papers in our corpus openly disclose difficulties.
Some common difficulties in conducting the research included:
the small pool of participants which limited the number of participants researchers could realistically involve in evaluations.
This is even worse when the evaluation requires that participants have a specific expertise.
We confirmed how the diversity between participants is difficult to handle, which can lead to excluding participants from a study,
or to including sighted people for purposes of statistical validity.
However, this becomes a limitation of the generalizability of the work.
Additionally, these issues are compounded by heightened practical difficulties such as scheduling,
non-adapted experimental procedures being too long,
and the lack of shared measures within the similar application areas.
In fact, only 59.2% of the paprs involved only people with visual impairment.
In summary, this area of HCI requires different standards for quantitative evaluations,
as we also need to avoid practices which weaken validity such as including blindfolded participants,
or the avoidance of quantitative approaches altogether.
We have some recommendations to help address these issues.
1st- for conducting evaluations:
Non-visual interaction is not just useful to visually impaired people,
but to make claims about their usability for visually impaired people, they need to be involved.
To address known difficulties, consider single subject design experiments.
For a second recommendation, when reporting evaluations
If possible, report median age, gender, education level, and experience with relevant technologies or subjects.
For evaluations of technical accessibility, use the world health organization or WHO classification of visual impairments
to provide detailed information about participant characteristics in a standardized manner.
And 3rd, when reviewing:
It is not always appropriate to require large samples of participants as this leads to harmful practices
like adding blindfolded participants to achieve statistical validity.
Internal validity (adequate description of participants, etc.) is a more important factor to consider
We wanted to finish with some Q&A.
Thank you to Meredith Ringel Morris for her questions.
The first question is: "We see papers being rejected because they include sighted participants
instead of blind participants in evaluations.
It’s highly recommended not to.
Why does the paper say it’s accepted practice and has this evolved?"
So for recently accepted papers from the corpus, from 2014 to 2019,
did use blindfolded participants. It seems to be generally declining but varied in practice across those years.
For example, in 2014, 3 papers used blindfolded participants.
But in 2017, that number was up to 5 papers then it came back down in the following years.
The second question is: "Should quantitative evaluations be considered a gold standard with this population?"
Our answer is no.
Especially if we tweak evaluations to fit expectations that are not realistic.
There is an opportunity to advocate for qualitative evaluation approaches at the policy level.
You can find all of the Q&A at: tinyurl.com/CHI-QandA
And then, finally, the two takeaways we want to highlight are:
We need to improve our quantitative evaluation practices to improve communication with other fields and technology uptake.
We need to better account for the specificity of this group (possibly through single subject experimental design).
Please reach out to any of us if you have questions about the work. You can find our emails here, or in the paper.
We also have more resources available at inclusiveeducation.tech
Thanks!