In the field of Human-Computer Interaction, one way to validate newly developed interaction
techniques is a controlled experiment.
To plan an experiment, a researcher needs to plan the sample size.
However, it is hard not to under- or overestimate the required number of participants.
If the number is too small, researchers are at risk of not detecting the effect which
is present in the population.
If the number is too big, it leads to spending extra resources.
One of the methods to plan the sample size is a priori power analysis.
Researchers need to input three parameters: the probability of type I error, statistical
power, and expected standardized effect size.
A simple effect size is a difference between the two means.
To compare to another experiment where the dependent variable is operationalized differently
or the context is different, researchers need to use a standardized effect size, which is
calculated by dividing the difference between the means by a standardiser.
Standardized effect size is measured in the units of standard deviations.
The intuition is the smaller is the effect researcher is trying to detect, the larger
sample size is required.
Where can researchers find the reference effect size?
They can use previous work with similar experiment designs, but it is hard to find due to interaction
techniques’ novelty.
They can use Cohen’s convention for small, medium, and large effect sizes, but he suggested
using a local standard of the field instead.
Or they can use results of meta-analyses, but there are no such analyses for typing
experiments.
The reference for small, medium and large effect sizes is referred to as an effect size
ruler in this work.
Such rulers have been created in other fields such as Software engineering, Psychology,
education, and Behavioural Sciences, and they are different from Cohen’s convention.
The goal of this paper is to create an effect size ruler for typing experiments.
To create such a ruler, I first selected the relevant papers.
Then, I collected the data, calculated the Hedges’s g effect sizes, and binned them
into three categories in the same way as in the software engineering paper: the lowest
33%, the highest 33%, and 34% in the middle.
Then, I used a three-level meta-analysis in each bin to get the pooled weighted estimate
of the small, medium, and large effect sizes.
For the selection, I identified 108 papers with CHI typing experiments on ACM Digital
Library.
Then, I selected the top 30 papers with the highest citation ranks.
From these 30 papers, two did not have human participants, and one did not have a controlled
experiment.
Thus, 27 papers were assessed for eligibility, four of which did not have typing performance
measured, and two did not provide necessary statistical information.
Twenty-one papers with 25 studies are included in this analysis.
The resulting ruler has the pooled weighted small effect size of 0.1, the medium effect
size of 0.36, and the large effect size of 1.04.
The shown graph is truncated due to large effect sizes having a long-tailed distribution.
I used the same process to create a ruler for words per minute separately, as it is
one of the most common variables in typing experiments.
The full results are available in the paper.
During the data collection process, I confirmed some of the previous works on data availability,
reporting practices, and authors’ reachability.
I contacted the authors of 14 papers that did not have the necessary statistics available.
I received six responses, from which one did not have access to complete data anymore,
and eight authors did not respond to my emails.
These findings are not new, and the situation is similar in other fields.
To counter this in the future, I suggest the authors, if possible, make their full data
and analysis available online on websites such as osf.io to allow future cumulative
science.
How can the ruler be used?
The first usage is to do a priori power analysis.
Researchers can input this effect size into software such as G*Power, Touchstone2, and
pwr package for R. The second usage is to compare the newly found
effect to the local standard to identify whether it should be considered small, medium, or
large.
However, as a limitation of this work, this ruler should be used only for typing experiments.
In other subfields, separate rulers should be created.
You can find the paper and the research artifacts using the link on the screen.