Hello in the next seven eight minutes I'm going to talk to you about a key element
in designing human computer interaction studies recruiting participants my name
is Mohammad Tahaei i'm a research associate at the University of Bristol i did this work
in collaboration with my PhD supervisor at the University of Edinburgh Dr Kami Vaniea.
participants can directly impact the results of a study for example think about privacy which
is interpreted very differently across countries and age groups someone from the Middle East may
have a completely different sense of privacy compared to someone from a European country
or someone who only programs using visual tools has different needs compared to someone who has
experience writing C code therefore when designing human computer interaction experiments researchers
should think about who they are creating their technology for and who is going to test it in the
past decade there has been a growth in empirical software engineering research this specific
type of research tries to make programming more accessible and usable to a broader population for
example by making programming languages IDEs and libraries easier to use however one key challenge
in these studies is finding participants for these type of studies often it's needed to
have participants with a programming background so some researchers recruit computer science students
because this population is expected to have some level of understanding of computer programming
finding professional developers is also a known challenge in these type of studies
some researchers have started using crowdsourcing services to recruit
participants some others began harvesting emails and contact information of developers
from developer forums such as GitHub which is actually against some forums privacy policies so
all of these approaches may not be sustainable over time and we wanted to better understand
and explore different recruitment channels for recruiting participants with programming skills
Our research questions cover three main areas
first we wanted to understand which recruitment channel amongst the
ones that we picked is the most suitable for recruiting participants with programming skills
second we wanted to know whether is it's possible to use self-reported demographics or programming
experiences as a way to screen participants and lastly we wanted to understand how different
recruitment channels compare regarding security and privacy attitudes and secure software
development self-efficacy we added the last question because one of the difficulties we had
in our own research was finding participants for usable security and privacy research
directed at software developers often reported as a challenge by other fellow researchers
To answer our research questions we chose to distribute our survey on five recruitment channels
four crowdsourcing websites and one mailing list for computer science students the crowdsourcing
platforms were chosen based on what has been used in the literature in HCI studies we used Appen,
Amazon Mechanical Turk, Clickworker, and Prolific we also utilized our local mailing list for
computer science students, a common practice in empirical software engineering studies
we first screened participants based on four questions mainly interested in whether they
would claim to have programming skills we then chose five survey instruments suggested by the
literature to assess programming experiences and skills general security attitudes general
privacy attitudes and secure software development self-efficacy we decided to use survey instruments
suggested by the literature instead of inventing new survey instruments because we wanted to
compare our results with prior work and make our results easier to compare for future researchers
our data set is also available online for anyone who wants to use it
I'm going to show you some of our findings though I highly suggest you look at the
paper because there are a lot of interesting insights in there and a huge table that shows
how much we spent on each specific step in each particular recruitment channel so it's gonna give
you ideas about how much money you need to spend to recruit participants with programming skills
In this chart you can see participants who took the main survey all of these participants were
screened against the question "do you have programming skills" and if they say yes
they were invited to take the main survey one of the survey instruments assessed participants
programming knowledge with five questions that covered the website that they frequently used
for programming description of a compiler and a recursive function and the most fitting value
for a boolean and identifying the parameter of a function in a pseudocode algorithm with a loop
As you can see Appen resulted in zero participants who passed all these questions
Clickworker and Amazon Mechanical Turk resulted in few participants which means that they do not
seem to be a suitable recruitment channel for recruiting this type of participants
Prolific however resulted in a good number of participants it also enabled us to screen
participants without running a survey making it much easier to screen participants computer
science students are the most likely to get all the programming questions correct so they are
also a good channel for recruiting participants with programming skills. We are also interested
in better understanding whether just asking simple questions such as how many years of experience do
you have in software development would result in finding participants with programming skills
this table shows our results from our generalized linear model the outcome variable is getting
all the five programming questions correct and the inputs are a bunch of demographics
for example if we ask people about their role in a software development team and if they say they are
a developer this can increase their chances of getting programming questions correct by
2.4 times compared to someone who doesn't work in a software development team
noting that we are not saying that this question should be used to screen participants but it can
be helpful to better understand whether a participant does have programming skills or
not by simply asking them if they are a developer or not quickly looking at the different channels
computer science students are about 26 times more likely to get the programming questions correct
compared to Prolific participants please see the paper for a comparison of different recruitment
channels for privacy and security attitudes as well as secure software development self-efficacy
our experience of using various recruitment channels and several other insights that can
help researchers recruit participants with programming skills thank you for listening.