Hi, my name is Saskia, and today I am going to present our paper CrowdSurfer
Seamlessly Integrating Crowd-Feedback Tasks into Everyday Internet Surfing.
This is a research project of my colleagues, Ivo Benke, Daniel Fischer, Alexander Mädche and me.
The continuous evaluation of software is an important part of every development process.
It is not only crucial for user acceptance and satisfaction but can also help to
identify problems and new requirements. One approach to evaluate software designs
in a scalable way is the crowdsourcing of feedback via dedicated software systems,
also called crowd-feedback systems. In this approach, usually, a large group
of people with no significant experience in software design is recruited on crowdsourcing
platforms like MTurk or Prolific to provide feedback on graphic or interactive designs.
We identified several problems with traditional crowd-feedback studies.
First, feedback providers are decoupled from an actual context of use. They are presented
with a design and are asked for their opinion often without completely comprehending the
situation in which the system would be used. Second, they are not actual or intended users
of the design which means their feedback might be less relevant. Maybe they just do not understand
the design because they lack domain knowledge. And third, traditional crowd-feedback studies
are not designed for a continuous feedback collection but only provide an evaluation
for a snapshot of the current design. So, in this study, we aimed to develop
a solution for these problems. Our goal was to develop a system to collect in situ crowd
feedback for websites via casual microtasking. Casual microtasking describes the integration
of microtasks into other primary activities. This will allow us to continuously collect
feedback from actual website users while they are using the website.
To develop the crowd-feedback system, we followed a three-step approach. First, we
conducted interviews with crowdworkers to better understand their requirements for such a system
and reviewed the literature. Then, we designed the system, and finally, we evaluated the system.
In the first step, we developed a simple prototype of our system, that we called
“CrowdSurfer”. The CrowdSurfer was implemented in the form of a Google Chrome extension that shows
pop-ups on respective websites in which crowdworkers can enter their feedback.
Based on interviews with five crowdworkers we identified three
core design goals that we needed to consider in further iterations.
First, the CrowdSurfer needs to be seamlessly integrated in crowdworkers’ everyday internet
surfing. That means it shall be noticeable but at the same time shall not distract the
crowdworkers too much from their primary task. Second, crowdworkers need to be in control of
the CrowdSurfer. That means the crowdworkers need a feature to control their boundaries
between work and non-work activities. And third, our crowd-feedback system needs to
achieve a high feedback quality in order to be valuable to feedback requesters
Following these design goals, we designed a working artifact of the CrowdSurfer.
After installing the CrowdSurfer via the Google Chrome webstore, crowdworkers need to log in by
using their ProlificID. First, they need to solve a demo task to learn about all the functionalities
of the CrowdSurfer. After clicking through the instructions, they have to provide feedback on
the color picker panel to finish the setup. Then, they can access the CrowdSurfer by clicking on
the icon in the upper right corner. Now, whenever they enter a website that offers a feedback task,
they see a feedback pop-up attached to an element. They can enter their feedback in
form of a star rating and a short text. The pop-ups also offer a menu in which crowdworkers
can see additional information on the task, such as the payment and the task requester,
and they can set a reminder for the task, in case they want to postpone it.
The pop-up also provides a feature to minimize it, in case something important is hidden by it.
Finally, crowdworkers can also reject tasks by clicking on the cross icon.
When they submit a task, they see how many tasks they have already solved on the respective website
so that they do not start searching for tasks when there are none available.
When they click on the CrowdSurfer icon in the upper right corner,
they can open a panel to manage the extension. They can click on the question mark to redo the
task or log out from the extension. They can turn off the search for tasks which means that
no information on them is tracked anymore, and they won’t be distracted by task pop-ups. Below,
they see an overview of their completed tasks per day, per month and on the website. On the bottom
they see the latest tasks they have submitted.
For our evaluation, we decided to compare the CrowdSurfer with a baseline which is a
simple feedback survey that shows crowdworkers a screenshot of a website element and asks them
about their feedback. We recruited participants for both treatments on Prolific. In the baseline,
we had 29 participants who participated in the survey and a post-task questionnaire
in which they were asked about their perceptions regarding task duration,
work flexibility and fairness of payment.
In the CrowdSurfer treatment, participants first had to install the CrowdSurfer extension. Then,
they had seven days to work on feedback tasks that were implemented on seven very popular websites,
like Wikipedia, Amazon, Ebay, and Youtube. After the seven days, they also received an
invitation to a post-task questionnaire on Prolific. 12 of our 34 participants
were willing to participate in an additional semi-structured interview, in which we wanted
to learn about the feedback process, their user experience with the CrowdSurfer and the impact of
the CrowdSurfer on their working conditions. Our results reveal that crowdworkers who used
the CrowdSurfer to provide feedback perceived the payment as significantly fairer and the
duration they needed for completing the tasks as significantly lower. There was no significant
difference in the perceived work flexibility. Interestingly, the feedback quality with the
CrowdSurfer was significantly lower than the feedback quality of the baseline treatment.
Our interviews showed that participants perceived the CrowdSurfer as simple
and fun to use and felt the feedback requests were very personal to them.
On the negative side, they perceived the requests as not specific enough. They
additionally hat security and data privacy concerns regarding the Chrome extension.
For the feedback process, they appreciated the seamless integration of the feedback tasks
into their normal internet surfing and felt more in the context when providing feedback.
On the negative side, they were sometimes interrupted by task pop-ups and silked
that they won’t find all available tasks. Crowdworkers also liked that the CrowdSurfer
made it feel like no pressure to them to work on tasks or reject tasks. They also liked that
they had less effort for background work, such as selecting tasks to work on. On the negative side,
some participants actively searched for tasks, which was not intended by us.
We assume that participants in the CrowdSurfer treatment perceived the payment as more fair
as they spent less time working on tasks and had less additional background work to do,
as they stated in the interviews. For the reduced feedback quality,
we found multiple explanations. The feedback texts in the CrowdSurfer treatment were much shorter,
maybe because the text field were smaller than in the survey. In shorter texts, of course,
less information is included. Another potential explanation is that participants in the baseline
were focused on providing feedback as this was their only task, so they spent more time and
effort on providing feedback. In the CrowdSurfer treatment, providing feedback was only a secondary
task, which participants wanted to complete quickly, and they did not put their main focus on
providing high-quality feedback. Maybe they also cared less about receiving the payment or not.
Based on our findings, we derived six design recommendations for designing
similar tasks for casual microtasking. Finally, we see three potential avenues for
future research with the CrowdSurfer. We assume that most promising one is the exploration of
further mictrotasks that could be solved with the CrowdSurfer. For example, the CrowdSurfer could
be used to collect feedback on image alt-tags in the web to increase the accessibility.
To sum it up, we designed the CrowdSurfer, a browser extension to collect in situ feedback
via casual microtasking. We learned that the collected feedback is of lower quality,
but Crowdworkers felt more in the experience when providing feedback and therefore perceive their
feedback as more relevant. Additionally, they perceived providing feedback via
the CrowdSurfer as less effort and fun to do. In our study, we demonstrated the feasibility of
integrating tasks into crowdworkers daily life. Thank you very much for your attention. In
case you have any remaining questions on our research, please just get in touch with us.