Hi, my name is Jonas Oppenlaender and will present you our research on crowdsourced design
feedback on behalf of my co-authors Elina, Andres, and Simo.
At the University of Oulu, we run a Human-Computer Interaction course in which Bachelor students
undergo the full design thinking process, from coming up with ideas for a mobile application
to implementing paper and high-fidelity prototypes.
Students also evaluate the usability and user experience of the prototypes.
Due to the pandemic, evaluation of paper prototypes is of course difficult to do, and this inspired
our research on crowdsourced evaluation of user interfaces.
The research is part of a larger doctoral project in which I investigated stakeholder
experiences of creative work on crowdsourcing platforms.
Crowdsourced design feedback is not new area of research, and several prior studies established
the feasibility of using crowdsourcing for design feedback in the classroom.
Our study is similar to the study by Wauck et al.
We set out to reproduce and extend these prior findings.
Students in our course were asked to peer evaluate the designs of other students.
Each of the 112 students received a feedback package with feedback from 9 other students
and 9 crowd workers.
Feedback was collected online and also included one open-ended item.
The results from the final questionnaire show that students found peer design feedback to
be significantly more specific,
actionable, explanatory,
and relevant to the task of designing a mobile application.
Peer design feedback was also perceived to be of higher overall quality and students
were overall more satisfied with the peer feedback.
We also investigated how students perceived the fairness of the feedback and found that
the students’ perception of fairness was shaped by 5 dimensions:
- agreeableness, - valence (i.e., affective tone),
- usefulness, - diligence, and
- credibility.
The latter two dimensions relate to the students’ preconceived notions about the crowd worker.
This notion was also evident in how the students valued the feedback for the given task of
designing a mobile application.
The students’ estimate ranged from zero to 20 EUR, with a mean of 5 EUR.
We divided the students into two groups:
One group did not find crowdsourced design feedback useful, and therefore did attribute
any value to the feedback.
We called this group hardhats because of their down-to-earth and matter-of-factly way of
valuing the feedback.
The other group - called Bungaloos - had a more bonafide and naive approach to valuing
the feedback’s monetary value.
In this group, we found an interesting mismatch between the usefulness of the design feedback
and the monetary valuation.
Students from this group often acknowledged that the crowdsourced design feedback had
little usefulness, but still attributed a positive value to the crowdsourced feedback.
This is because students from this group thought of the crowd worker as a professional and
expert who must be paid a handsome amount of money, even if the feedback is not useful.
Our paper consolidates a number of recommendations for using crowdsourcing in the context of
the classroom.
For instance, to avoid the mismatch mentioned earlier, it is important to set the right
expectations, both on the side of the worker and the student.
To summarize, we presented an empirical case study of crowdsourced design feedback in the
classroom, focusing on how feedback is perceived and experienced by students.
We explored the how students valued the crowdsourced feedback, and the paper provides some reflections
and recommendations for using crowdsourcing for teaching.