Hi.
I'm Emily Hastings, and today I'm going to be talking about a work I completed with my
colleagues at the University of Illinois titled, "LIFT: Integrating Stakeholder Voices into
Algorithmic Team Formation."
Many of you in the audience have probably worked on a team project before, and for the
instructors, you may have incorporated team-based learning activities into your courses.
You may have even used algorithmic team formation tools to form the teams for these activities.
What we hope you take away from this presentation is some insight into how you can incorporate
student voices more closely into this process.
The overarching question guiding our research is, "How should instructors form teams in
their courses?"
This is an active area of research, and three main approaches have become popular.
First is self-selection, where students form their own teams.
There's also random assignment, where the teams are formed randomly.
These approaches are both easy for instructors to implement, and do have their own strengths.
They have drawbacks as well, however.
For example, teams formed using these approaches are often overly homogeneous, and they can
lack important skills necessary for the assigned tasks.
In response to this issue, the third approach, criteria-based team formation, has become
popular.
Using this approach, instructors strategically select team members in order to achieve certain
team compositions.
Some compositions you might aim for include skill diversity, a balance of personality
types, a balance of genders, among many other options.
This approach addresses that weakness I mentioned of the prior approaches, although it can become
difficult to implement as course enrollments grow large.
For this reason, many instructors are turning to algorithmic team formation tools in order
to implement a criteria-based approach to team formation in their courses.
One representative example of a tool like this is the CATME Team-Maker tool.
For those of you not familiar with it, in general, the way it works is that instructors
select which criteria they want to use to form teams in their course.
The tool then distributes a survey to students, in order to solicit a response relative to
each of these criteria.
For example, you can see the survey here is asking students what gender they identify
with, and what their strongest skills are relative to the course project.
The instructor then sets weights for each of the criteria, which determine how important
they are relative to each other.
The tool then forms teams based on this configuration.
The instructor is able to rerun the algorithm as many times as they like until they are
happy with the teams.
An issue with this class of tool, however, is that they tend to assume the instructor
is the only one who should configure the inputs to the algorithm.
This excludes students from having input, despite the fact that they have potentially
useful localized knowledge about their team experiences, and the fact that this process
affects their learning and experience with the course.
Prior research has shown many potential benefits of increasing stakeholder knowledge and control
over algorithmic processes like this.
For example, increased control can prevent users from viewing the algorithm as a "black
box," which can decrease trust in the system.
It's also been shown to increase satisfaction and acceptance of these systems.
And finally, giving students more control has shown that they will take greater ownership
over group problems, which for example can help with problem solving.
In response to this issue, we developed the LIFT workflow, which stands for "Learner Involvement
in Forming Teams."
Using LIFT, the configuration of the team formation tool is crowdsourced to the students
in the course, in a three step process.
In the first step, students discuss and propose the criteria that they think are important
for forming teams.
They then vote on which of the criteria that they discussed they think should actually
be used in the tool, and finally they configure the weights that I mentioned for each of these
criteria.
This workflow is based on prior work in crowdsourcing and collective intelligence, and in particular
learnersourcing, which is the application of crowdsourcing techniques in learning environments.
In order to evaluate LIFT, we conducted a mixed-methods, between participants experiment
in five courses at our university, which was approved by our IRB.
We also conducted interviews with 18 students and 6 instructors of the courses, in order
to get richer qualitative data about the workflow.
For our experiment, we compared two conditions, the first of which was the Learner condition,
where students were formed into teams using the LIFT workflow, as I discussed.
They discussed which criteria they thought were important-- in our case, this was held
in the online platform Piazza.
The research team then extracted the criteria from this discussion to prepare a survey for
students to vote.
And finally, they configured the weights for each of the criteria in the team formation
tool while completing the survey that I showed you.
In the second condition, students were formed into teams using the traditional instructor-led
approach, where the instructor configured the team formation tool without any input
from students.
Our experiment had a number of dependent variables, which in general were related to performance,
satisfaction, and perceptions of students' agency.
For example, we looked at project grades, which were collected through regular course
grading, as well as a number of measures collected through a survey students completed at the
end of the semester.
Some of these were perceived performance, their satisfaction with the team formation
process, and how much of a voice they thought they had in the process.
Now for the results.
Our first research question was, "Which criteria do students select when they have the chance,
and how do these configurations differ from those that the instructors create?"
In total, students discussed 75 criteria across all the courses, 48 of which were newly-proposed.
And by that I mean they weren't included in the predefined list of criteria included with
the tool.
For example, one of these newly-proposed criteria was organizational style.
In general, these criteria fell into three broad categories, the first of which was team
management, which included things like leadership role, teamwork experience... in general things
that were related to the functioning of the team, or how individuals function on the team.
Second was academics, which included things like GPA and software skills, and finally
aspects of identity such as gender or personality type.
We saw that the voting phase of LIFT eliminated all of the criteria that could be considered
less serious.
For example, in one course, a student had proposed favorite color as a criterion, but
this was voted against by the majority of the class, who favored things that they thought
were more relevant.
In general, we saw that the most popular criteria among students were related to scheduling,
skills, work habits... things that they thought would contribute to completing the project
more efficiently or conveniently.
And the least popular criteria among students were those related to aspects of their past
or identity that they didn't have present control over, for example, things like gender,
GPA, race.
Students either disregarded these criteria or actively voted against them when they were
brought up.
In contrast, the instructor criteria were all selected from that predefined list in
the tool, and in general, their configurations prioritized learning and long-term success
over the more immediate minimization of conflict.
For example, one of the instructors discussed selecting GPA so that high-performing students
tended to be placed together.
This was in the hope that students would have the opportunity to work through disagreements
with each other and have that sort of growth opportunity.
Our second research question was how students perceive their agency when they have this
increased input into the process.
We saw that across conditions, students thought it was important to have a voice in the team
formation process.
This was a median of 6.0 out of 7 on that survey that I mentioned.
And we did observe that the median agency score reported by students in the Learner
condition was higher than that in the Instructor condition.
This was 5.0 vs. 4.0.
However, this difference was not statistically significant.
And this was surprising, because the students in the Instructor condition really hadn't
had that much control over the process, so it was interesting that they rated their agency
so high.
One possible explanation for this finding is that there is a distinction between participation
and choice that our survey item may not have captured clearly.
And what I mean by that is that students in the Instructor condition participated in the
team formation process by completing the survey in the tool.
However, they didn't have the same choice or control over the process that the Learner
condition students did, in actually configuring the tool and determining what was on the survey.
Future work can look into this distinction more closely.
And now for some of our interview data related to this question.
Some of the strengths of LIFT that students identified were that it can provide insight
to instructors who are often disconnected from the student team experience.
It also contributed to an increased sense of ownership over the team and its functioning.
Some weaknesses that were identified were that students are often not experts on what
makes a good team, or on the requirements of a particular course.
There were also some concerns of gaming the system that were raised.
For example, students reported hearing of groups that had conspired to be placed together
by setting very specific schedule requirements, and there were some concerns over the fairness
of this.
Our third research question was, "How does allowing the students to select criteria affect
their team experiences, in terms of performance, satisfaction, or any of our other measures
that I mentioned.
We saw that across conditions, our outcome variables were very high, so students in general
were performing very well and were very satisfied.
And in fact, we saw no significant effect of the criteria selector on any of our outcome
variables.
In terms of potential explanations, it may be that the specifics of a criteria configuration,
such as whether you select a 4 or a 5 for a weight for a certain criterion, may not
be the most important factor in determining these outcomes.
Instead, it could be that explaining the benefits of the team formation tool to students could
contribute to their performing well or being satisfied.
And for more on this, I invite you to check out some of our prior work which was published
at CSCW.
Finally, our last research question was, "How do instructors perceive this transference
of agency to students, and what do they learn by participating in the LIFT workflow?"
We saw that instructors generally found student choices to be reasonable overall, and this
included in some cases confirming doubts they had about their own configurations.
For example, one of the instructors had been unsure about including GPA, and when he saw
that students hadn't put it on their list, he was surprised and wished he had known about
this earlier.
On the more negative side, some instructors had doubts about irrelevant criteria, such
as programming skill in a course where programming wasn't that big of a deal.
They brought up some of the same gaming concerns students had mentioned, and additionally they
were concerned about students excluding certain important criteria from the literature.
For example, this instructor had seen that in her course, students hadn't selected gender
or race for example, and was worried about some biases that this might bring up.
When asked whether they would adopt LIFT in their courses in the future, three of the
instructors said that they would use the process as-is, while a fourth was willing to integrate
the most popular student criteria into his own configuration.
And these instructors cited increased responsibility, and motivation, and a sense of ownership in
students as reasons for this decision.
The remaining two instructors were more reluctant to adopt LIFT, either due to this issue of
excluding criteria that I mentioned previously, or due to very large course sizes.
For example, one of the instructors had a course in the upper 600s, and was worried
about satisfying everyone's preferences.
So, implications for instructors.
The most important thing we hope you take away from this is that it's possible to incorporate
student input into the algorithmic team formation process without adversely affecting grades
or team experiences for students.
And in addition, it's not necessary to complete the entire LIFT workflow every time you form
teams.
You can do a simplified version in order to make the process more convenient.
For example, students could vote only for the weights of a specified set of criteria,
rather than proposing the criteria themselves.
Additionally, you could integrate student- and instructor-chosen criteria into a single
configuration, which would give instructors a chance to include some of these criteria
that students tend not to, and help protect the voices of minority students who might
otherwise be drowned out by the rest of the course.
For tool designers, you can start thinking about ways to incorporate features that delegate
some algorithmic control to students.
For example, this could be through surveys or discussion forums.
You might also think about ways to start to reduce some of this possible manipulative
behavior that we saw brought up by our participants.
For example, this could be done through reducing reliance on data that's self-reported by students.
In conclusion, this work contributes a deeper empirical understanding of the effectiveness
of including student voices into the team formation process.
We share a learner-centered workflow that instructors can use to find out what criteria
are important to the students in their own courses, and finally we offer practical implications
for how designers of team formation tools can give stakeholders more control over the process.
I'm happy to take any questions over email.
Thank you for listening.