Hello. I'm Kelly McConvey, a PHD student at the
Faculty of Information at the University of
Toronto.
I will present our work on algorithms and
higher education. This work was completed
with Doctor Shion Guha and Doctor
Anastasia Kuzminykh.
The use of algorithms for decision making
in higher education is steadily growing
promising cost savings to institutions and
personalized service for students but also
raising ethical challenges around
surveillance fairness and interpretation of data.
To address the lack of systematic
understanding of how these algorithms are
currently designed, we reviewed an extensive
corpus of papers proposing algorithms for
decision making in higher education.
In this project we set out to answer two
key questions
what are the current trends in the design
of algorithms in higher education?
And how are human centered algorithm design
approaches used in the development of those
algorithms?
First, we'll review our data collection
coding and analysis methods.
Using combinations of the keywords higher
education, algorithm, predictive analytics,
and machine learning, we searched the ACM
digital library identifying nine thousand
thirty eight papers.
We removed four thousand six hundred and
twenty duplicates and screened four
thousand four hundred and eighteen titles.
From there, four hundred and thirteen
abstracts were screened using our inclusion
criteria and two hundred and thirty six
full papers were screened. Our final corpus
included sixty two papers for synthesis.
The included papers were peer reviewed,
published work containing an algorithmic
approach with the technical discussion of
the data used. They use student data within
higher education and were published between
twenty ten and twenty twenty two.
The papers were reviewed
for their statistical approaches or machine
learning models, target outcome, variables
and specific data input types. Coding
categories were developed from the results.
In order to analyze the papers in the
context of human centered algorithm design,
we also deductively coded each paper as
demonstrating the dimensions of theoretical
design, participatory design, and/or
speculative design.
We adopted Baumer's human centered
algorithm design framework to consider
how the design of algorithms aligned with
or led by educational theory,
how stakeholders were meaningfully included
in the data selection, algorithm design
model evaluation, or implementation
processes,
how the model was evaluated using
stakeholder feedback, and finally how the
researchers envisioned their proposed
algorithms being used in real world higher
education institutions and scenarios.
Next we'll talk about the results of our
analysis
the proposed models increasingly relied on
individual student data such as gpas,
enrollment, pathways, and learning management
system or LMS activity as input features.
Use of protected classes like age, race,
gender, disability status, has grown significantly.
Results indicate a trend away from rule-based systems
towards neural networks and deep learning methods.
This potentially leads to less interpretable and less explainable results.
Results show a growing trend
towards models developed to
predict student success, including grades and attrition.
These models are generally built
to support institutional retention efforts.
Our review also considered the design
of the study through the lens of Baumer's
human-centered Algorithmic Design framework
to identify which papers employ theoretical, participatory or speculative
dimensions in their design. No papers to our knowledge,
include all three dimensions and only one paper
included more than one element of the framework.
All feature categories, except institutional and student survey,
were accounted for in papers using HCAD strategies.
Interestingly, while three papers using
participatory design included participant
surveys in their design methodology,
they did not include surveys as feature sets.
Only one paper used student survey data in the feature set
in the design of a student services chatbot. The HCAD papers generally
targeted the more common outcomes of Grade Prediction
and Retention, though two papers were
focused on Pathway Advising.
All of the HCAD papers employed machine learning techniques.
Next, we’ll discuss the challenges
and opportunities identified in our research,
Three major needs emerged from the trends identified by our review
The need to establish valid and theory-informed ground truths.
The need to consider interventions within the algorithm design process.
The need for governance as algorithms move from identifying
and informing to decision-making.
We found a growing trend toward the use of Learning Management System
data as input features. Student interaction
with their web-based learning platform was used as a proxy for their engagement
in the course with an underlying
assumption that LMS activity like clicks on videos,
frequency of logins, and views of
course content is correlated with student
interest and participation in the material,
yet in none of the papers was this assumption
questioned or was educational theory presented to support it.
LMS interaction as a proxy for engagement,
and an indicator of academic success,
is then accepted as a ground truth within the algorithms.
LMS activity, though, can take many forms
and there was no discussion within the papers as to the
predictive power of specific types of activities.
The viewing of a video may not have equal engagement value
as participation in a forum discussion,
or infrequent but substantial logins may or may not
be as effective as frequent, brief ones.
Additionally, activity can be impacted by many factors, including
technological factors such as
interface design and privacy support and the combination of the
LMS itself, the course design, and the course type produce
different student experiences and influence student behaviors.
The ground truth of these models is that students
with less LMS activity are less likely to
succeed the output of which will trigger a
student level intervention and not a review
of the LMS, course design, or user experience
Beyond incorporating educational theory in
the design of algorithms, computational
scientists should allow for more
collaboration with education subject matter
experts and social scientists to analyze
the importance of these LMS
interactions critically.
More research is needed to understand the
relationship between LMS activity and
engagement. Additionally, researchers must
consider how the importance of the
individual features of their algorithms
aligns not just with target variables but
also with the greater goals and outcome of
the algorithm
Before 2018, research focused
primarily on designing algorithms to
predict student grades and retention.
The motivation behind this research is to
reduce attrition through the early
detection of students who are identified as
likely to fail,
allowing institutions to provide these
students with timely and focused
interventions.
Interventions vary from automated email
notifications, supplementary readings and
assignments, recurring meetings with
advisors, and counseling services.
Few of these papers however consider what
interventions exist, the efficacy of the
interventions, or the institutions legal and
ethical obligations to provide
interventions once at risk students have
been identified.
Additionally, intervention efficacy is
difficult to measure. We also do not find
any literature demonstrating that these
models have been used effectively within
higher education institutions. The
algorithms seem to be designed within data
science silos, with little consideration for
their role within the institution or as
part of the greater student experience.
An algorithm designed to increase retention
cannot only be evaluated by its performance
in predicting which students may not
succeed.
In other words, researchers should consider
the relationship between the specific
predictors used in making the prediction
and the intervention proposed to then
correct the prediction.
Even more concerning, however, is the move
from retention and grade prediction toward
pathway devising and admissions.
Algorithms designed for course selection,
program admission, and pathway advising have
the potential for more direct and limiting
impact on students.
Algorithms in higher education are moving
beyond the ability to simply provide
greater insight to institutions through the
identification of at risk students
but to actually make the decisions for them.
As we expect to see this trend continue and
the use of algorithms for decision making
expand beyond its current scope, our results
suggest the need for more governance within
higher education institutions.
Our review indicates that certainly
technologically possible to leave these
student decisions in the hands of
algorithms but whether it is desirable for
algorithms to shape students academic
careers without the oversight of a human in
the loop, remains a legal, ethical, and
pedagogical question.
We encourage you to read the full paper and
look forward to any questions or comments.
Thank you.