Hello everyone in the cyberspace of UMAP 2020 from the sunny Trondheim in Norway.
I am Zacharoula Papamitsiou and I would like to thank you all for being here today, for this presentation
and especially the organizers for all the efforts they have put on making this event come true.
Today's presentation is on one of the remaining elusive issues in adaptive online learning
and specifically on the determination of the defendence structure between learners response-time and knowledge mastery.
Together with my colleagues Kshitij Sharma
and Michalis Giannakos from the Norwegian University of Science and Technology.
we conducted a study to explore the nature of the relationship between those two variables,
which we already knew that it's not linear.
We all agree that the adaptive learning environments are in the
epicenter of the technology enhanced learning research
and community.
And in order to adaptively scaffold the learners throughout the learning process
it is a prerequisite for the learning systems to make decisions by considering the learner models.
Essentially, the learner models are estimations of learner’s current states
and approximations of next states based on the available observational data from their activity
and behavior within a learning environment.
The prevailing approaches for modeling learners knowledge mastery can be distinguished in two
generic families. The knowledge tracing techniques
such as Bayesian Knowledge Tracing
and the logistic models such as Performance Factor Analysis or Item Response Theory.
However, in any ability or mastery measure both the result of interacting with a task
and how long it took to reach the result, needs to be considered. In other words the observed response
and response time jointly affect mastery.
Learning more about the shape of the relation between those two variables is
expected to allow us to study in more detail the impact the time spent on the response has on the response accuracy,
which in turn can be informative for the possible underlying processes.
However, determining the dependence structure between response time
and correctness of response is not a trivial task and still remains an open issue.
Lots previous work has been done on this topic with most of the
approaches employing a combination of response time
and response accuracy to improve the overall accuracy of the learner models, without,
though exploring the relationship between those variables.
The early approaches to understand and determine this relationship,
mostly utilized response time to directly predict the correctness of next response.
Furthermore approaches such as the z-scored response times or squared response times were also tried.
Overall the approaches employed in adaptive learning settings indicated that the relation is not linear.
Further to that in adaptive assessment and testing settings.
the most popular approach is the hierarchical framework,
which has as a major assumption that response time follows a log-normal distribution.
In that model the conditional independence of response accuracy
and response time is assumed, given the overall speed and ability.
And while the correlation between response time and ability can be interpreted within a test taker,
however, this assumption might be violated in the correlation
between the variables across the test takers.
To address this issue it has been proposed to explicitly model the residual dependence between
time and accuracy by incorporating the effects of the residual response time.
The exploratory findings from several studies revealed a very stable curvilinear dependence when the residual dependencies are considered.
However, this approach appears to have overfitting problems describing
a random error rather than a true underlying relationship.
And again in these settings results indicate the nonlinear
dependence structure between the variables
Overall, previous works converge on the fact that the dependence structure between response time
and knowledge mastery needs to be determined
and the objective of this study is to explore the degree of
association between those two observations both in learning and in assessment settings.
The dependence between two random variables is contained in their joint distribution.
Previous research results showed that we need to go beyond common measures of linear dependence.
So here we adopted a copula based approach.
The idea behind the concept of copulas is to separate the joint
distribution function into the dependence structure and the marginal behavior.
In a sense copulas are functions that “couple” the marginal distributions
to the corresponding joint distribution functions.
The theoretical
foundation for the application of copulas is provided in Sklar’s theorem which
in brief claims that any multivariate joint distribution can be written in terms
of univariate marginal distribution functions
and a copula function that describes the dependence structure between the variables.
Here, we consider only response time
and knowledge mastery and thus we restrict ourselves to the bivariate case.
Most of the copulas are parametric
and there are basically two types: the copulas of normal mixture distributions and the Archimedean copulas.
The first type refers to the elliptical copulas such
as the Gaussian and the students t copula,
or for simplicity the t copula
and the second type refers to copulas that have simple closed forms such
as Gumbel and Clayton copula.
If the joint distribution is well represented by an elliptical distribution
for example Gaussian or t , then the dependence structure is linear.
However, it has been argued that between response time and knowledge mastery there is a non linear dependence.
Outside the world of elliptical distribution, the use of the linear correlation
coefficient may induce misleading conclusions about the dependence.
There exist standard and useful dependence measures solely related to the copula
and not on the margins.
There are for instance rank correlations like Kendall’s tau τ
and Spearman's rho ρ or tail dependence coefficients.
The tail dependence coefficients are measures
of extremal dependence that quantify the dependence in the upper
and lower tails of a bivariate distribution of two random variables.
Note, that The Gumbel and Clayton copulas can only capture one side of the tail dependence
and they cannot display negative dependence.
And if this is the case, then these copulas would not fit.
Therefore, these copulas should be “rotated”
for 90 degrees, 180 degrees
or 270 degrees and then applied again.
To illustrate how the dependence between response time
and knowledge mastery can be investigated using the copula based approach,
we applied this method to
four datasets, two coming from intelligent tutoring systems and two from adaptive assessment systems.
And we initially plotted the datasets to gain a first
intuitive insight about the observation their distribution
and the existence of extreme values.
Next we sliced the student population data using the 10th and 25th percentile.
but any other fine-grained splitting can be utilized
and we splited both the response time and knowledge mastery
into smaller datasets of sub-populations,
in smaller blocks in order to
explore the bivariate dependence structure for each subpopulation separately.
After that for each block, we fitted the theoretical models
and concluded to the empirical models, choosing the ones with the lowest AIC.
and the strength of the decided copula for each block
was measured by the respective coefficient,
which was lambda upper for Clayton types, lambda lower
for Gumbel types, Kendall’s τ for Gaussian, and Spearman's rho for t.
The figures depict the fitting results for all datasets in the 25th percentile for simplicity reasons.
In the figures, each color corresponds to one of the detected copula families,
In all figures, 2 corresponds to survival Clayton (which is Clayton rotated for 180 degrees),
6 corresponds to survival Gumbel,
7 corresponds to rotated Gumbel for 90 degrees, 8 corresponds to rotated Gumbel for 270 degrees,
and 9 corresponds to Gaussian .
We can note that the t copula was not detected that all in data
and the Gaussian was detected in very specific blocks.
For the learning data the overlap of the detected copulas
across the datasets was almost 94%
when we only consider the generic copula family
and it was around 70% when we take into consideration both the direction of the dependence
and the copula family.
The respective overlap for the assessment dataset was almost 63% despite the dependence direction.
The table summarizes the results
for the tail-dependence coefficients for each block
for all datasets in the 25th percentile.
In this table the blocks correspond
with the respective ones in the previous figure from bottom up
and left to right and for example block 1 - 1 corresponds to low response time low knowledge mastery,
while block 4 -4 corresponds to
high response time high knowledge mastery.
For most of the learning data the copula reveals a strong
or moderate dependence structure
and which means that the tail-dependence coefficient is above 0.5
For the assessment data,
most of the dependencies were moderate.
Note that the number of students in each block is included to showcase that even
for smaller partitions of learners the copula based method can map a bivariate function
for joint dependence between response time and knowledge mastery.
Overall out of the ten theoretical Copulas, five of them were empirically
detected across datasets. From the elliptical Copulas that imply a linear
dependence structure the t copula was not detected at all
and the gaussian copula was detected in specific blocks,
and the nature of the underlying relationship between the variables of interest is nonlinear,
but some linearity is detected as blocks in specific percentiles follow the linear behaviour.
Moreover in each context the detected copulas
for your generic patterns across data sets for that context.
In the terms of knowledge Mastery
and response time those Copulas shows that the co-movement of those
variables in the tails of the distribution follows specific patterns,
and the above-mentioned copulas are utilized to model those patterns.
Furthermore the overlap of the detected empirical
patterns between the different datasets was high
the dependence was strong or moderate across datasets.
The approach allows for nonlinear dependence detection and characterization
for the identification of the exact underlying patterns of dependence
and captures both positive and negative dependencies.
It is purely data-driven without over fitting the data.
It is parametric without assumptions on the underlying distributions,
and it is not affected by the differences in the marginal distribution.
In terms of response-times and mastery, the amount of time used by the learners to respond to a task
and their knowledge mastery is reflected in the patterns: according to the pattern detected
The Learners are located in one of the blocks and this can guide the adaptation mechanism and strategy.
Furthermore, although the dependent structure is non-linear, some linearity is detected
as a gaussian copula is selected for mapping in specific blocks.
And the detection of this copula is in line with the High Speed high Stakes rule,
mostly detected used in adaptive testing settings.
However, here it is stronger in the learning data indicating
that some Learners subpopulations can be modelled
using the hierarchical approach. Still the rotated Gumbel
and Clayton copulas
allow for capturing both positive and negative dependencies
and as such, they can be utilized in the study of rapid guessing.
Last but not least the method allows the detection of dependence structure
even for extreme values in the data set. Considering the contemporary massive open
online courses in which the populations at the tails of the distributions might be thousands
the copula based approach can be scaled up
and allows for detecting the nature of association of response-time and
knowledge mastery for those students, and open the path
for designing adaptive systems
for such environments. There were also some limitations. Specifically we
did not consider the characteristics of the tasks at all,
which have been found to affect both response time and response accuracy.
And these characteristics can be employed
as exogenous predictors, as covariates in future explorations.
In addition the assessment data sets were relatively small compared to the bigger learning datasets,
and some of the dependencies there were weak. Analysis of bigger assessment datasets
is as such, required.
To conclude as a next step,
we are planning to incorporate the information we gained from this study regarding
the copy of families into the learner models to drive the adaptation
and to evaluate the accuracy of the new models.
So thank you for your attention.