Hi, I am Fiorenzo Colarusso
And today I'm going to present you My paper titled
A novel Interaction for Competence Assessment. Using micro behaviours
extending CACHE to graphs and charts
before starting. I just wanna thanks my colleagues. Peter Cheng
Grecia Garcia Garcia. Aaon Stockdill, Daniel Raggi and Mateja Jamnik
The motivation behind this work is double because first, we are
introducing TIPS (Transcription with Incremental Presentation of the Stimulus)
as a new method for competence assessment with data visualisation.
And then, we are introducing a new CPM GOMS model for the graph comprehension.
OK, so before moving next, let me give you some brief ideas about how we designed the tips.
So, if you're learning something, you are basically building a chunk hierarchy
And if you're doing a task such as copying data visualizations
you're getting micro-behaviours associated with the processes of the chunk hierarchy
micro behaviours has ,two main constraints
the working memory capacity, which is about four chunks
and then the serial processing Because, we can only copy one element at a time
micro-behaviours also give us potential measures of the competence such as
measures of the contents such as, temporal chunk-signals
which are pauses between actions which occur in a range of
100 milliseconds to two seconds and,
chunk size measures, which allow to quantify the size of the chunks when we are doing a task.
moreover, since we are measuring the competence
we're also measuring the, amount of learning
ideas about chunking and micro behaviours have been applied by Cheng (2014)
to design CACHET competence Assessment by Chun Arch Evaluation with the Transcription Task,
a computer based assessment which allows to record the micro-behaviours
during the coping of learning material. The CACHET approach has been
successfully applied to measure competence in several domains such as
algebra, programming and English.
We also applied the typical CACHET approach in our previous experiment with data
visualisation. But unfortunately, we got a lot of
interferences in the chucking signals due to the different drawing
strategies produced by our participants. The limitation
reported, in our previous study, have been addressing,
designing TIPS, transcription with incremental presentation
of the stimulus where we where we were able to standardise the
drawing strategies and, uh, amplify the chunking
signal.
So let me introduce you now how we designed the coping task
and how TIPS works.
Tips can be done using Interactlog, which is a special software written in our
lab and a laptop combined with a graphic tablet and the
keyboard. The tips User interaction has been designed around the two
main phases the familiarisation phase at the very beginning,
where the participants can look at the stimulus for three or five seconds
based on the complexity
and the visualisation phase. Uh, where the participants, um uh,
have to visualise several parts from the world stimulus by
pressing the view key, which correspond to the right key on the
keyboard. And, once they have displayed the
amount of information they are able to copy in one go.
Um, they can start the copying by, uh, by pressing the draw
key. Uh, which is the down key on the keyboard
applying tips. We are able to record the several measures of the
contents, such as, the participant chunk size,
uh, through the viewing time, the viewing episode
size and the drawing time. But also we are able to
record the temporal signals represented
through the View Pause, which is the pause between
successive view key
and the Draw Pause
which is the Pause before pressing the Draw key.
The the tips design has been tested over four main
hypotheses.
So the first three hypotheses aim to test, the participants chunk size
over the stimuli,
with each chunking measure reflecting a bigger chunk size for
familiar stimuli.
Uh, otherwise, with our fourth hypothesis, we want to
test the different duration of the view and the,
on familiar stimuli
with the view, uh, being shorter for familiar
stimuli than the Draw Pause
for this experiment. We recruited six participants, uh,
with the stem background. And before joining in the lab, they
completed the graph familiarity questionnaire.
As you can see from the small standard deviation, Uh, all the
participants, uh, approximately share, uh, the same level
of familiarity with the individualizations.
These are the stimuli. We use it for our experiment. We have, uh, a A
which is a familiar stimulus for everyone. A Rey figure, which
is, an unfamiliar stimulus. One simple line graph. One
simple bar chart, one complex line graph and one complex bar
chart.
So let me show you the results.
As you can see through the box plots, all the three hypotheses have been
confirmed with the Rey figure having a smaller chunk size
than the house and the data visualisations reflected to, uh,
less consecutive key prices and a shorter drawing
time and a shorter viewing time.
We also confirm with our four,
because the median value for the view pause was smaller
than the for all the familiar stimuli.
And, the only exception is represented by the
Rey figure because in this case, the median value for the view
was longer than the
in order to explain the differences in the post duration, um, we
also conducted an analysis of variants using, uh, a
Two way ANOVA
So the statistically significant main effects and,
the two way in interactions effect revealed that both in the
individual factors and their interactions, uh, were able
to explain, uh, the post durations. Only
the three way interaction factor, uh, wasn't a significant
to summarise. In recent tips, we we address some of the limitations
we found in our previous study standardising the rowing
strategies and reducing the noise for the chunking
signal. we also added that the viewing gap is of
size and the viewing time to the catalogue of available chunking
measures in CACHET
Moreover, we demonstrated that, pauses during the visualisation
phase, can be also used to reveal recognition
processes, and the of boundaries
in this section, we are going to discuss the cognitive task analysis using CPM GOMS
model in order to better understand the cognitive processes
required by TIPS.
So particularly we build the CPM GOMS model in order to explain the
participants behaviour reflected to the view
which is the pause between, uh, two view keypresses and the draw pause
which is the pause before pressing the draw key.
to do that, we use the CPM GOMS
model, which is a particular model, uh, which allows to combine
together cognitive, perceptual and motor
operators using as a framework the model of human
processor.
We designed our model around the main goal, which
is coping and loops in the form of decision
boxes. Uh, representing the sub goal, uh,
through the view loop, uh, which is necessary to display the next
part of the stimulus and, two sub sub
goals. the loop
required to
Comprendre working memory loop required to, uh,
refine, uh, the mental representation. Uh,
before copying the stimulus,
the loops are able to explain and predict the
differentiation of the view. And the
with the view po having less loops over the stimuli than the Dr
uh,
except for the figure where the number of, uh,
loops for the view was bigger than for the,
uh, more over the the synthetic data produced by our model
represented by, uh, the black lines. Uh, in
both the pictures, um, are able to predict the DR
Duration with mean absolute percentage error
of 10.42% and the
duration with a mean absolute percentage error of
9.34%
to summarise. We can say that, Um uh, our model, uh,
it's able to predict the variability of the usual interactions across
stimuli of different complexity and familiarity.
And, uh, moreover, thanks to the introduction of reasoning
operators, uh, our model was able to deal with the problem
solving nature of our task.
Thank you for the attention. And, uh, every question is more than
welcome.