Hello everyone, my name is Daniel Davison and I’ll be presenting our study that took
place in schools over the duration of four months.
In this talk I will briefly discuss our approach to setup and conduct such a longterm study
in a real world context. I will also highlight some of the main results.
For all the details I refer you to our paper.
The study was conducted in the context of the EASEL project, where we develop a social
robot to support interactive learning tasks for children in primary school, aged 6 to 9.
These tasks use tangible materials and revolve
around investigating problems from the domain of physics.
For example, this picture shows a balancing scale task, which is used to investigate the
moment of force. Children work with our tasks unsupervised,
they start and stop interactions at their own initiative and can progress through tasks
at their own pace. Task instructions are displayed on a tablet
device, which is also the primary way to provide input to the system.
We use inquiry learning to offer structure to children while they work on a task.
Children conduct iterations of small experiments, each of which consists of several steps.
First they make preparations for the experiment, like getting the materials ready.
Then the make a prediction about the outcome of the experiment.
After doing the experiment they observe what actually happened and draw conclusions about
whether their prediction was correct or not.
The child conducts small experiments with the learning materials according to inquiry
learning principles. Meanwhile, the robot follows their actions
through embedded sensors in the task and receives the child’s responses through the tablet
The robot then offers personalised social
support to the child while they progress through the task.
So what does it mean to conduct these type of studies in the wild?
When doing research in our labs, we usually have full control over the environment and
other factors to design our interaction. However when working in a context of a school,
things are very different. We involved the main stakeholders during the
design of our study to figure out how our learning task could blend in with the school’s
regular way of life. With children and their parents we discussed
what it meant to participate in such a study, and we gained insights into their expectations.
With school directors we ensured that our task fell under the school’s curriculum
and educational vision. Finally with teachers we discussed how the
task could fit in their day-to-day activities. We found there were restrictions on the environment:
there was usually a small designated space which we had little control over.
There were also restrictions on the interactions. We used headphones to ensure that interacting
with the system was not distracting for other children working in the classroom.
Teachers also indicated that there should be a maximum session length, to prevent children
from spending all their time playing with the robot.
Finally, we were of course bound by the school’s schedule: the system could not be used at
certain moments of the day and during holidays and other events.
Since we were interested in capturing spontaneous interactions, children could work with our
system without supervision from a researcher or teacher.
Children initiated a session by scanning their RFID badge.
The system then loaded their personal progress and continued where they had previously left off.
off. We designed our interactions and learning
tasks to be robust so that children could work independently without assistance.
The dialogues included built-in repetitions, elaborations and explanations, which were
triggered upon request or after a timeout. Interactions were designed around reliable
input methods, like a touchscreen, in favour of more error-prone methods like recognising
speech or facial expressions. We included fallback options for non-critical
system components to ensure that children could continue working on the core task in
case of malfunctions. Maintenance and repairs were scheduled outside
of school hours to minimise interruptions.
How did we collect useful research data, given all these limitations?
We focused on automatic data collection while children worked on tasks.
We used embedded sensors in the learning materials to generate logfiles of their actions and
interactions. We also used in-task self-reports to capture
their subjective experiences. This data allowed us to follow development
of interactions over time
Three variants of our system were deployed in three schools for a duration of around
four months. One system was with a robot, two others were
without a robot. 61 children played with two different learning
tasks and progressed through several levels of difficulty.
They completed a total of around 1900 assignments in more than 500 sessions, with an average
session length of around 6 minutes and 15 seconds.
When we take a look at the interaction frequency graphs we can see some patterns develop over
time.
In the first few weeks we found that children engaged frequently with the system, which
tapered off over time. Throughout the study there was no complete
abandonment: we saw a sustained frequency of engagement for the duration of the experiment.
Additionally, we saw moments of renewed engagement after introducing a new task, and after children
returned from their holidays. From our logbook data we could follow how
children progressed through the difficulty levels of each task.
In the first task, with a total of 8 levels, all children initially started at level 3.
After only a few sessions we found that many children had already reached the highest level
of difficulty, which they continued repeating for several more sessions.
For the second task all children started at level 1 instead, and an additional level of
difficulty was added, bringing the total to 9.
We saw that children now took more sessions to exhaust the available levels.
From their self-reported measures we found that children were successful in self-navigating
through the available levels of difficulty at a appropriate pace:
Assignments that they got wrong were rated as harder than those they got correct.
When getting answers wrong, they were more inclined to select a lower level or stick
with the current level. And when getting an answer correct they would
often select a higher level. Overall, children rated all levels to be of
similar difficulty, suggesting that they indeed progressed to higher levels when they were
ready to do so.
In conclusion: Our robot with limited interaction capabilities,
and a limited amount of learning content was sufficient to maintain children’s engagement
throughout four months. Furthermore, through a mix of embedded sensors
and in-task self-reports we could follow how children interacted with our system and how
this developed over time. We have given a first impression of the feasibility
of conducting unsupervised research in the wild over longer durations, opening up the
possibility to do more of such studies in the future.