In modern online learning
instructional videos are a common way to deliver content.
There are many different ways to format an instructional video and
you don’t need to look further than this presentation to see an exceedingly common style:
the picture-in-picture ‘talking head’
where an instructor or a speaker can be seen on a small portion of the screen alongside learning content.
Now, how does the visual presence of the instructor
impact learning and other related outcomes in viewers of the video?
What if the video is formatted as a dialogue between an instructor and a student?
These are the questions that we explored in the paper presented here:
"Instructional video design: Investigating the impact of monologue- and dialogue-style presentations",
authored by myself, Bridjet Lee, and Kasia Muldner.
And we're from Carleton University in Ottawa, Canada
So let's first talk about the three different formats of interest.
Many instructional videos do not show the instructor,
they show only the instructional content, and the instructor narrates these slides.
Some videos include a picture-in-picture video of the instructor,
alongside the learning content shown on the slides.
And one potential benefit of being able to see the instructor
is that learners are given visual social cues.
These cues can take the form of facial expressions, gestures, or tone of voice
that can motivate learners or guide their attention.
In contrast, the visual presence of the instructor has the potential to distract learners from the learning content,
causing them to miss information
or have to split their attention between the
instructor's video and any diagrams or text being displayed.
Studies comparing these two formats
have reported mostly null effects in terms of learning outcomes.
Subjective measures of the learner experience have been mixed,
with some studies reporting that
viewing the instructor or being able to see the instructor
is more helpful, more motivating, and more enjoyable than videos with an invisible instructor,
and other studies conclude that
the instructor video is distracting, unhelpful, and more cognitively demanding on learners.
A third approach involves showing an instructor and a student
talking about the learning content as a dialogue.
This is a less common format but it is gaining traction.
Some researchers propose that dialogue promotes learning
because there is a student involved - a student visible -
and the student is someone with whom observers can empathize
and see as a positive role model.
And indeed, some studies have found dialogue to be more effective for learning outcomes than monologue presentations
and other studies have reported null effects
So, it's still an open question as to which format is better.
Our study adds to existing work on instructional video design
by collecting additional measures relating to visual attention and
other affective factors in a single study
and using Bayesian statistics to provide additional evidence for the results.
Bayesian statistics have an advantage over frequent test statistics
because they provide evidence for not only the alternative hypothesis
but also for the null hypothesis.
So our work was guided by three broad research questions:
the first being, How does delivery format (instructor visible or not, and monologue or dialogue)
impact visual attention?
This question focused on comparing where participants were looking while watching instructional videos.
Number two, Does delivery format impact cognitive load and learning outcomes?
And three, how is- how are psychological factors impacted by delivery format?
And by these I'm referring to
certain characteristics specific to an individual observer's mental state and beliefs,
and I will cover those in just a moment.
In order to answer our questions, we designed a study with three conditions:
so there was a Control condition,
in which an instructor narrating the slides of instructional content could be heard but not seen,
a condition labelled Monologue in which the instructor narrating the slideshow could be heard and seen,
and a condition labelled Dialogue in which the slideshow was narrated in the form of a dialogue
between an instructor and student,
both of whom could be heard and seen.
We collected the following measures.
First, we used an eye tracker to compare- to capture data, rather,
on where participants were looking, for how long, and how frequently
while watching the instructional videos.
Second, we measured learning by
calculating the gain from a pre-test to a post-test on the learning material
presented in the instructional videos.
And third, we measured four psychological factors,
and these were cognitive load,
interest in programming,
social presence indicated by interest in the instructor,
and self-efficacy, or confidence, in programming.
The instructional videos used for each of our conditions were created in-house.
So the content of the videos was a scripted introductory lesson to programming in the language Python,
and we filmed two versions of the lesson to form our conditions:
so first, the instructor and the student were filmed together going through the lesson
this became the dialogue condition.
And then the instructor was filmed again separately
going through the lesson as a monologue,
and this became the base for the control and monologue conditions.
These recordings were then synced to screen recordings that included mouse gestures and animations.
The total amount of content for each condition was about 25 minutes,
broken into two parts to avoid fatiguing participants.
After each part, we showed a review screen,
which was just a static slide of content that contained information presented in the video,
and participants could look at for as long as they wanted to.
With our materials in place, we conducted a laboratory study between-subjects
with 77 participants who had not taken any university-level programming courses.
After signing a consent form, participants were administered a knowledge pre-test
and questionnaire for self-efficacy.
then they were brought to the eye tracker and calibrated,
after which, they watched both parts of the videos
which were immediately followed by a review screen and a five-minute break in between.
And after that was done, they were administered a knowledge post-test
and questionnaires for all four of our psychological measures.
So our data were analysed using standard frequentist statistics,
namely Null Hypothesis Significance Testing, and Bayesian analysis.
I’ll present some select results here organized according to our research questions.
To obtain information on visual attention,
we analyzed where participants were looking as they watched the instructional videos.
We extracted data based on areas of interest, or AOIs,
which were drawn around the Python content in each condition,
and around the speakers in the Monologue and Dialogue conditions.
Visual attention was operationalized using dwell time and fixation count on each AOI,
and normalized to account for slight variations in length between conditions.
So accordingly, I’ll refer to these two dependent variables
as dwell time percentage and fixation percentage.
So, participants spent about 20% of trial time looking at the speakers in the Monologue and Dialogue conditions,
and this is shown with our dark grey bars here.
We observed that this came with a tradeoff in attention paid to the Python content in those conditions –
indicated here with the light grey bars.
Participants in the Control condition had a significantly higher dwell time percentage and fixation percentage
on that Python content than their counterparts in the Monologue and Dialogue conditions.
We confirmed these with-
these results with Bayes factors providing decisive evidence for the alternative model.
So given that, firstly, the speakers in the videos attracted visual attention,
and secondly, that this was at the cost of attention paid to the learning content - the Python content -
how were factors like cognitive load and learning impacted across the conditions?
There are three types of cognitive load identified in the literature:
extraneous load, intrinsic load, and germane load -
and so we checked if each video type affected these and the results are displayed here.
Our results show that these were not impacted by the type of instructional video,
as confirmed by frequentist and Bayesian statistics providing substantial evidence for the null model.
So given that the presence of the speakers did not increase cognitive load,
we could expect that learning would not be diminished in the Monologue and Dialogue conditions.
These results are displayed here, and these results basically show that to be the case.
Collapsing across conditions, participants improved significantly from pre-test to post-test.
But when we compared the conditions to each other,
we did not find significant differences in gain scores and once again,
Bayesian statistics provided substantial evidence for the null model.
Despite the lack of conditional effect on learning,
there may still be some benefit to including the visual presence of the instructor
or instructor and tutee in a video if it inspires,
for example, greater self-efficacy or interest in the subject matter,
in our case it was coding, or programming.
But as shown here, we did not find differences between our three conditions on
measures of self-efficacy, on the right here,
or interest in coding, on the left.
This was confirmed with Bayesian analysis,
again yielding substantial evidence for the null model for these variables.
Overall, you'll see that self-efficacy in programming decreased after the intervention,
and this is possibly because participants were better able to estimate their knowledge of programming
programming after being introduced to it.
Interest in the instructor, shown in the middle here,
was also unaffected by condition
and Bayesian analysis provided substantial evidence for this result.
For the null model, rather.
I’ll close with a few points of discussion and future work.
These will be presented in brief, but you can find full details in the paper.
Previous studies have shown that when unscripted instructional videos of monologues and dialogues are compared,
students learn better from dialogue videos.
It’s found, however, that features like misconceptions and deep questions
play a large role in improving learning
and these features tend to appear more in unscripted dialogues than monologues.
So when both types of videos are scripted to include these elements,
the benefit of dialogues has been found to disappear.
Our study followed this pattern
since we consciously decided to have equivalent content across our conditions.
So we also added information on the benefits of dialogue-style presentation on self-efficacy,
cognitive load, and interest.
We did not find evidence that dialogue-based videos improve these measures,
but there are still reasons to believe dialogues could be beneficial to learning.
One line of reasoning focuses on what video observers are doing as they watch monologues or dialogues.
Prior work has found that when participants were asked to complete a workshop in pairs-
er, sorry, a worksheet in pairs while watching unscripted instructional videos,
participants who saw a dialogue learned more than participants who saw a monologue.
Why might that be?
One explanation is that the dialogue observers engaged in more constructive, substantive discussion
than their monologue-observing peers, and so were less passive.
It would be interesting to see if future work can find similar results
and maybe develop this and other explanations.
Turning now to visual presence,
the picture-in-picture view of the instructors- sorry, of the speakers in our Monologue and Dialogue conditions
were-
had the potential to add visual social cues to their respective presentations.
They were predicted, on one hand, to facilitate learning by increasing participant interest in the content.
And on the other hand, a competing prediction was that the visual social cues
would distract learners away from the Python content, thereby increasing cognitive load or impairing learning.
We found neither of these to be the case.
We might find that learning would be impacted differently
if participants were to pay more attention to the instructor,
and this could be affected by factors that we did not focus on in our work,
for example, having the instructor wear bright colours, or use sudden, large gestures,
or have a very animated or gregarious personality,
so all factors of instructor salience.
So the popularity of online learning platforms that rely on instructional videos
makes it particularly important to investigate how to design videos that improve interest and learning.
Here, we compared three styles of presentation and found that
while the visual presence of speakers in the videos
drew participants’ visual attention away from the learning content,
learning and other psychological measures were comparable across the three instructional video designs.
Bayesian statistical methods helped us provide additional evidence for the null model,
extending prior work that relied on Null Hypothesis Significance Testing alone.
And while our study adds to the mounting evidence
that learning from a dialogue-style presentation is not better than a monologue presentation
when the content is equalized,
and that speaker visual presence does't necessarily impact key learning variables,
in our opinion it’s a bit premature to conclude that these factors don't have an effect in all contexts.
For instance, a context in which observers are actively applying the video material
is an avenue for future research.
And other- maybe using other domains that involve more dynamic visual content as learning material,
and manipulating instructor salience could be other areas that could be addressed by future work.
So, we’d like to thank Sara Nafar for her role as the tutee in our instructional videos,
as well as our anonymous research participants who helped make this work possible.
And of course, to you for taking the time to watch this, so thank you very much!