Hi everyone, I’m Laton Vermette, a PhD student in Simon Fraser University’s ixLab.
And I’d like to introduce you to our work examining how instructors integrated an example-based
customization sharing platform into their course management workflows.
So, instructors can face a steep learning curve when they try to customize feature-rich
applications like learning management systems (or LMSs), which they often need to adapt
to the unique demands of their courses.
So for example, an instructor like Sam here might need to dig through a bunch of settings pages
to figure out how to change aspects of the LMS’s look-and-feel or its functionality,
to best support the unique circumstances of their grade level, subject, teaching methods,
or plenty of other factors.
And that might differ greatly from how other instructors set up their courses, like how
Alice may need to restructure her course pages, or how Bob might need to install external plugins
that add extra features he needs in the LMS.
And learning all these different ways to customize can place a lot of extra burden on instructors.
Now, instructors are often too busy to spend a bunch of time reading through documentation
or tutorials to learn all about the features of their LMS.
So they might instead turn to online Q&A sites to get social help, but we’ve seen that more
complex questions, especially about customization, tend to be less likely to receive answers there.
On the other hand, many instructors prefer to learn by borrowing examples of how their
own colleagues have set up their software.
But finding relevant customizations can be challenging, and sharing examples manually
still takes time for both parties.
There can also be risks involved in making changes to student-facing software, when there
might be unintended side effects.
So a lot of instructors are understandably a bit hesitant to experiment with customizing
their courses.
To provide a more streamlined alternative for instructors, we previously designed and
tested a platform called Customizer, which allowed instructors to easily share example
customizations with one another inside the Canvas LMS, and to try out those examples
within a safe testing environment.
And our prior paper at DIS 2020 explored findings from our lab-based usability tests, where
instructors were largely fond of the design… and promisingly, most of them wanted to use
it with their real courses.
But to get to that stage of wider use in real-world contexts, we faced some broader questions
around how this approach would fare at supporting a range of instructors in practice, with all
their own diverse needs and circumstances.
So in particular, the main questions that we sought to answer in this work were: "How
do instructors of varied backgrounds incorporate example-based customization sharing into their
day-to-day course management workflows?
And what variations exist in how they perceive and apply these example-based customization
sharing features?"
To investigate these questions, we needed to extend Customizer’s original design to
facilitate longer-term usage and real interactions between users.
So in our new design, Customizer’s sidebar that appears on every Canvas LMS page, provides
several recommendations of customizations shared by colleagues, weighted by how relevant
they are to the current page, and their popularity with other instructors.
And users can click on any of these suggestions to open up more details, like the author’s
full explanations and information about what parts of the LMS each customization affects.
We also included Q&A threads here where instructors can discuss or get help.
So from this dialog, users can import customizations directly into their courses, or instead use
the "Try It" feature to test them out first.
So in this Try It mode, Customizer generates a "test copy" of the user's real course with
one or more shared customizations applied, letting them interactively explore how they
work, and importantly, do so without any permanent changes or consequences.
And finally, we extended the scope of Customizer’s authoring tools, which automatically generate
a list of a user's own customizations that they’ve made to their courses, to which
they can then add their own rationale, and share them with other instructors.
So, having extended the design, we proceeded to conduct a two-week field deployment of
Customizer to 10 university instructors, aiming to capture a variety of experience levels,
roles, and teaching subjects like I was mentioning earlier.
And afterwards, we conducted follow-up interviews to get in-depth insights into the nuances
of their perceptions and their usage behaviour.
So what did we learn?
Well, to start out with, we saw a great deal of usage of Customizer's features for discovery
and exploration across our participants.
And, we noted that four of our ten participants were willing to share examples of their favourite
customizations into the system, which is pretty substantial considering that most content
sharing systems have a far lower rate of active participation.
So as one example, P6 here shared a customization that lets instructors log how students are using
the LMS while they take a quiz, explaining that it can be used to deter students from cheating
or inform future improvements to the quiz.
And on the whole, almost all participants felt positive about continuing to use Customizer
with their real courses over the long term, especially for things like helping them discover
new features and for testing out new changes to their courses.
And as for the ways instructors were using Customizer to complement their workflows,
there was a great deal of diversity in how they valued different features and use cases.
I’ll go through two of the most important ones, and I'll leave the others for you to read about
in the paper.
So most instructors noted that they found Customizer useful as a sort of always-available
tool for serendipitous learning throughout their usual day-to-day Canvas usage.
Or as P2 put it: “You can easily go ‘Is there anything that somebody has suggested
about this page?’
[...] You might not need anything, but maybe if you’re just curious, you can just kind
of scroll through like, ‘Oh, this is something nice that I've never thought of.’”.
And along the same lines, some instructors felt that Customizer gave them more opportunities
to take control of their learning and experiment more, like P3, who said: “A colleague just
briefly told me ‘Oh this is how I did automatic grade weighting’.
So I knew it existed but I didn’t know the details.
But Customizer gave me the opportunity to try things out and look it up at my own pace.”
Beyond that, many participants also appreciated how they could avoid some the social burden of
“bugging their colleagues” for help.
So for example, P7 thought “a major advantage is you can learn entirely by yourself.
You do not have to rely on anyone […] especially for someone who is introverted like me.”
On the other hand, more experienced instructors also felt Customizer could help streamline some of
their knowledge-sharing routines and avoid them having to manually write up instruction
emails or screenshots every time they give help.
P6, who frequently helped other instructors with Canvas, said she could “totally see
Customizer being like the starting point for instructors.
[...] I could just tell them, `Hey, you know, go to the Customizer system, take a look and
see what other people have done.”
So to wrap up, one of the main takeaways here is that this example-based customization sharing
approach seemed to be successfully supporting instructors’ individual workflows in a variety
of different ways.
And while field study methods always involve some tradeoffs, our deployment approach was
useful for garnering rich empirical insights into how instructors made use of example-based
customization sharing tools in practice.
And finally, we put forward some promising future extensions to these design concepts,
such as tailoring suggested examples to suit an instructors’ level of experience with
the LMS.
And if you’re interested to learn more, I would encourage you to check out our full paper,
and feel free to get in touch with any questions.
Thanks for watching!