Hello.
I'm Laton Vermette and I'm a Ph.D. student in the ixLab at Simon Fraser University.
In this video I'll be introducing you to our work exploring the possibilities of example-based
customization sharing, and the tool that we built to support that within learning management
systems.
And this work was done with my supervisors, Joanna McGrenere and Parmit Chilana.
So, learning management systems, or LMSs, are a class of software used by educators
worldwide to create and deliver course materials online.
And these often include a wide array of features, for everything from managing assignment submissions
and grading, to things like integrated quizzes, discussion forums, and even online lecture
tools.
But having such a wide range of features, it's no surprise that there's often a steep
learning curve for educators to get the most out of them.
So in particular, educators like Pat here need to learn how to customize their courses
within these LMSs, by making changes to the look-and-feel, the functionality, or the arrangement
of content.
And this is often needed to match the unique circumstances of the grade level being taught,
the subject, their teaching methods, or plenty of other factors.
And so, depending on the LMS and the course itself, this customization might involve digging
through long lists of application settings, restructuring course pages and navigation,
or even installing external plugins to add extra features to the course.
So, figuring out all of these different ways to customize only ends up adding to the overall
LMS learning curve.
So although popular LMSs tend to have a wide range of online help and documentation available
to learn from, educators are often too busy to spend time scouring the Web to find help
resources.
Some of our prior work with K-12 teachers found that they instead rely a great deal
on spontaneous social help, like getting a colleague to demonstrate how they've set something
up, in a practice called over-the-shoulder learning.
And of course, this can itself be challenging, because of time constraints and the need to
coordinate with someone else.
So, we wanted to know, if instructors had a software tool to sort of mimic this process,
letting them peek at and freely experiment with examples of their colleagues' courses,
would that help them more easily customize their own LMS courses?
To better understand this problem, we first interviewed ten post-secondary educators about
how they had customized their past and present courses.
And through this, we learned that instructors are generally quite fond of discovering examples
of how others had customized, not just to help with task-based learning, but also to
get inspired and find new ideas.
And many of them had often relied on exploratory learning, or experimenting on their own with
different LMS features to learn what they could customize.
But the main drawback here was that when they're teaching a live course with students in it,
it's risky to be making experimental changes to their settings that could potentially break
some features that they rely on.
So, from this study, we derived a set of design goals, and those guided us in designing Customizer,
a novel platform for example-based customization sharing in an LMS.
So, at a high level, Customizer is split into three major components, and I'll go into more
detail about each one later in the talk.
But right now, these three components are the Discovery Interface for browsing a colleague's
customizations, the Exploratory Interface for safely experimenting with them, and finally
the Authoring Interface for users to review and share their own customizations.
We implemented Customizer as a browser extension that works as a layer overtop of the open-source
Canvas LMS.
So the user interacts with Customizer as though it's built right into Canvas itself.
And although we chose Canvas for our implementation, the design should also be generalizable to
other LMSs.
So, to show you how Customizer works in practice, let's start by opening our Canvas course,
currently showing the home page.
As part of the Discovery Interface, Customizer adds a sidebar to every page on the Canvas
LMS, listing several customizations we can make to the current page.
So because we're on the home page right now, the sidebar shows us customizations that affect
the home page.
And looking through these items, we can see that each customization is shared with us
by a different instructor, alongside a preview image and a brief written description of what
it does.
If we click on one of these, it'll bring up a dialog showing more details about that customization,
alongside other changes that that instructor has made to their courses.
So each of these customizations has some pieces of useful metadata attached, such as the other
instructor's full explanation of why or how they made this change, a slider indicating
how complex that change is, and an indication of what parts of the LMS are affected by it.
Now, if we find some customizations in here that we think would be a good fit for our
own course, Customizer allows us to import them directly.
But again, it might be risky to do that without testing them out first, so Customizer also
gives us the option to click this "Try It" button next to any customization, to interactively
explore how it works.
Or, if we want, we can select multiple customizations to try them all out at once.
So after we've made our selections, we can click this button at the top that says "Try
4 selected customizations".
And that enters into the next major part of our design, the exploratory mode.
So this creates a separate "test copy" of the user's real course on-the-fly, importing
all of their course content and settings, but also applying the other instructor's shared
customizations on top of it.
So the elements highlighted in yellow here are showing what's been affected by those
shared customizations we selected earlier.
And Customizer also adds a small help badge next to each one, showing a tooltip with details
about the customization when it's hovered over.
So the core concept of the exploratory mode is to provide this temporary environment in
which the user is free to experiment and make changes in the interface without any permanent
consequences, so they can engage in exploratory learning without the risks.
Finally, Customizer is able to automatically check for differences between the user's own
courses and default course settings, and then it lets them review all the customizations
they've made to those courses.
So from here, in the Authoring Interface, they can add their own metadata to each of
their customizations, like a written explanation, and then they can share those customizations
with other instructors using the platform.
To evaluate this design, we ran a usability study with a different group of ten post-secondary
educators, giving them three tasks to perform using Customizer on the Canvas LMS.
So, I'll briefly go through some of our high-level findings from this study.
First, one of the strong points of the Discovery Interface was its page-specific retrieval
of customizations, so finding ones that affect the current page in-context.
And this was something that participants primarily relied on to complete the tasks, even though
they also had the ability to use freeform search queries to find specific customizations.
Most participants also described a fondness for the exploratory mode, with some participants
comparing it to existing LMS mechanisms for viewing another instructor's course as a "guest"
or as a "visitor", with limited read-only access.
So they appreciated that the exploratory mode instead gave them the instructor perspective,
with full permission to make changes and experiment.
At the same time, they also demonstrated several different ways to use the exploratory mode,
such as confirming their intuition about the purpose of a certain customization, or getting
a quick visual comparison of two customized courses from different instructors.
So, just to wrap things up, one of the key takeaways from this work should be that example-based
customization sharing is an effective tool for helping educators learn how to use and
how to customize their learning management systems.
And Customizer lets them do this without ever having to leave the LMS.
Importantly, this method also has the potential to generalize well beyond educational tools,
so into broader classes of customizable software.
Remember also that we implemented an exploratory mode as a key part of our design, enabling
risk-free experimentation and exploratory learning of a feature-rich interface.
And finally, in our evaluation, participants overall were quite positive about the features
in our design, and were spontaneously coming up with new ways they'd like to use them in
their real courses, which we found promising.
So, if you're interested to learn more, I encourage you to check out our full paper,
and feel free to get in touch with any questions you have.
Thanks for watching!