[MUSIC]
We're going to stay in the line of motivational, segmentation.
And I want to introduce to you a doctoral candidate,
in the Department of Communication.
Rene Kizilcec has been leveraging the vast amounts of
data that have been accumulating.
By tracking the behavior, observing the behavior of students.
Who are taking the massively open online courses.
That essentially originated in four or
five, now maybe six different platforms here at Stanford.
There have been many questions about the attention, the engagement, and
the learning in those courses.
Re, Rene and his colleagues in the learning analytics group
are studying that, and he's going to give us some early findings.
René.
>> Beautiful, thank you Martha.
>> Good afternoon everyone.
Thank you very much for having me, today.
I'm very excited to share some new work from the Lytics Lab,
the Learning Analytics Lab at Stanford.
It's it's an interdisciplinary lab from students all over campus.
Who came together because they're interested in what online
learners are doing in massive open online courses.
So let me tell you about what's am I going to do in massive online courses.
I am assuming that everyone of you has heard of massive open online courses.
And I have had the pleasure to for the last two and
a half years study behavior and behavior change in these courses and.
Thanks to the media, really, whenever we hear MOOCs we,
we basically think drop-out rates.
High drop-out rates, right?
That, that, that's what we know about it.
Around 5 to 10% of students only finish these courses,
those are the statistics that, that we have in mind.
I like to think of MOOCs in a somewhat different way.
I propose that one way to think about these courses is as them
being the intersection between two worlds.
One is the world of the grammar of schooling.
Which means that there is a instructor who sets goals for a class.
And students go through this course.
They go to the lecture, they take the assessment.
And they try and achieve some level of mastery.
The other world is the world of online media, where there's no
one who tells you what to do and expects that you follow those goals.
But every individual has their own goals and
their own motivations in achieving what they want to achieve.
MOOCs in my opinion lie in the middle of those two worlds.
And that is why they've caused some confusion in how we
should understand what people in MOOCs do and what they don't do.
And whether it's right or wrong.
And that's probably why we've been talking a lot about low
completion rates because that is really a notion.
Coming from, from the Grammar of Schooling side.
So what we've done with some colleagues of mine with Chris Peach and Emily Schneider.
This was with the early MOOCs that came out.
We looked at how do learners engage in MOOCs.
And is there something in between the, the person who completes, and
the person who drops out.
Because that was this, this, this binary notion for, for a long time.
And what we did is we looked at behaviors across the course, and
tried to find prototypical paths of going through this experience.
Okay, and so we, we looked at four states.
One is auditing, where the person is just watching lecture videos.
One is being out of the course.
One is being on track.
Meaning that you are taking the assignments, you're watching the videos.
You're doing well.
And one is being behind.
Meaning that you, you're doing it but you're doing it a little too late.
And we look at how these classes move,
how students move in those classes through those states.
And what we identified are four prototypical paths.
There is this sampling path which is what most students who enroll in and
enter the site end up doing which is just looking at one of the videos.
Maybe trying one of the assignments and then leaving.
There is the auditing path.
When people come to the course and they watch the videos and
they just watch the videos and they're very satisfied just watching the videos.
And maybe never intended doing more than that.
They're the completists that we know a lot about.
The people who do everything as they should including assignments.
And then those are really our dropouts in some sense.
Those are the people who started off as completers, but
then ended up not quite completing.
They left the course early.
And so if you think about it,
if we assume that these people never really wanted to be up there, right?
This, this proportion, it's not very large.
So this work really raised some questions, and one of the things it suggested is,
well, what is driving choices to be in one of these buckets?
Is it, is it motivation?
It's how people motivated.
And when we asked people in this group and
in this group how satisfied are you with what you achieved in your, in the course,.
They were actually very similar.
So, it seems like people who were auditing were motivated to just audit.
And that, and they were satisfied doing that.
And so, that drove us to look at motivations.
What motivates learners?
And how can we maybe, distinguish between learners by knowing what
their motivations are and then serving them with just the right content?
So that led you to these following three questions.
First of all what motivates learners?
Secondly based on those motivations that probably high dimensional sets.
Can we come up with some types to simplify it?
Maybe there are just like now four types that we can nicely categorize people in.
Or, and, and the other, or
is it, or is it harder to actually group them, cluster them?
And then finally, what motivations are predictive of what behavior.
And how can we, then,
intervene and give people with certain motivations just the right content?
Overall what,
what we were trying to achieve here is a data-driven user-centered design approach.
Where we look at all the data that's coming in.
Just like we heard from, from the panelists who've got a lot of data at
their hands to, to decide what is going on.
Centered on the user, understanding what the user really wants and
then designing for that.
The first step was, of course, to measure motivation, which, which is not so easy.
What, what we did is we asked learners.
We asked, 71,000 learners about motivations, and
we, we, we sampled from those responses.
Those were open-ended texts responses.
So they, they wrote what, what they thought their motivations were,
and we coded these through Mechanical Turk.
Iterations after iterations, to really come up with the right set.
When I say right set I mean, a set that gives people the responses that they
would expect, doesn't leave out responses.
And keeps them mutually exclusive, okay?
And so, this is the one we came up with based on several iterations, and you know,
you can see there are a large number of motivations there.
What we looked at next is whether there are these types,
these types of motivations for learners and we looked at correlations.
This is simply a correlation matrix,
with colors representing how strong the correlations are.
And, and this is, this is the legend down here,
and these are not very strong correlations.
These are too weak to really justify throwing out dimensions.
And collapsing dimensions that have a lower dimensional model of
what's motivations going to tap.
Okay?
And there was, there was good in some sense, because it showed that the, the,
the design of the measure was good.
Another sense it, it complicates the picture a little bit.
But we have 13 dimensions
companies like Facebook have 30,000 dimensions so it was still pretty good.
We then went on to take those motivations in,
in ten courses that we had with several thousand people in each.
And we looked at how do these motivations, saying that I am motivated because of X,
versus I am not motivated because of X, predicts behavior.
And we looked at a bunch of different milestones as
we called them in the course.
This is just one of them here.
Whether or not they get a certificate.
And we see that there are two of these motivations that
are negatively predicting that you get a certificate.
Namely taking the course because it's relevant to your school or
degree program, and taking the course because it's relevant to your research.
And one of them, and I love the fact that it is this one,
rather than maybe because you want to earn a certificate.
Which, interestingly enough,
is not predictive of actually getting a certificate.
People tend to not put their money where their mouth is,
I guess, is the interpretation there.
But, people who take the course with others.
Who find friends, who find colleagues, who, who go with them through this.
They're actually more likely to make it to the end, and get the certificate.
Another behavior that is of interest is how many videos you watch.
So that is sort of lower end behavior than getting your certificate at the end.
And we look and look at how many whether look, watch more than 10% of the videos or
watch 50% of the videos and we see these patterns.
We see people who've got academic ambitions they are less
likely to watch more videos.
We can see that people who want to change their career.
Who take the course for fun and challenge.
They are more likely to watch more videos.
People who want to experience an online course.
Which sounds a lot like the sampling group that we identified previously.
They are less likely as we would expect.
People who want to earn a certificate.
They actually do the right thing.
To do, to get a certificate, you watch more videos, so
that is, that is, they understand the concept.
And people who think the professor is prestigious or
the university is prestigious, they also watch more, I guess, to get more exposure.
So we find these, these different patterns.
And the way we, and
this is just a bunch of them there are many more that we look at.
And the way we want to interpret them is through this
Newton gratifications theory framework.
Where we identify needs and how learners try to gratify them.
And we see when there are breakdowns or how we could support them to do so better.
So, one example is, those learners who said that they want to earn a certificate,
actually, less than 50% in a typical course went on to, sorry.
Less than 50% said that they wanted to earn a certificate.
Those who did were not more likely to get one.
But the fact that it's in the glamour of schooling, so clear that people want to
earn a certificate, but only less than 50% say that they want to.
Really tells us that it shouldn't be as much of a focus.
Many, many learners said that they have social motivations,
either want to meet new people or they're taking a course with other people.
Current books really don't give you much to to, to facilitate that.
The discussion forums that's just one very simple way of doing it.
But there are many rich possibilities for
enabling social, social exchange between students.
Like video chats or group assignments.
Which are catching on now, but there's, there is a lot more room to go.
Academic motivations we saw school and
research being, predicting negative outcomes.
It looks like these people are looking for an archive.
They're not, they're not looking to take this course.
If I was taking a MOOC I wouldn't be looking to take a course.
They're looking for an archive and so maybe what we need for
them is giving them the opportunity to, to search properly through the course.
Have good tagging in the course so they can know what to do.
So I'll leave you with three conclusions.
First of all, MOOCs.
One way to think about them, one way that I found to be very useful.
Is as a combination of the grammar of schooling and
the online learning, online media norms.
Another one is that motivations are a powerful way of,
of looking at behavior, and understanding people's behavior.
And the third is, this idea of this, which is not new, I,
I understand, but that, that a data-driven user-centered design.
Which, which seems to be very powerful.
And we went here both ways from, from the behavioral data to the,
to the interpretations, and back, from the survey data to the behavioral data.
And with that I'd like to thank you.
[MUSIC]