[MUSIC PLAYING]
Good morning.
It's great to be here I had an amazing middle school math
teacher, Mr. Frazier.
And he told us that learning is teaching,
and that the first person that you teach is yourself,
and the second person that you teach is your friend.
And one goal of this Google X series
is to foster order of magnitude, even multiple order
of magnitude changes.
And so I wanted to enlist your help
today in thinking with me about how
we can scale Mr. Frazier's insight.
And in particular, I'd like to focus
on doing this to learn creative skills.
An increasing number of jobs are knowledge sector
creatively intensive jobs.
And this is also a lot of what we
do outside of work, and in our own lives as well.
And there's a huge amount of enthusiasm for this right now.
It was amazing this morning to see the line snaked
around the block to be able to get in.
I'm not used to that at conferences that I go to.
And there's a whole lot of people who don't have access
to the creative and technical content that's available today.
And we want lots of people to have these skills.
Lots of people want these skills.
How could we address this need at scale?
Before becoming a cognitive and computer scientist,
I was originally trained as a designer.
And here you see a couple of design studios.
The traditional design fields rely strongly
on this studio model of education.
It was founded in 1819 when the Ecole des Beaux-Arts in Paris
got going.
And it's flourished and really grown over the last two
centuries, I think because of the benefits
of peer learning, the ability to see where your friends are
working on, to be energized by critique.
But this doesn't scale.
I mean, this is the most high touch teaching approach
you can imagine.
And so how would we do something at scale online?
I think that in a lot of ways we can
turn this problem into a solution.
My colleague, Jim Holland, famously
observed that the most exciting opportunities for online media
are when you get to go beyond being there,
as opposed to just making a poor facsimile of the real world.
Do something that's different that leverages
the power of computation.
So we've been doing in my research group,
is we've been exploring ways that we
can use the scale of massive online classes
to be able to foster new kinds of peer interactions.
And I'm going to share briefly a couple examples of this
with you today.
A couple of years ago in collaboration
with Coursera we started to put my design class online.
And we were thinking about, well how would we
have the project work be something
that was available for all the students?
And my colleague, Ed Hutchins, gave me
the courage to use peer assessment
and see if we could scale that up to a planet scale class.
He'd been using it for a couple hundred students
really remarkably.
And so Ed gave us the courage to build this out.
And it worked remarkably well.
The way that it works is that we have students first practice
assessing their peers.
This is a new technique and a new skill for them.
By the way, it's also a really useful skill
to have in addition to having the outcome of being
able to grade assignments.
So after we train you up on assessing your peers,
you'll go out and get five randomly assigned assignments,
one of which we know the score for,
so that we can check a ground truth.
And at the end of it all, we have
you self-assess your own work as well,
which is, it's valuable to learn to become your own critic.
And what we found is that this works really well.
More than 90% the students in a pass-fail class,
like a lot of these online classes,
get the certificate that they deserve.
Peers are a little bit lenient, so there
are a couple of people who sneak in and get a certificate,
even though maybe they didn't do all of the work.
There are, however, corner cases.
And one reason that I'm excited to be here today
is that I think things at scale, the corner cases scale to.
And I want to think a little bit about what we can do there.
And this has been used in more than 100 other online classes.
In general, it works really well.
And what we found is that it doesn't work automatically.
You can find blog posts where people are really
frustrated by having a wing nut grade their hard wrought work.
And what we've seen is that there are some strategies that
are important for being able to do peer assessment well.
And so we've worked a lot on being
able to teach the teachers and being
able to bake these into software.
One of the things that's really fun is, getting a score
is nice and all, but it's valuable to be
able to get some more personalized feedback.
And for many of you out there who are teachers,
you know that students always crave
more personalized feedback.
This is true even at the massive scale.
And what we did is we realized that a lot of the feedback
that we were seeing is variations on a theme.
So we package up these fortune cookies
which codify common bits of feedback,
and then students can customize those to give them to peers.
A worry about these massive online classes,
you may have tens or even hundreds of thousands
of students who are taking these classes,
and yet, you know, maybe we're all alone together.
Maybe it's just 100,000 people around the world
staring at a laptop and having no engagement.
And again, what makes design so exciting
is that you have that opportunity
to be in the studio with others.
The community is really the most important part.
And so what we did is we came up with this system called
Talk About, which leverages Google Hangouts to be
able to offer many, many, many small group discussions
in massive classes.
We first rolled this out in Scott Klaus's social psychology
class, and students described the experience
of being able to talk with peers as, wow, it's like a mini UN.
We had an Australian in Dubai, an Afghan, a Romanian,
an Indian, and myself, a Pakistani.
And what we see is that not only do students feel really
energized after participating in these discussions--
and they're really cool to be able to sit in on-- students
who participate in a randomly controlled trial
in a high diversity discussion condition
actually performed better subsequently in the class
than those that were in a low diversity condition.
And so I think especially for domains like social psychology,
seeing diverse perspectives can be a really powerful learning
motivator.
One reason that people are often surprised when
I talk about being able to scale peer learning
is that there's an intrinsic paradox.
You're asking novices to do something
that definitionally only experts can do.
The reason that you're the teacher and not the student
is that you have the ability to do the grading
and create the content.
And when we flip this and have more peer-focused efforts,
I think that the key to making it
work is creating micro experts.
And this is similar in spirit to the work
that, for example, but the fold it at home protein
folding folks have done, where you can train people up
in a narrow domain and build useful skills,
as long as you design that experience very carefully.
And so recently we've been exploring moving from a score
based approach, which actually requires
a fair amount of expertise, to a more attribute or label
based approach.
So what are the attributes of this assignment
that you see here?
And that works a lot better.
And it discourages laziness as well,
which is important in reducing those outlier wing nuts.
Another thing that we can do is, although machine learning
is not at the point yet at least where we can automatically
grade creative assignments, it can help a lot.
And so we've explored-- because we
have this large corpus of training data,
we work with Richard Socher and colleagues
to be able to have an NLP based system be one of the peer
graders, or something that helps do triage
on student submissions.
And we found that by adding the structure,
combining students do the label based grading,
and then verify it, we can add a machine
in to be able to reduce the overall workload
and at the same time give people higher quality feedback.
So the moonshot that I'm sharing with you today
is, I think that we can use strategies like this-- so far
we've just scratched the surface--
to be able to in situ build practical theory
with large scale real world experiments.
And part of what enables this to be both possible and powerful
is that with online education, one thing that I think
hasn't been noticed enough, is that your baking
a lot of pedagogy into software.
And this makes it much easier to transform
learning to other classes.
If we come up with something really good in one class,
by virtue of being in software it automatically
moves to the other one.
And when I talk about theory here,
I'm really inspired by the work a Kurt Lewin, who
was a social psychologist in the last century, who
said that the best way to understand something
is to try and change it.
And so to generalize this strategy,
to X-ify this strategy, what we want to be able to do
is, now that we have, with the power of lots
of connected devices, the ability to run
actual real controlled experiments.
The kind of thing that we used only be able to do in the lab,
we can do that in the real world now.
We can do those experiments toward goals
that people really care about.
And we can use that to bubble up theory.
And this is the step that I think the industry doesn't
always need to do immediately, so it often
gets left on the cutting room floor.
We can bubble that up in a way that's
reusable, that can help other people as well.
So what we can offer here in this is,
we have some open source platforms at this URL
here that have the analytics, and course materials,
and instructor resources as well.
And your mission, if you choose to accept it,
is to think with me today about how we might scale more
personalized, mastery oriented learning
experiences to this new global community.
I want to thank the ton of people
who did all of this work.
You can read more about it on my website, or Twitter feed.
And I hope that some of you will join the many students who
are going to take the next round of the HCI class
starting on June 30.
Thank you very much.
[MUSIC PLAYING]