JOSEPH JAY WILLIAMS: So I just want to mention, my background
is in cognitive science.
So I'm really interested in doing experiments in terms of
understanding how people learn.
And to get a sense of the way that fits in with the research
landscape, because there's a ton of research on learning
and many insights.
I guess the best concept is thinking in terms of
qualitative analyses.
So these rich investigations that you see from a
sociological perspective.
Education schools that will take a student and really take
a very close look at what they're learning.
Explore how they're understanding of algebra and
all the difference misconceptions they can have.
Then there's also things that are much more like randomized
control trials.
So policy workwear.
You might take a bunch of students in the school and
give them a treatment where they
undergo a training program.
And then you can see if that actually has an effect
compared to control group.
And that's a very large scale.
There's longitudinal studies, which, again, are very long
time scale.
Collecting measures, like people's grades or students'
outcomes as they progress through traditional education.
And of course there's working computer science and
Educational Data Mining, where you take very large data sets
in terms of observations, then try and induce what's going on
with learners.
As is terms of the way cognitive science fits in with
this, I think it's in between something like a randomized
control trial, longitudinal study and some light
qualitative analysis.
Because most of the experiments we do on a short
time scale.
And it really involves precisely controlling what
someone's learning in a particular situation and
trying out different forms of instruction and then assessing
how learning is occurring after that.
And you might think that from such micro experiments you
can't learn much.
But that's actually the expertise
in cognitive science.
And I think it's ready insightful.
It's obviously using a lot of insights.
But I think it's ready well suited for online education,
where often you want to ask questions that intermediate
between again, quality of assessment of what's going on,
and running a full-out randomized control trial where
you give people two different versions of the course.
There are questions about how you should frame instruction,
what kind of video you should show someone, and the kind of
strategies you should teach them.
So it just lets you know where it sits in there.
And what's also nice about cognitive science as an
approach is it's pretty interdisciplinary.
In a sense, you've got psychology, which is heavily
focused on experiments.
Philosophy, in terms of really trying to do good conception
analysis of what problem you're facing and classifying
different kinds of learning.
Linguistics, anthropology, as I mentioned, neuroscience, and
of course, AI.
And what's also nice, I think, is that it's also a bit easier
for people in cognitive science perhaps to talk with
the kind of researchers at Google who are interested
often in things like modeling and machine learning.
So to give you a bit of a preview of [INAUDIBLE]
cover.
So I'm going to talk about two ways you
could think about learning.
One that I think gives us a lot of rich insights into how
to improve education.
And I'm going to talk about very quickly three finds in
cognitive science that I find have been
particularly powerful.
It's thinking about what can you do before someone starts
learning in terms of framing their learning as
an answer to a problem?
What can you do during learning in terms of
requesting explanations from the learner?
And then what can you do after learning?
A new and interesting finding here is that you can actually
use assessments as instructional tools.
Having people take a test can actually be more powerful for
learning than having them study material again.
And then having looked at some of what we know about how to
promote learning of a concept or some set of knowledge, it's
worth asking, well, what knowledge can we teach someone
that's actually going to have the biggest impact
practically?
And that's why I think thinking about this idea of
what do people learn that's going to help them learn more?
So it's not just on this concept but across a range of
situations.
And then I'll talk about how you can change people's
beliefs and have a massive effect on their motivation.
You can teach people strategies for learning that
then have trickle down effects on all the content they may
come across.
And then finally, I'm going to talk about online search,
which is something I really started to think about
seriously recently since looking [INAUDIBLE]
of a power searching course.
And I actually think this is a really fascinating topic
that's right at the junction of things that are really
important to industry and the business world, and also
really important to education.
And I think that's actually a great place to focus, because
it allows you to get the benefits of private innovation
as well as academic work.
And also it means that you can use insights across areas.
When you're teaching a business person about search,
you're also learning something about how you could help a
child to be more inventive in the way they discover
information on the internet.
How is my speed in terms of talking?
Because with an accent, it probably sounds
like twice as fast.
And I speak twice as fast, so you're hitting a lot of powers
being raised there.
OK.
And one thing I'll cover at the end is I put on the
website just a list of resources that I found really
useful in terms of knowing what the literature is that
has shown really impressive effects on learning.
When you go in Google Scholar, I mean literally there are
thousands of papers there.
And I think that can be daunting.
And it's just easy to start with your
problem and work on that.
But there's always tons of really
interesting relevant work.
And so I tried to put aside some of the resources that I
found most useful.
So I have hyperlinks to some of the papers there and brief
explanations of what they're about.
And also information about other things that I'll talk
about at the end.
So in terms of learning, I think one kind of challenge,
even after studying learning for all my adult life, I still
think that there's this kind of intuition or this intuitive
theory that I have about learning that holds me back,
or misleads me when I'm making instructional decisions.
I think it's something that's sort of common
across many of us.
In the Cambridge Handbook of Learning and Sciences they
refer to this idea that when you learn something, you're
just dropping it into a bucket.
Learning is transferred from the teacher to the student.
So let's think about some content.
For example, learning from a video, like from a power
searching online video, a Khan Academy video.
Or reading text, which is how we absorb most of our
information.
Or you could think about learning from an exercise.
So a student solving a math problem, or someone attempting
to process a financial statement or do some budgeting
or solving a problem of managing
relationships with coworkers.
Anytime you can see those icons I need to represent, you
can stick your personal content in there.
So if you have a favorite example of learning or one
that's particularly relevant to your everyday experience or
your job, just feel free to insert that every time that
flashes out.
But I'll try to illustrate it with examples.
Because again there's good evidence that that's an
important thing to do for learning.
So I think there's this intuition that what learning
is about is just adding information.
So it's almost as if we have this model where
the mind is a bucket.
And so what does it mean to learn?
Well it means to take information and drop it in
that bucket.
Whether it's videos, text, solving stuff, just put it in.
And later on you can come back and get it.
But actually I think a much better way of thinking about
learning is that it's like taking a piece of information,
integrating it into the internet, and assuming it had
a web page.
So that galaxy is the internet.
And you can guess what those big giant planets are.
So how might you think about this analogy shedding more
light on learning?
Well under this kind of view, when you're adding content to
the internet, first of all, you have to
think about three stages.
Well what do you do before the content arrives?
Or what do you do in linking it to what's already known?
So it's really important now.
You're not just dropping in a bucket.
You're actually linking it to web pages that already exist
by citing them.
You're putting on a particular domain.
And so I think this is very analogous to the challenge we
face as learners of how do you get information to link up to
what people already know?
And in what kind of knowledge are you linking
new concepts to?
And I think that some things are obvious
when we look at it.
But it's definitely not at the forefront, I think, of a lot
of instructional decisions.
Also if we think now even of processes people engage in
while they're learning.
Everyone on the internet, time is limited, they're not going
to read your whole page.
If you're lucky they will even read the first few lines.
So we have to ask questions like, how do you structure the
information on a web page so that the really core
concepts jump out?
What are the key principles?
Or depending on the person, is the web page structured so
they can get to the information they need?
And so this is analogous to when we're processing
something like a video or a bunch of text, it's actually
just flat out impossible to remember all that information.
So what are the cognitive operations you're engaging in
that pick out some information as being more
important than others?
Some relationships or some principles or some details of
whatever content you're learning.
And then finally, what's the last part of getting a web
page off of the internet?
Again, I think you guys all have a better advantage in
answering these audience type [INAUDIBLE] questions.
But it's actually being able to find it.
And this is probably the most underlooked part of what
learning involves.
Because anytime you learn something, it's only going to
be learning if it influences your mind in a way that, at
some point in the future, you're going to act
differently because of it.
So you have to retrieve that information somewhere.
So for example, the final stage of learning is it's got
to be set up in a way that it's got the right cues or the
right connections that when someone goes looking for that
information, they're actually going to find it.
For example I might be really interested in things about
mathematics from a video.
And you could say that I know it.
But when I am actually faced with a situation where I have
to solve or differentiate something, am I actually going
to remember that fact at that moment?
Or am I going to remember something else I learned about
differentiation, or something else about functions?
And so really a key challenge in learning is the instruction
has to actually ensure that people are putting information
in the right way that they can access it later.
And actually a key part of that is going to be that after
instruction takes place, it's actually really important to
give people practice in testing them, and sort of
accessing that information.
What could be called retrieval of practice.
OK.
So one thing I wanted to hammer home at this point, in
terms of why it's really important to think about how
hard learning is and why these two models kind of matter, is
that there's this phenomenon called transfer, or this
distinction which is that learning might just be, again,
a piece of information.
A transfer is when you take an abstract principle from one
context and use it in another one.
So transfer is extremely rare.
And I think this is probably the biggest, most important
thing I feel like I know now that I didn't
know eight years ago.
Is that I assume that if I am told something and I don't
remember it, well, I just forgot it.
I have to practice a bit more.
But actually it's a much more profound problem than that.
Some people make this statement that almost nothing
that we learn in one context is going to get transferred to
a very different context.
And so actually the way we do most of our learning is by
really learning things in very specific contexts and slowly
generalizing them out.
So for example, let's think of a problem.
Let's say that you're a general
trying to invade a castle.
And you've got 100,000 people.
And you know that you can take it if you send them
all at the same time.
Unfortunately the person in the castle knows that as well.
And they've mined the roads so you can only
send 20,000 at a time.
So that's just not enough.
You can't send 20,000 people to take the castle.
So how do you solve this problem?
You need 100,000 to arrive at the castle.
But you can only send 20,000 along any
single road at a time.
OK.
Yeah.
There are multiple roads.
So the idea is that you divide your force up, and
they go and do it.
Here's a separate problem.
It's going to use a different kind of technique.
You're a physician.
And you need to kill a tumor in someone's stomach.
And you've got really high frequency rays that can
destroy the tumor.
But it's also going to take out a large part of the
person's flesh.
And you've got low frequency rays.
But they're not actually going to kill the
flesh or the tumor.
How do you solve that problem?
Yeah.
So here I gave you guys an advantage.
But what's shocking is that if you have people, you can bring
someone right in front of you.
They'll do the problem, turn the page to the next problem.
20% of people will actually transfer that spontaneously.
You could even help them and get them to elaborate the
principle, do all kinds of things when they're learning
their first problem.
They don't hit the transfer spontaneously.
If you tell them, I think the one before is relevant, then
they might stop and really think and
they're like, oh yes.
I see how to solve it.
But that kind of transfer, you obviously can't have someone
running around telling you what's relevant to what.
I think it really, in my mind, highlights something that
seems so obvious, like the common principle there.
It does not happen for people at all.
It just doesn't come across.
It's almost like they learn something about generals and
invasion, and they learn something
about medical practice.
They didn't learn that abstract principle, and it's
really tough to get people to.
In fact, I can bet I could give you guys similar versions
of this problem and bet in your life in a week, in a few
weeks, and you'd be surprised how high a rate
you'd fail at that.
They've even done really compelling studies where they
take business students.
And they have them learn a negotiation strategy.
For example, we all think negotiation is about
compromise.
But actually one much better strategy is to find out what
the other person wants that you don't really care about
and what you want that they don't really care about and
trade off on that.
So we could have an orange, and if we're fighting you
could take half, I could take half.
But maybe I want the peel because I have to
flavor up some food.
And you actually want the inside because you're hungry.
That's a trade off negotiation strategy.
So what's interesting is when you take MBA students and
teach them this strategy, they do everything that you want
your students to do.
They understand the strategy, they can explain it to you,
they can give you other examples of the strategy.
They really get it.
Except when they actually have to use it.
If you put them in a face-to-face negotiation, they
fail to use the strategy.
So that's really compelling.
Because even in a situation where someone's done
everything that we like students to do, and students
almost never get there all of the time, it's still the case
that they're going to fail the transfer.
And so I think that really highlights the importance of
thinking carefully about what sort of processes you have
people engage in while they're learning, and also having
assessments that can really help you understand if people
have transferred or not.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Oh sorry, yeah.
I'll clarify.
So they could tell the strategy to you and they could
write out and explain the principle.
But if two minutes later they were then in a situation where
they had to negotiate with someone face-to-face as part
of a team, 20 to 30% would do it.
20 or 30, but it was very low.
AUDIENCE: Was anyone asking the question, what do you want
that I don't want?
JOSEPH JAY WILLIAMS: Exactly.
And it's shocking, because they really did
understand the strategy.
But face-to-face, it's almost like you have a different set
of concepts when you're dealing with people and a
different set of concepts when you're learning something in
your class.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Yeah.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: I guess it's probably both true, but I
would say I lean towards the second one.
The way some people write about it, your mind is not
designed for transfer.
It's not designed.
You've got concrete concepts that correspond to different
things, and if you want transfer you have to do
something special with them.
Actually that's the original paper that showed the general
and the medical study.
This is one that showed it with business students.
And actually this guy wrote a book that sort of
put transfer on trial.
Where he pretty much argues, he thinks transfer almost
never occurs.
And so what we need to do is teach people specific skills
in specific contexts.
Obviously that's an extreme position but that's just to
let you know the tune of how people think about this.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Yeah.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Oh sure.
The question was, are these theories of transfer also true
if while they're learning about the concept they demote
mock negotiations?
Or the other point was maybe if they do three different
negotiations across different contexts.
And so those things definitely help and they make a big
difference.
The really key principle I'd seen is that there's a big
focus in business and medical education
on case-base reasoning.
And actually there's good evidence that how people
structure abstract concepts around a specific case is
really useful.
But the experiment by Genter, Loewenstein, and Thompson,
what this showed is that even if it helps to have a specific
case, you still don't necessarily get
the abstract principle.
So what is really nice about the experiment is, they would
give people two cases, one after the other, as was
suggested, versus giving the two cases side by side, and
actually tell people what are the similarities and
differences between these two cases.
In this story, it was about business, but let's say a
story about general, the story about the medical doctor,
which agents are playing what role?
And what's the relationship that exists here?
And so when you actually have that pretty extensive process,
which is really good at helping people really get
abstractions in a way that's telling it to them and words
that just doesn't, when they actually have to process this
and align items, that actually makes a big difference in
abstraction.
But it's not necessary enough for them to just see cases.
That helps.
But you've really got to add something extra onto it.
In comparison, I would say one of the super processes that
we've seen for getting people to really grasp an abstract
concept that they wouldn't get in words or they won't get
just from examples.
OK.
Well, in terms of thinking about if someone's about to
learn some content, what are things that
we're going to do before?
Are they going to watch a video about some training they
have at work?
Or are they going to read a text about mathematics?
So what we typically would do is give them that information.
And we might ask one or two questions or tell them what
they're going to study.
You're going to learn about math.
But pretty much you give them the information very much in
line with putting it into their head like it's a bucket.
But actually there's this really interesting set of work
on what's called Problem Based Learning.
And so the idea here is that with a very minimal change you
can influence how people are going to process
that video or lesson.
If I ask them a question before that's going to make
them interpret everything they learn as a
solution to a problem.
Instead of learning, for example, reading a bunch of
text about solving algebra equations, if that was
prefaced with if you have two variables in an equation and
you're trying to figure out how to isolate one, how
would you do it?
The student will probably not actually know how to do that.
They'll fail.
They'll try all kinds of things that are wrong.
But the mere fact that you got them thinking about that
problem, that means that as they go through the lesson,
they are encoding all that information as a solution to
that problem.
For example, you can think of cases where people have told
you something and you take it for granted or encode it, but
if at first you'd first been trying to solve a problem and
then got that piece of information in response, that
would've been much more effective for learning.
And so what's really nice about this manipulation is
that it's very minimal in that all you've got to do it just
take whatever content you have and proceed
it with some questions.
For example, if you're about to learn, let's say, about
power search, you could ask someone as well, how do you
find this kind of content?
Or if you're learning mathematics, is it possible to
construct this kind of relation and then just having
people think about that will then lead them to process
everything they see after differently?
It's also more motivating for students in that they're not
just absorbing facts, but they feel like they're actually
learning with some kind of purpose in mind.
In terms of thinking about what you can do during
learning, and here I'm going to focus on the work I've done
on explanation just because it's what I know best in terms
of presenting.
So the idea here is that we all have this intuition that
it's really good if we actively process information.
Or if we have ever tried to explain a concept to someone
else, something about actually explaining it to them might
actually resort to something new.
But despite that, it's not so often that instructional
videos or texts have questions embedded in them or activities
that force you to generate explanations.
And so there's a good bit of evidence that this does
actually help learning.
If you can prompt someone as they're learning to generate
explanations, they understand it more deeply and they're
more likely to transfer the principles on the situation.
The question that was interesting is understanding--
well, not to be too mad about it--
but why is it that explaining why helps you learn?
And this is really relevant in terms of thinking, when are
you going to ask learners for explanations?
You can't ask someone all the time.
Or is this some kind of content you should ask for?
Or what kind of knowledge is going to be targeted when you
ask for an explanation?
So it's got really important practical
implications in that sense.
So now we contrast just throwing someone into learning
and letting them use whatever strategies come to them, with
actually prompting them while they're learning to explain
why they're doing something, to explain why a fact is true
instead of just accepting it.
And so we looked at this in three contexts.
The first was in terms of learning artificial
categories.
So even though there's lots of work on explanation and rich
educational settings, this is the first study that's
actually in a lab context with artificial materials where we
can really control things very precisely.
Even if we have a sense an experience is helping, it's
very hard to pin down exactly what kind of knowledge people
are gaining.
Is it just that explaining is making
people pay more attention?
So the mere fact that you have to answer a question means
that you've got to pay attention to what you're
learning instead of dozing off or looking up information.
You've got to spend more time processing it.
It could also be that because you have to provide an
explanation you're more motivated.
There's a standard social context where you have to
produce information that could possibly be relevant to
someone else.
And so if those are how experience works, then that
suggests one way in which we'd use it as an
instructional tool.
But the idea that we wanted to test is actually that
experience effects is a lot more selective.
That there's something about explanation that doesn't just
boost your general processing or attention, but actually is
constraining you or forcing you to search for underlying
principles or patterns or generalizations.
And so this is a question we could test more readily when
we look at a constrained context like learning
artificial categories.
We also examined explaining people's behavior.
So for example, if you notice a friend donates to charity,
that wasn't me, right?
OK, that's fine.
If you notice a friend donates to charity, you could just
encode this fact about them.
You might make other predictions from it like,
they're going to have less money in their pocket at the
end of the day and probably won't treat you for drinks.
Or you could try and explain it.
Well why did that person give to charity?
And again, that's a different way of learning
about another person.
And so are facts just to boost your general attention to what
they're doing?
Or is it actually going to do something deeper like make you
search for an underlying principle?
And the final context we looked at this in was actually
trying to learn something more educational.
So the idea here was that we had people learn a bit about
the concept of variability.
And this is an example.
I'm actually going to talk more in depth so that I don't
spend too much time on this part, but I'll sum up the main
conclusions.
Across all of these domains the basic form was for
different materials and different kinds of knowledge,
but the basic form was asking a why question.
Explaining category membership, explaining why
someone did something, or explaining
the solution or problem.
And that would be contrasted with letting people choose
their own strategies, so being able to study freely.
And we also contrasted it with asking people to say out loud
what they were thinking.
Because it could be that the main effects of explanation
are actually having you clarify your thoughts.
Once you state it explicitly you can see what you were
thinking, what was missing, what the gaps
in knowledge were.
And so we're trying to pick apart that effect from whether
there's something else we're explaining, driving you to
search for underlying principles.
And then following the study, we gave people learning
measures the tapped exactly what they've learned.
So to talk about going into that third one in more depth,
I'll just run through the procedure so that
you can get a sense.
So the idea here is that people have to learn a
university's ranking system from examples.
So you're told that a university has a way of
ranking students from different classes.
And you're going to see examples of students who have
been ranked by last year's officer.
And you have to learn, well, what's the
system for doing it?
What information are they using?
So for example, you've got information about someone like
their personal score, their class average, the class
minimum score, the class deviation.
So for example, this might be John.
He got an 86% in history.
You might see what the top score is, what the class mean
standard deviation is.
Then there's another student.
Then in physics, he got about 80%.
Again, you see the top score, the class mean, the class
standard deviation.
And so, you'd actually be told which one was ranked higher.
So for example you'd be told the person on the left was.
And so I'm trying to think about how we
can learn about this.
There are a bunch of different ways you could approach it.
You could pay attention to lots of examples, and then
that would inform future situations.
You automatically get an intuition for
how people are ranked.
You could try and look for different patterns.
So for example, maybe people are ranked higher just if they
had a higher score.
The university is pretty egalitarian across courses.
You could think it's actually how far they are above the
class average.
So for example, this person's 6% above the class average,
whereas this person's 4%.
So even though they got a much lower score, they might still
actually be ranked higher.
Actually sorry, this is consistent in this case, but
the reason this person's ranked higher is not because
they got a higher score.
It's actually because they're farther from the average.
You could also think it's actually just a matter of
being close to the maximum score.
If you're close to the top person then you deserve to be
ranked higher no matter what.
And then finally, if you think about what statistics would
suggest, there's this idea that you should actually look
the distance from the average, but weighted by whatever the
standard deviation is.
And so the idea here is that what you really want to know
is, how far above most of the other students in the class
was each of these people?
And that'll give you a measure of whether you should rank
someone highly or not.
How many standard deviations above were
they, what is the Z-score?
And so we gave students a bunch of these observations.
They saw five ranked pairs in total, and the actual true
rule was using the number of standard deviations someone
was above the average.
But the other rules were pretty salient.
And in pre-testing, we find that a lot of
students endorse them.
Whoever got the highest score should be ranked higher.
Whoever's closest to the maximum, and so on.
And so some of the observations just didn't fit
with those rules.
So this might fit sometimes with the higher raw score, but
sometimes they don't.
They might match the close to maximum rule, but not always.
And so the idea here was, if explaining really forces
people to seek underlying principles, then explain these
sort of observations and the ones that conflicted with some
rules and others should drive them to actually find this
underlying principle, the rule above deviation that actually
applies to all the observations.
And so this is what we found.
This is showing you people's accuracy in classifying ranked
pairs before and after and how much improved.
And what you see is that people who had to explain had
a much bigger increase in accuracy, of about 40%,
compared to those who were writing their thoughts.
That's interesting because writing your thoughts is a
pretty elaborative activity.
You have to express what you're
thinking about the materials.
You have to process them deeply.
But explaining [INAUDIBLE]
this broader principle.
And that's just an illustration because across
all those contexts where there was learning about categories,
explaining how people find a broad generalization that
counted membership for all the robots.
If it was explaining people's behavior, again, you were more
likely to discover a general pattern like, young people
donated to charities more often than older people.
So this suggests that, in terms of thinking about
educational contexts, when you want to ask someone a why
question, it's not necessarily a good idea to just always be
prompting them, or just trying to boost their engagement.
You actually want to be selective in terms of when
they're given examples they can point them towards an
underlying principle, then you need to prompt
them to explain why.
Or if you think they don't have enough knowledge to
induce the principle yet, then it might actually make more
sense to engage in other activities, like helping them
acquire new concepts or have more facility with
understanding what the different facts are.
And then they might reach a stage where they can actually,
with prompting, be able to induce
the underlying principle.
What's another interesting thing is that in the second
line of experiments, we found that explaining actually
boosted the extent to which people used their knowledge in
finding underlying principles.
So the idea was that in the absence of explaining you
might have a lot of relevant knowledge that might suggest a
deviation is a good rule or something else is.
But you don't actually bring it to bear.
If you explain though, and you don't have that knowledge,
well then you might find some pattern but it's not
necessarily the correct one.
So the idea is that explaining of prior knowledge, interact
or guide what patterns people are searching for, and what
they're likely to detect.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: You mean in general, or with respect--
AUDIENCE: You said [INAUDIBLE]?
JOSEPH JAY WILLIAMS: So we ran it online and we just asked
people afterwards how familiar with it they were.
And it was a pretty big range.
Some people had never heard of it before, but a decent number
had at least come across the concept before.
And those people though, they still did ready
badly on the pre-test.
So in a lot of experimental situations you can look at
interesting effects of learning.
Because the truth is that learning just doesn't really
accrue that well the first time around.
So often there's relearning the material.
And even with extensive practice, it's hard to get
people to get that idea of the Z-score principle.
And we didn't find differences between low and high people
who had low and high exposure.
But again that might be because we need
to broaden the sample.
Because it was online we might just look on people who just
didn't have a lot of knowledge.
They wouldn't sign up for a math experiment maybe.
Also in terms of thinking about explanation's effect, it
really was selective.
And actually what we found is that it didn't help you
remember information better.
Sometimes even impaired it.
But especially if there wasn't a reliable pattern present, or
if you couldn't actually find it because you just didn't
have enough knowledge to, explain actually impaired
learning in some cases.
Because it drove people to keep searching for a principle
that didn't exist.
Or one that was only partially reliable.
And so this is something you might think about in terms of
when people make these mini generalizations, or they make
these kind of bugs that's sort of seem half right.
Children do this a lot in algebra for example.
And it might actually be from trying to seek explanations
and construct some kind of pattern.
But it's just a case that it's characterized as cases they've
seen so far.
But it's not actually going to help
them with future learning.
And so again that's another reason to think carefully
about well, when are you going to prompt someone for an
explanation?
You need to at a point where it's not going to hinder their
learning or be worse.
On the flip side, it's not saying explanation is bad,
it's just saying it's sort of like a powerful selective tool
that if you use in the wrong cases,
will actually be harmful.
And then since most of the workers have dealt with adults
we also looked at the effects of explanation in
five-year-olds learning causal relationships.
And we found very similar results.
Even though they were young and their ability to verbalize
are pretty limited, it was still able to, when prompted
to explain, discover underlying patterns.
And especially to bring prior knowledge to bear, and
actually use it to discover patterns that they wouldn't if
they weren't explaining.
And in terms of thinking how to take these kinds of
findings online, one really good context is thinking about
things like online mathematics, exercises, like
the one that Khan Academy has in their framework.
So that's a mini shrunken version but basically people
will get a problem statement, and they'll try and solve it
and type in an answer.
And along the way if they wanted hints, clicking on a
hint button gradually exposes the actual solution.
And so in the literature this is called a worked example.
And it's sort of infamous that sometimes they can be much
better for students learning than just trying things out
and not succeeding.
But it's really hard to get people to process them deeply,
to actually not just skim over it but understand each
component, each step, and how it relates to something more
general, so to get the principle.
As the last one case explained could be really helpful.
And so there are a number of different ways we're thinking
of approaching this.
But the simplest version is something like, when someone
enters a correct answer, or even if they're following a
solution, you can prompt them to try and think, well,
explain why that solution's correct.
Or explain why that person took the step they did.
Even though it seems, of course, you just saw the
correct answer, you should be thinking
about why it's correct.
This is actually not necessarily that common.
I mean, it's not guaranteed.
A lot of students may be trying to memorize the answer.
They may be trying to explain other aspects of the solution.
There are many things that people may be doing.
And when you do detailed analyses of children's
strategies, you find incredible variation.
And so that idea, just having that prompt to focus them on
understanding how that particular answer, or that
particular solution is just an instance
of a general principle.
What's the general strategy that was used here?
What's the key concept?
Now one thing that could be problematic is even though
this is promising because you can get people to learn just
by being prompted, so you don't necessarily have to give
them feedback, when people don't get feedback they often
tend to feel like they're throwing
explanations into the void.
They're just talking and it's disappearing.
And so one thing I'm going to try is try and couple this
really closely with actually giving them an explanation, a
correct one, or almost correct one.
And tell them that's another student's or teacher's
explanation.
And so the idea is not only change a task from do
something that the teacher or instructor's asking you to do,
type something in, to generate an explanation, and then
you're going to be comparing it to what someone else said.
Actually the idea here is that at this point you could ask
them to do things like, once they've got an explanation
written down and they see someone else's, well, what if
you grade both of them?
Do you think that was a good one on a scale from one to 10?
Was your explanation good on a scale from one to 10?
You could have them elaborate on what they thought might be
better or worse about an explanation.
Or you could even just have them make ratings like a
similarity or dissimilarity.
So it's not very common to ask for these kinds of ratings
that's in a MOOC or online.
And I think it could be because often if you just ask
a question, and at random, 1 to 10 rating, people may not
give informative answers.
But if you use the kind of assessments that psychologists
have really honed down on for understanding things like what
concepts people think are similar, or people's ability
to predict when they make errors.
I think there's some really useful information which we
get out of this.
And what's even better is you're getting information
about what students think, and you're helping them learn at
the same time.
You have to comparing what they degenerate into what
someone else generates.
It could be very powerful.
There's a good reason to think it would be very
powerful for learning.
Another thing that would fit in really well here is that
there's work by people at Khan Academy, like
[? Carolyn Rosie ?]
and David Adamson.
And what they've got is they've found a way to
leverage the minimum amount of AIU you can get into a web
based system to get lots of students learning.
And so the idea here is that when people might type in
something like this, like explain why you
think that's correct.
They're placed in a small chat room with a
couple of other students.
So then this is not embedded in the context of necessarily
a Khan Academy exercise.
But it's embedded in the context of trying to solve a
problem together.
And so someone has to provide an explanation.
And then the chatbot can then say things like well, John,
what do you think about that person's explanation?
How would you say it in your own words?
Do you disagree?
And so these are simple questions you can ask people.
But they've been shown to have a really powerful effect on
students' learning.
The term for this class of strategies, like accountable
talk moves.
And the idea is if I get people to really think about
what they're saying and how it differs from other people's
ideas, you can get a lot of learning.
And once you have that chatbot in there, it can also act as a
sort of instructional tutor, but one that doesn't need a
lot of attention, a lot of time.
Because students will often carry a lot of the
conversation themselves.
And the key thing is just doing things like adding
prompts to keep it going.
Or matching a student's explanation to something else
in it's database.
And then not saying, and just asking a student, is
this what you meant?
And so it's a great way that very simple technology you can
give an answer and the answer's probably not going to
be taught right because NLP isn't there.
But then actually the student's having a learning
experience because they're now trying to correct what you
said and put exactly what they were trying to get across.
And it's a really exciting line of work to look at.
So in terms of thinking about the benefits of explanation, I
think it's a great way to have instructor guidance, because
you can ask people questions that
guide how they're thinking.
While at the same time having learners generate information
or generate knowledge.
So they're actually constructing it.
It's also especially on online courses useful because you can
do learning without any feedback.
So you might have to do things to get people to actually
generate explanations, but it's a great way of actually
maximizing learning without necessarily having, even if
you don't have the resources to give people
individualized feedback.
And again it's really key, because as I mentioned,
transfer is really challenging to people, so the fact that
explanations help him understand these abstract
principles can have a very powerful
effect on their learning.
So in terms of thinking of what you can--
do you have a question?
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: That's a planned study, that's a
planned study, yeah.
So--
AUDIENCE: [INAUDIBLE]?
JOSEPH JAY WILLIAMS: Yeah.
I think we're really excited to look at--
oh sorry, the question was, did we find effects of having
them grade other people's explanations?
I was just saying that that's a planned study.
Because we can update tons of things in the lab when using
software [INAUDIBLE].
But when we have to deal with a platform that has its
constraints, unfortunately it takes a lot of adaptation.
Just on a side point, actually, one way I think that
research psychologists can be really relevant to people
actually doing practical work, like in edX and of Coursera,
in Khan Academy is they have a lot of constraints in terms of
the system they have to build has to scale,
getting things done.
But actually I think something like a sand-boxing addition to
any of those platforms will be very useful.
For example there's like 50, I mean literally 50 softwares
that don't require programming experience.
You can author an experiment really quickly.
It won't be as beautiful or as glossy as maybe, we think,
Khan Academy's perfected.
But it would be a great way to test out very quickly and
cheaply lots of different strategies for learning.
And then once you figure out which ones seem to work really
well for learners, that's what you put your
development time into.
Into actually investing it.
For example with Google's Open Course Builder, maybe there's
functionality you're thinking of implementing.
Well before doing that, you could try and add an addition
to a power searching course where some people will opt
into take in a different software, and then we can
replicate tons of things in terms of the questions they
asked or the examples they see or the format of video
presentation.
And then when you see which ones are really effective
that's some functionality we can then
build into the platform.
So in terms of thinking what people can do after learning,
after the study is done that actually has a
huge effect on learning.
I think a really interesting idea here is to use
assessments as instructional tools.
So normally we think of a test as just being to measure what
someone's done.
So it's like an inconvenience.
It's taking away time from learning.
And actually at Mackenzie, for example, especially when they
do training of soft skills, the instructional designers
complain that when they spend 20% of the time actually
testing people on this, it actually didn't think it was a
waste of time, because that was less instruction people
were getting.
I think that's something that maybe needs to be re-thought
in some ways.
Because there's now really good evidence that actually
testing people can have a really powerful effect on what
they learn.
If you think of the traditional model where you do
lesson one, lesson two, and lesson three.
And then maybe you do them again.
You drop some more information into the bucket.
Versus actually after each of those you actually add an
assessment.
So there's actually tons of reasons
testing can help learning.
For example if it gives you feedback, if it makes you more
motivated to study.
But I'm actually going to talk about the effects of testing
that are only due to taking the test.
So there's no feedback involved, and you're not
allowed to restudy.
So the idea is that something like mnemonic or there's
something cognitive about having to
generate information yourself.
Even though you get no new information
and you don't study.
And on a test you normally generate much less than what
was actually in the actual lesson.
So this refers to the testing effect.
As you can imagine, if you give a learner the option to
study something once, and then following that they have the
option of studying it again, versus they had the option of
studying it and then taking a test.
Most learners would opt for this first one.
And in fact, there's good data that they do.
And in a way it's sort of strange to think that taking a
test would be helpful because you can't recall the 200 facts
from the video.
You can only recall a subset of them.
But what's actually been found is if you look at immediate
testing, so let's say you do the two videos, you do a
version of test, and then you, let's say, do one more test
short after.
You actually don't find so much for difference, or you
find a slight advantage for spending extra time studying.
But if you wait an hour or a few days or weeks, what you
actually find is that studying twice is definitely much worse
than studying and being tested.
In fact the most recent paper in science found that you
could have people study four times in a row, and it wasn't
as good as studying and then testing.
So the key thing that's going on here is thinking about the
retrieval stage of learning.
Where it's not just about having
information in your mind.
It's actually able to get to get at it.
And when you do a test you're forced to form the cues or the
retrieval, pathways or connections that will help you
get back that information.
What's interesting is that in all these studies, even though
they were doing worse in the study/study condition.
They claimed that they were doing better.
And they expected their performance to he higher.
Another really interesting finding actually that I think
has gotten less attention but it's very powerful is what's
called the Mixing Effect.
So let's say you have questions of type A, B, and C.
Sorry, lessons A, B, and C. It's natural to put the
assessments for A, whatever 3x says you have right
afterwards, and so on for B and C.
But the idea behind the Mixing Effect is, don't make new
exercises that do anything different.
Or just scramble the order.
So if after you learn A, you get one question A. After you
learn B you get a question B, a question A, and a question
B. And then when you finally finish it out on C, you get a
mix of all of them.
And so this is especially dramatic because you haven't
increased how much exercises you have to make.
Learners do terribly when they're doing this and they
think they're doing really badly.
But it has profound effects on their ability to actually
generalize.
Because the idea is that when you do an exercise and them
move straight into having a bunch of questions, it's
really easy to know which principle applies.
When you mix them together it's really necessary for them
to figure out what the correct principles are
and where they apply.
And so even though they do much worse in the learning
stage they do far better in generalizing.
And this something you can easily do
in a textbook, right?
Take all the exercises and start mixing them around.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Yeah.
Let me think a second.
So there's no study I can think of that said that.
Even though there might be one out there.
But I should have thought.
Because that would definitely be, as you
said, a great point.
AUDIENCE: The idea is to--
before you [INAUDIBLE]?
JOSEPH JAY WILLIAMS: Yeah.
That's a really good point, actually, yeah.
So I'll check the quiz.
And so this is the quiz [INAUDIBLE] on Mixing Effects.
So I'll take a look at the paper and see if these
[INAUDIBLE].
But that's true, that would work really well.
And it's not something, I think, that would necessarily
be obvious in this perspective.
But if you put together the problem-based learning work,
then it does make a lot of sense.
So basically I think what's really exciting about this
work is, that basically when you have online education you
can't have a real world lab.
We have to be in a lab of psychologists because you need
control and you have to be able to measure
what people are doing.
But now that people have moved online, you've got a real
world environment that's exactly like a lab context.
You can do experiments, randomize assignments,
precisely deliver and control.
And you can have really quantitative measures of
learning that are automatically connected.
And so I hope that in the future what that means is that
you get a lot more cognitive scientists and experiment
psychologists actually working directly in
online education platforms.
Whereas normally it's incredibly difficult to do
education research.
You can't have a teacher around and assign instruction
to different students.
They just can't do it, you know?
Whereas you can have two different videos show up.
What's also really nice, then, is that we're doing research
in a context with really rapid ecological validity.
And you're actually doing research on a product.
Where when you're done with your theoretical research it's
actually been improved.
It's better for learning.
And so what you can have is a lot of
evidence-based decisions.
You can take that product and deliver
it directly to someone.
So there's perfect fidelity to treatment.
Whereas a lot of research conclusions, even if they work
really well when the research is carried out, it's hard to
disseminate it, because the way someone else implements it
isn't guaranteed to be true to that.
There's great scalability because you've got these
implements in online system.
And especially you've got the knowledge really wrapped up in
the precise things that the researchers decide to ask.
And of course iterative improvement where it's
constantly getting better.
So given that all of these methods are improving
learning, I'm coming towards the end so I'm going to touch
on these things really quickly.
But the idea is, if you've got these ways of teaching a
concept that seem to be effective, well, what concepts
do you want to teach that are going to
have the maximum effect?
And any kind of concept that improves people's future
learning is a good candidate.
So compare with your motivation, learning
strategies and online search.
And so in terms of increasing motivation, the idea here is
that one really powerful way to motivate people is actually
by targeting their beliefs about intelligence.
So do you agree, for example, that your intelligence is
something very basic about you that you
can't change very much?
Or do you think no matter how much intelligence you have,
you can always change it quite a bit?
And so Carol Dweck writes that as being a fixed theory of
this malleable theory of intelligence.
And it's pretty shocking what this predicts in terms of
people who start off with equal IQ's will diverge.
Because people with fixed theory
avoid taking hard problems.
People with a malleable theory always ask questions, they're
happy for challenges, they don't think it means something
negative about them.
And so what are the effects of actually teaching people a
malleable theory?
And so this is work that's done by PERTS, a project for
education that researches [INAUDIBLE] at Stanford.
That's Dave Paunesku and Carissa.
Actually Carissa's is right here, she just
wanted to sit it.
And so what they did is trying to instill all these insights
into a single lesson, where people will be taught that the
brain's malleable.
Then they would have some questions and make them think
about how it applies to their life.
And you can contrast the control commission where the
same lesson will teach people about the brain, but not
necessarily emphasize yet that your brain changes every time
you learn a new fact and that intelligence is constantly
being improved, do exercises, and then try and take some
measurements of what people's mindsets are.
And so this is great in terms of actually being able to see
what effect this has on students.
And instead of this whole system of actually running in
high schools and middle schools and so on.
So it's got all those great features of a real world lab
that I mentioned.
But in addition to asking about mindset, they also just
collect the student's grades.
And normally I wouldn't collect measures on
[INAUDIBLE] grades, because it's so hard to impact
something that has so many variables fitting into it.
And there's a lot that goes on.
But actually what they found in this randomized controlled
trial is that just two lessons of about 45 minutes actually
boosted students' GPA.
Now you might think grades are not the best measure of
things, but if someone knows how to change grades they
really understand something really deep about the
psychological structure of what students are doing.
And you can imagine these kinds of effects applying to
all kinds of contexts.
For example in a workplace in terms of changing people's
beliefs about hard work or about what it means to get
negative feedback.
And so in terms of how I was thinking it would be good to
bring cognitive and social psychology together in this
way is just applying some of these principles like
preparatory questions during the actual learning of the
materials, have explanations, and after, questions that
repeatedly have applied the concept.
And so with Dave and Carissa, I have developed some initial
measures of how people might apply mindsets.
And we'd like to create video versions for MOOCs and Khan
Academy and that you can give to someone.
But we haven't gotten to that yet.
But what I've done is taken those cognitive principles and
applied them to the text they had.
And try and take this simple idea of a mindset, but teach
people it in a way that they're more likely to
remember it and apply it to situations in everyday life.
And so that's actually running now, but in so many lab
studies at least it is having an effective where even if my
inset is better than the control for getting people to
believe in a growth mindset and make judgments about other
people that are consistent with the malleable theory, the
effect of addition of [? Khan ?]
[INAUDIBLE]
is increasing that further.
Well, that's backward.
But we have our fingers crossed that it's going to
turn out successfully, and especially
if it impacts grades.
Another thing that is really interesting is there's so many
ways this mindset idea can be implemented into courses.
For example changing feedback in exercises.
It's natural to tell someone who does well
that they did great.
And it's even better to tell them they're smart.
But the problem is when you tell someone that they're
smart, you're emphasizing something internal and fixed
about them, a quality.
And so there's actually work showing that that can actually
undermine children's abilities.
By praising them for their intelligence they can actually
start to become very resistant of trying things out or being
right about things, showing that their dumb.
Whereas praising children for their effort, or praising them
for the way they've set up a problem or approached
something or tried to find out more is actually a lot more
effective in the long run.
As you can imagine, Khan Academy exercises in workplace
evaluations try to frame things from the perspective of
improving or people's ability to be malleable can have a
really powerful effect on learning.
And so just quickly to go through, I guess I'll just
skip over this.
So basically I'd like to apply the same concept to learning
strategies.
And so the idea here is to integrate work not just from
cognitive psychology, but also ideas about mindset and habit
and behavior change, which is it's own separate literature.
Get people to stop thinking about learning in terms of
memorizing facts but actually in terms of understanding.
And in particular that what it means to learn a concept is to
be able to understand it, to be able to teach
it to someone else.
So the idea would be within a MOOC where they collect grades
automatically.
It's a great situation to introduce training videos on
learning strategies.
Like whenever you're studying something, imagine you're
going to be teaching it to someone.
Or when you finish something, imagine if you had to explain
what you learned to someone else.
And how technology can prompt people to do this.
And then see if it actually does have an
effect on final grades.
And MOOCs provide a great way to actually do that data, that
kind of longitudinal study that you
just couldn't do before.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: What?
Oh, sorry.
OK.
A MOOC is a Massive Open Online Course.
So a course there on Google Power Search.
You have 20,000 people and you teach a course.
OK.
And online search.
I guess you guys are all pretty much on board with
online search being important.
But actually I think everyone knows it's good
to be able to search.
But often we think if it as just like a tool, like it's
for something like fishing.
Versus like a tool kit for problem solving.
And online search I think is powerful for teaching because
if a student has problems in the math problem, and they
just need a habit or they have the skills to be able to find
resources on the internet.
They can find a site that's better for them than maybe
their current textbook.
They could find a site that offers free tutoring.
If they ever later on in life need to solve a similar
problem, they can go online and find those resources.
And so I think online education that teaches
students better search skills or especially to search for
knowledge as opposed to just being able to find product
reviews online.
It can be really powerful in terms of learning.
The fact that there's Google Scholar, it simply means that
if I hear a medical claim and I don't know if it's true, I
can go and look on Google Scholar and just
see what turns up.
I mean, I don't understand exactly just the journal, but
if I find a journal article relevant to it,
it tells me a lot.
So these kind of strategies could actually be a great way
to improve scientific literacy.
Even of course being able to pursue a project, errors that
are not under your current interest is also a great power
of getting students in the habit of online search.
Well then you're allowing it to a bit more independently
driven if they want to learn about a
topic that wasn't covered.
And you can even imagine using online search to do things
like, what are good learning strategies?
Or, how can I improve my grades?
We don't think to go and Google that online but it's
amazing what you will find on Google these days.
So in terms of just--
OK, I'm over, but the one point I'd make is cognitive
science doesn't just have to use instruction design.
I think it ready interfaces nicely with the machine
learning and work that people want to do.
Right now we've got binary answers on
multiple choice questions.
And it's true it's big data in that it's on the order of tons
of information.
But if you have a narrow bandwidth, you just can't that
get much out of it.
And so I think subtle changes to these things can make a big
difference in what we can get out of machine learning and
data mining.
For example, having people rate the plausibility of each
of four multiple choice questions can raise our binary
score into four numbers that each correspond to a different
kind of concept.
Just including those are standard practice.
For example, you just have them read one to seven
[INAUDIBLE].
And so be like, everyone understands.
Having people predict their accuracy is also a really
powerful thing, because there was a study that [INAUDIBLE]
teaches students, and then saw how it correlated with their
later performance in undergrad and the SAT college
performance.
But it didn't correlate as well as how accurate students
were in predicting which questions they got wrong and
which they got right.
So this study had you do an essay question and say how
likely are you to get that right.
And the fact that you were good at judging what you're
were getting wrong or right was a much better predictor of
undergrad success than actually the SAT itself.
And so if you're looking for an individual difference in
learners, that would be a great one.
How good they are at understanding gaps in their
own knowledge.
We think similarity.
It seems really simple, but that's again something that's
just been shown across tons of studies to reveal something
very deep about people's minds.
And I think also grading explanations.
So again that would be a much richer data set than is
currently available for actually understanding
what's going on.
And that's where you might see data turn into
the best data yet.
So let's say I give you two concepts and I say, how
similar is this mathematical concept to this
mathematical concept?
Well two explanations, and I say, well,
how similar are these?
How different?
And that's actually a really powerful way of getting to
what people think or believe.
OK.
And I'm happy to talk about this some more.
But I've tried to put together the resources that have been
useful with me on a website in terms of citations.
And if you're really interested in improving
education, this is a great book in education policy that
looked at all the systems across the world and tried to
see what insights that they have for US education.
And I think especially online education is different.
Obviously there's a lot that's already known.
So we could benefit from a lot of that.
All right.
Thank you.
[APPLAUSE]
And do people have any questions?
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: So the question was, I was talking
about subjects to explain by drawing on knowledge that they
already had, and how did I measure it?
There are many different ways people take to it.
Mostly past studies like you said administered a pre-test.
The way I did it was actually I manipulated knowledge
experimentally.
So they explained or didn't.
And there's another experimental factor, I gave
them extra training or I didn't.
And so that's a way we can know exactly what
they've been learning.
AUDIENCE: That's one of the benefits of [INAUDIBLE]
is that they have learning without feedback.
But you mentioned that explanations were [INAUDIBLE].
Or those people could have had an idea of what you were
thinking, have a way of learning without feedback.
JOSEPH JAY WILLIAMS: Yeah.
So I think that's a really good point.
I think that's why I tried to see well, what is exactly is
an effective explanation?
Because then, once we have a bit of a better handle on what
explaining is doing, if it's driving people towards
principles, then we have a better sense of, OK, it's not
a good idea to ask this person to explain at
this point in time.
We should wait to ask them to explain when
they have more knowledge.
Or it would be a bad idea to ask them to explain why these
two problems are correct.
Because that might lead them to a local generalization that
only applies to those two math problems.
For example with the statistics example, it's
pretty reasonable to say people with higher scores
would be ranked more highly, but that's just accounting for
the observations you've seen so far.
It won't hold in the general case.
So I think features like that in terms of how you set up the
environment and which cases you ask people to explain
would be important variables to look at.
So I think then more research or using research that's
already there would be useful for that.
Another thing to keep in mind is actually that sometimes it
can actually be helpful for people to produce a wrong
explanation.
And there's a paper that suggests this and has some
evidence from the recording they did but it wasn't tested
experimentally.
Which is that once you've produced a misconception and
qualified it, it's then a lot easier to recognize it.
Because then you're not just implicitly
thinking through things.
You've actually made a statement.
Like, I think that that person's ranked high because
they got a higher grade.
And then when you see data that actually goes against
that, you can then revise your belief.
And for example, that's how a lot of children's learning
might work.
Where you could say, my hypothesis' case is flat.
And I'm not going to do something if I
know a lot of data.
Or you could say I'm going to try a lot of things out, and
then just be really open to revise them quickly.
AUDIENCE: [INAUDIBLE].
JOSEPH JAY WILLIAMS: Yeah.
So on that resource page I have some reference of the
[INAUDIBLE].
And I'll let you know some more.
But Eric Mazur, who's a physicists at Harvard, has
pushed this idea of, it's not a flipped classroom, but where
every class actually is structured around questions in
the sense that he'll start a lecture.
After five minutes he'll put a question on the board.
And people will start discussing it.
They'll turn to each other and they'll say what they think.
Then they'll argue for, they'll argue against.
Then he'll pass around and talk with them and they'll get
back together.
And so I think that's got a lot of really
good lessons in there.
And also, this idea I think I
mentioned, reciprocal teaching.
Trying to teach someone a concept and have them teach it
back to you.
And I think that in terms of online environments, what
would be really powerful would be having people generate free
responses, and then again letting them see other
people's explanations, and use those as a base for
comparison.
And so that's something that I think you could get people to
be a lot more engaged.
Plus get the benefits of generating information, plus
it gets the benefits of things that are
like the right answer.
So that's what I'm really excited about
also trying that out.
A big thing that I think that people do now is because they
think that they have to give someone an explanation, there
are very few free responses.
So like the cognitive tutors, which are these adaptive
programs, will actually give people a multiple choice
instead of explanations.
And I think this simple manipulation of not having
multiple choice, having a free response, and then putting the
multiple choice can make a really big difference in terms
of what people learn.
You had a question?
AUDIENCE: You're running out of time.
JOSEPH JAY WILLIAMS: Oh, OK.
AUDIENCE: And so I think if anybody had questions for
Joseph, feel free to ask them after the talk.
JOSEPH JAY WILLIAMS: Thanks for having me.