So in the physics department, where Louis and I
are both faculty members, we were
starting to get interested in questions
about departmental-level learning
objectives and department-level learning outcomes.
So a little bit of history behind this.
So Chris Stubbs, who recently served a term as interim chair
and previously had been chair of the physics department,
has long been interested in improving
undergraduate education.
When he was the department chair,
he brought Carl Wieman to campus to talk to faculty
about creating learning objectives for their courses
and helping faculty to try to define, what do you really
want them to learn when they take electricity and magnetism
or quantum mechanics or whatever that individual course is?
And Chris, at the time, said, OK, I'm department chair.
I can just wave my magic wand and make everyone make learning
objectives for their courses.
And of course, we know that didn't work.
But Chris realized that this idea of actually defining
learning objectives is a very powerful notion
that you can really articulate what
it means to learn the content and skills and concepts
in a course.
And a few years later, when Chris was serving a term
as interim chair of the department,
he wondered, is there anything I can do, one semester as chair,
that could have a lasting impact on undergraduate education
in the department?
And he turned to Louis Deslauriers,
who is director of science teaching and learning in FAS
and also a senior preceptor in our department.
And Louis is one of the great experts in science education
here.
And Louis immediately said, well,
there is something that no one else is doing,
which is actually looking at the department level
at learning objectives.
What, as a department, are we actually teaching
our physics undergraduates?
And Louis said, no one's doing this at other institutions,
because they think it's just too hard.
OK.
They think that, logistically, trying
to get all of the students and trying to get faculty
to agree about what the department learning objectives
are, these will be too challenging.
So Chris said, OK.
Let's do it.
And so this project was launched with some specially arranged
help from HILT at the HILT Conference, where
those of you who were there remember
that Chris was presented as a Guinea pig, if you will,
with an opportunity for the entire HILT conference
to weigh in on what Chris ought to do in his one
semester as interim chair to improve teaching and learning
in the department.
And one of the options was create
a departmental-level assessment to assess teaching and learning
in the department.
And lo and behold, that was one of the recommendations.
So with that ringing endorsement,
one of the curriculum-mapping projects that was started
was this very project.
So we wanted to know, how do we know
if the department as a whole is actually successful in teaching
the concepts and skills that we think
are important for undergraduate success in physics and future
success as they go into their careers or graduate schools?
How do we actually engage faculty
in thinking about departmental-level learning
goals?
As you can imagine, that can be a contentious issue.
And how do we increase our students' awareness
about what they know and what they don't know?
So we're looking here both at the undergraduate concentrators
and at the faculty to try to understand this better.
So our goals by June of next year
is that we'll have, we hope, a system in place
where all of our undergraduate concentrators
are assessed twice--
first around the end of their second year
and again in their fourth year--
that we have student-level and departmental-level
interventions in response to these results,
and that we can actually validate the assessment itself.
This is a little snapshot of part of the assessment, which
is something that we think is a very simple question,
conceptual question, in quantum mechanics
about the eigenfunction of a Schroedinger
equation for a semi-infinite-- well, anyway, I won't go on.
And our hopes for curriculum mapping
are that, in the future, when we have our curricular
reforms in the physics department,
we can actually target those reforms
at places where we know that we need
to focus in terms of outcomes.
And then we can actually assess those changes
on a department level.
And finally, we hope that our students can actually
be more confident and can articulate
what the skills are that they've learned
as undergraduate concentrators in physics.
And now, I will turn this over to Louis,
who understands this all much more deeply than I do,
to explain a little bit more about the details
of this project.
OK.
So a little bit about the design here.
So first, the test is made up of a broad range of topics, right?
We want to make sure the faculty and students feel like it's
quite representative of what it means
to be a physicist by the end of your fourth year.
So I made sure to see as many colleagues and department
as possible to get their input.
And I asked them, what do you think
are the most important skills, most important concepts,
that your students should master by the time they get out
of the program?
And I got a lot of feedback, obviously.
But a feedback that was common to everyone was about--
they wanted to make sure that this assessment did not
look just like a final exam at the end of a course, which,
in physics, oftentimes involves end of the chapter, end
of the textbook problems.
So they didn't want the exam to look like that.
And that's why, if you look at the assessment,
we have some practical lab physics questions on there.
We have even some open-ended questions.
So it really does feel quite different.
But most importantly, I wanted to make sure that it only
had conceptual questions.
And it's well known, now, in science education
that the retention of concepts is quite robust over time
when you compare that to retention of facts.
Facts like procedures, steps that you
need to solve an equation or something, that's just
forgotten almost at the same rate as a phone number
or someone's name, right?
In fact, several years ago, we published something
where we looked at how students retain quantum mechanics
concepts.
At the end of their first quantum mechanics course,
students were given a test of learning.
And the test of learning comprised deeply
conceptual questions, as well as procedures, right?
So a year and a half later, we gave them
the same test of learning.
And we made sure that they had no exposure
to quantum mechanics in between.
And interestingly, the concept was retained almost
at 100%, the deeply conceptual ideas.
But the facts, procedures, not surprisingly,
they had decayed almost to zero, to noise level.
So that's why I wanted to make sure
that the feedback that this test would give us,
faculty, and students was meaningful.
So in fact, you'll see, when I talk about student interviews
here, not a single student said, oh, I learned this before.
But I think I did not answer it correctly, because I forgot.
Because people know, intuitively, these questions.
If you just don't know how to answer them a year later,
that's because you didn't learn it in the first place.
So that's a very important part of the design.
So I tried to use as many validated questions as
possible.
I called my colleagues at other universities,
but we can only use a few.
So this assessment will have to be validated this year.
So let's talk about results a little bit.
Students did not perform as well as they hoped,
so with an average of 46%.
And the distribution was pretty wide.
So now, you should know.
If you ask, is this test too hard?
Well, I guess I started to give it to graduate students,
so they score a lot higher.
But still, not that great, much worse than they would like.
And I just got the green light, I think, in the last faculty
meeting that we had, to start giving it to my colleagues.
[LAUGHTER]
Anyway, it'll be anonymous, right?
I'll say I'm the only one that has the key, right,
so I'll be able to get favors from people.
So here's what's really interesting here.
Everyone, no exception, expected to get an A on this test,
and they were all very disappointed.
So that's a bit of a downer, right?
Very disappointed, in fact, to the point
where I was quite concerned with the few first students.
But I interviewed every student.
So I met them one on one.
And part of the script at the beginning for the interview
was about their perception of fairness for the test.
I asked them, is the test fair, in your opinion?
And I just let them talk.
Then after that, I provide them with a definition
we can both agree on for "fairness."
The definition I gave them was that I said,
let's imagine, for example, the questions on quantum mechanics
for this test.
Let's say that you had been asked those questions
at the end of your course where you learned quantum mechanics.
Would it be fair for the instructor in the course
to expect you to get all these questions right?
And without exception, students said yes, absolutely.
At the end of a course in quantum mechanics,
I should get those quantum mechanics questions right.
So they did feel like the test was fair.
So it was very disappointing to them.
But on the upside, students really
did appreciate the feedback.
I mean, many of them went out of their way to thank me, to say,
oh, I'm so glad that the department is
spending a lot of time and energy
to give me additional feedback.
And many of them asked, OK, what can I do practically, now?
These are students that are about to graduate, right,
in fourth years.
They said, only got a few months before I go to graduate school.
What can I do?
Give me some advice.
So it's pretty clear that we can't just give this assessment
and leave it at that, right?
We need to provide students with opportunities
to bridge some of the gaps.
So another upside is that the assessment can definitely
help students improve their self-awareness as they progress
through the program.
And that's in addition, of course,
to helping us improve teaching and learning in our department.
So something that's very interesting here
as part of the interviews is I asked
students, what is it that made you think that you should
have gotten an A on this test?
And not all of them, but almost all of them,
they talk about their grades.
They said, well, it's too bad.
The only feedback that I get from the program are my grades.
And I did pretty good in my classes.
So that's why I would have assumed that I would do well
on something like that.
So that was the source of their disappointment, basically.
So as Logan mentioned, when you think
about this, that's a pretty backward way to design things,
right?
You're supposed to do learning outcomes first.
And then from that, you're supposed
to build an assessment, right?
But we know it practically just doesn't work that way.
Well, you should have seen this in the faculty meeting,
the last one that we had.
I mean, people were really engaged, right?
So that might be a good way to trick faculty into starting
to talk about these things.
So the next steps here for this coming year
is, of course, we need to design follow-up opportunities
for students, some seminar-type courses.
We're bouncing ideas, what we should do.
We need to revise and validate the assessment.
And that will involve a lot of interviews.
So I want to make sure that we make
this available to everyone around the world
so they can use this instrument.
And Karen Pierce from Institutional Research
is already helping us now with looking at correlations
with other institutional data.
For example, what courses did students take?
What grades did they take?
And does that inform their performance on this test?
And then finally, and not the least,
is that we need to develop a sustainable
way to give that to students.
So we want to give it at the end of second year
and at the end of fourth year.
So students did mention in interviews,
I wish I would have had something
like this two years ago so I would have been
able to make some corrections.
It's too bad.
So giving it the end of second year, that's easy,
I would think, not just in physics
but in most departments.
Because we have core courses, right?
Like this fall semester, I'm teaching quantum mechanics.
And every student has to go through quantum mechanics.
So I'm going to give it to students
at the end of the course.
That's easy.
But at the end of fourth year, though, that's a serious issue.
This semester was really hard to communicate with each student
individually and schedule time and so forth.
[APPLAUSE]
Can you imagine a world where that was
happening in every department?