[Music]
[Applause]
hello world this is the cs50 podcast my
name is David Malan my name is Bryan you
and it's abit entire last episode but
that's only because Bryan and I and C
Swifty's whole team have been rather
busy here in Cambridge and in New Haven
teaching the fall version of cs50 on
campus at Harvard and Yale but we're
back now and we thought we'd begin the
coming year but with the discussion of
how the past fall went indeed the story
really begins back in June of 2019 the
start of summer here in Cambridge when
we gather the team together for
so-called summer 50 a retreat where we
all go away together for a few days to
discuss the coming summers projects the
coming Falls changes and really the
coming years curriculum for cs50 cs50.tv
AP and the theme that we decided to
focus on back in June of this past year
was a chapel on Tay to move forward to
evolve the course to change and make a
real thoughtful reformulations of what
it is we wanted to do and what it is we
wanted to teach in the class and that
led to a whole list of changes that we
actually announced toward the tail end
of this past summer by a cs50 s medium
blog titled a chapel on Tay as well it's
kind of amazing looking through this
list just like how many changes there
were that we were trying to put into
effect this year like it feels like the
course definitely changes from year to
year but this year especially it feels
like there were a lot of new parts to
the class and things that we were just
doing differently yeah absolutely this
was probably in the 13 years that I at
least have been helming the class
probably the most significant overhaul
the curses structure and curriculum and
design really so we thought we'd share
some of the thinking behind what those
changes were us do a bit of debrief as
to how we think things went and
highlight not only the successes we
think but really some failures things
that we thought were a good idea in June
of 2019 but didn't really pan out as
we'd hoped
so lectures for years have they been one
of the defining characteristics of the
course insofar as they're designed to
introduce the week's concepts
particularly those concepts that
students need for application on the
courses problem sets or programming
assignments and in recent years it's
worth noting that attendance here in
Cambridge in person on campus really has
been declining and this is not in and of
itself problematic and indeed we'd the
class and I philosophically have very
much been in favor of what we call here
on campus simultaneous enrollment the
ability for students to enroll not only
in cs50 but maybe even in some other
class on campus that meets at the same
or an overlapping time the implication
of which is that they have to physically
choose to go to one class or the other
insofar as cs50 s materials have long
been filmed and we played such emphasis
on the quality of the digital materials
we have been comfortable in telling
students go to the other class and watch
cs50 in person however this is of course
had the side effect quite causally of
relatively few students attending the
courses lectures in person even after
just a few weeks of the class case in
point I think in fall 2018 we had maybe
150 200 students out of nearly 800
physically attending by only the third
or so week of class and I don't think
that's really specific to cs50 like I
feel like all across Harvard and
probably universities more generally
like if there's a way for students to
watch lecture from the comfort of their
home without showing up there's gonna be
much fewer people that actually end up
showing up to those lectures yeah that's
quite fair certainly in computer science
and other fields here on campus there's
a long-standing tradition of filming
classes and that it certainly has value
and so far students can review material
if they miss material they can catch up
but indeed I do think we see students
walking literally with their feet
elsewhere well I'm curious when you were
in college where any of your classes
filmed and did you go to class or were
you more of the watch the video oh I was
super uptight and always the one going
to class I don't know if I ever missed I
I can think of a class whose sections I
missed I'll I'll admit that I realized
halfway during the term that an
economics class I was taking actually
had sections which somehow I forgot
about but I went to all of my CS
lectures and sections and at the time if
you missed a class back in what 1995 to
1999 you could go to the library for
some classes and ask to borrow the VHS
tape which you then watch on a machine
there although I did take a networking
class at Harvard I think in probably
1997 taught by Professor HT Cohen who's
still on the faculty here in computer
science and he was to my knowledge
the first professor to stream his
courses lectures online so it wasn't
live streaming but it used as a
technology Microsoft net show I think it
was so this is well before we had these
sort of YouTube and MP fours and all
streaming technologies of today and we
actually in a tiny window that was
probably like a hundred and sixty pixels
wide we're able to watch lectures from
home if we wanted well I think we've
made a lot of progress in swimming now
but in any case this is something that
we have been encouraging or certainly
allowing and endorsing but I do feel in
recent years and certainly last year
when we saw these numbers that too many
students I think were sort of following
in the footsteps of their classmates
without necessarily making an informed
decision themselves as to what would be
best for them to learn the material I
think too many first years of freshmen
who had just arrived on campus who had
spent the past four years presumably in
high school going to classes and now
we're suddenly stopping because they saw
more upperclassmen doing the same was
probably not the ideal informed decision
and ultimately it was just having a
ripple effect I dare say like cs50 has
such an operation behind it in terms of
so many Teaching Fellows in so many
sections and so many office hours that
if a student does not go to lecture and
then for whatever reason ends up not
watching the lecture live or on-demand
later on in the evening or even the next
day it means they show up at our
on-campus sections led by the Teaching
Fellows unprepared for that section and
then they dive into the problem set and
they're unprepared for the courses
material and then they come to office
hours and then they ask us questions for
which they should have been prepared had
they been prepared for section and prior
to that in person in lecture so it
really had an operational cost I daresay
which then cost other students insofar
as we had fewer resources to answer
their questions yeah and I really
noticed that I think that really
improved this year I felt like in
Section in office hours like students
came to them prepared having been to
lecture whereas normally a lot of
students would wait until later in the
weekly cycle in order to actually watch
the lecture and begin to engage with the
material so it was just more time to be
able to think about things to take the
pace of the problem sets a little more
comfortably just to make the week a
little more manageable I'm really glad
to hear that because that
was our hope that we would actually have
this ripple effect in a positive way on
subsequent support structures and oh my
god so liberating as a teacher to be
able to say something or repeat
something for a student and be able to
assume that they were there
the first time so that you can draw
contextual references and put things
into context where as in years past
there was maybe a 25% chance that a
student in talking with them had been to
the most recent lecture so I dare say
this change that we rolled out which was
quite simply to effectively require
attendance at lecture the course had
some nine lectures in total in person
this year and the expectation was that
students attend at least the first eight
of those which is not all that
unreasonable there are a couple of hours
a week it's only eight lectures and so
that felt like a reasonable compromise
even though philosophically I I dare say
I lean more libertarian and that
students should be just free to do what
they want but I do think it was a net
positive to try to help students help
themselves by effectively requiring
attendance but even that keeping track
of who was at attendance was non-trivial
oh yeah we're instrumental in solving
that yeah we spent weeks trying to
figure out like how do you take
attendance for a class this big like a
lot of other classes you can do the like
just call off all the names and mark
checkmarks but that's not gonna work in
a class with 800 students so we had to
go through a lot of different possible
options all the students here have ID
cards and you can think about different
ways of trying to swipe them or try to
scan them but even swiping them is gonna
take a long time if you only have one
swipe for like 800 different people all
of whom arrive in the five minutes
before class is supposed to begin of
right exactly yeah and you want to make
sure that it doesn't have a huge
bottleneck in a long line so we
eventually settled on a system where we
have these like four scanners the type
of scanners that you might see like at a
grocery store checkout when they're
scanning the barcodes on various items
we laid yeah laser scanners scanning the
barcodes on the ID cards and we had four
of them at different entrances to the
lecture hall and that tended to work
pretty well I think there were a couple
of lines that built up right as lecture
is starting but generally it worked
pretty quickly and we were able to get
people inside well I'm so glad it worked
out because it's a slippery slope we had
talked about writing our own custom
software and how you go about attaching
some kind of scanner to a tiny device
whether it's an iPod or an iPhone or
maybe
Raspberry Pi and there just felt like it
would be fraught with with technical
support challenges if something went
wrong you're sort of dead in the water
but these things are pretty dummy proof
they're meant to be used we saw for like
warehousing applications someone going
up and down an aisle presumably taking
inventory yeah I think that was the
intended use case and it works pretty
well when you're trying to scan a lot of
things very quickly so for our use case
I think it was a pretty good choice yeah
and so to incentivize students to come
because it's not a requirement that you
come to lecture but creates an
expectation we actually did for the
first time really though we saw some
prior incarnations of this idea and
experimental form in past years
essentially had a participation portion
of students grade where you get it all
if you meet all expectations coming to
section and coming to lectures as well
but it's amazing and it's amazing to me
that not only asking students to come to
lecture but incentivizing it as with
some fairly small numbers of points
toward ones final grade it worked
wonders we went from like 200 somewhat
students to 650 consistently week after
week I was amazed every time you guys
took attendance or reported back after
the scanners were done it being used
like it was around 650 every week so
does it feel any different when you're
lecturing in front of 600 people as
opposed to a much smaller Oh big time
feel more pressure when they're that
many people that oh yeah no I have
photographic evidence I sweat every
lecture this year whereas in the past I
actually wonder I used to blame the
temperature in Cambridge and the lack of
strong air-conditioning in the building
but I think it's my nerves my heart rate
and I should probably check my Apple
watch's history of my heart rate during
the 3 to 5 p.m. slot on Mondays to see
if that partly explains but ya know I
was much more nervous you know as well
as anyone how much more I and we
prepared before each lecture because you
just have so many more eyes on you and
it's good pressure in a way and the
energy is just so much better we had
many more volunteers there was more
audience participation both in terms of
questions and answers and when you have
the occasional applause or laughter it's
sort of infectious and a good way and it
actually does motivate I think the
experience of coming to lecture so we
didn't want people there just for the
sake of being there we wanted them to
have a hopefully and inspiring or at
least an eye
opening experience and I think you can
pull that off like eight times a
semester I I would not want us to do
what we did some thirteen years ago and
have three lectures a week so that we
now have some 30 classes that students
have to attend in person beyond the sort
of upfront cost of setting up for and
striking our AV setup for each of those
lectures I just don't think you need 30
times together to sort of feel that
shared community experience like eight
or there abouts feels reasonable yeah I
think so and I student it's nice because
students get to get to know each other
based on the people there sitting next
to and they can have conversations about
the material which you just can't do as
well if you're just yeah at your desk
alone watching the lecture video well
especially for first years theoretically
maybe even rub shoulders with more
people that you meet and then stay in
touch with more than you would if you
were just in the library or at home
watching on your laptop now we should
concede that we also changed the day of
lectures this past year the past couple
of years we've been using fridays or at
least last year we did and it was friday
mornings of all things I think was nine
nine a.m. officially to 11:45 that we
typically started a little later closer
to 9:45 so that we still had a two-hour
window and we know statistically from
cs50 zone data and Harvard College's own
data Friday is hands down the least
popular day for classes both among
students and reportedly among faculty
it's hard enough to get students awake
in the morning let alone Friday mornings
let alone David and the team but but we
steered toward Friday because of exactly
this it was wide open on Sanders
theaters schedule which meant we had the
ability to book the room the entire day
which is great for cs50 production team
because they Skaar I've some 6 a.m. for
a 9 a.m. or 9:45 a.m. class just to set
up the a/v equipment and technology and
test everything so there's a huge amount
of setup and so just having that runway
was liberating plus theoretically and
tragically it didn't work out this way
you'd think more students would be free
on Friday mornings 9:00 a.m. because
statistically they don't have conflicts
or other classes and there probably
aren't extracurriculars going on at that
hour but there is a lot of sleeping or
non attending going on it
- yeah a lot of students just tried to
like free up your Friday like it was a
goal among many of my friends senior
year to like try and arrange your
schedule so you didn't have to go
anywhere on Friday and people tried to
give you that day yeah they were very
successful at that in fall 2018 so in
short I think this was a net positive I
daresay I think the lectures and
hopefully the audience is the students
engagement with them was so much more
positive and experiential this year than
in past years hopefully we really did
set students up for more success each
week insofar as they now had gone to
lecture and therefore went to section
prepared and then went to office hours
or didn't go to office hours because
they didn't necessarily have as many
questions and ultimately succeeded the
problem sets because they had all of
those building blocks ready ready for
them and without jumping ahead too much
I think it's okay I would like to thank
although there's many other factors that
we should discuss some day cs50 s grades
ultimately skewed higher this year I'm
on the courses problem sets and we think
this might be for a few reasons but I
would like to think that one of the
reasons was that we actually set
students up for a better educational
experience simply by incentivizing
motivating expecting that they take
advantage of the resources that frankly
have always been there but were perhaps
being underutilized yeah that would be
nice I hope I certainly hope that's the
case
me too me too well this wasn't the only
change certainly and we hypothesized
that by moving cs50 s lectures to Monday
afternoons and not having sections until
Tuesdays and Wednesdays that allowed us
Monday evenings to even provide students
with an opportunity for review and this
is a term we keep deploying in different
forms for several years but this year
supersection took a new and improved
form yeah so the goal of the super
sections which were kind of new this
year were to be an opportunity for
students after lecture after they've had
a little bit of time to think about
things have some dinner to come back and
ask questions about that day's material
because after the lecture the problems
have to immediately release that
theoretically students could begin to
work on it but oftentimes you know in a
long lecture where a lot of topics are
covered not all students are going to
get a chance to ask all of the questions
that they might have especially in a
class as big as cs50 so one of the hopes
I think of the super sections was to
just be a chance for students to sign in
virtual
in a virtual classroom we use zoom which
is like a video conferencing tool to do
it and students can sign in and then I
would lead these weekly super sections
where students could ask questions I
would show some additional examples
based on the lecture material just to
give students some more opportunities
for Q&A and some more opportunities to
really get a firmer understanding in
that day's lecture material in
preparation for that week's quiz but
also in preparation for that week's
problem set overall I feel like they
were very helpful for the students who
were there like they got the questions
answered and there were always a lot of
questions that students didn't have they
weren't super well attended I'd say
maybe we had anywhere from fifteen to
thirty people on any given week I think
part of that is probably just you've
already been to a two-hour lecture on
Monday and now that's another hour plus
on a Monday evening maybe was
overlapping with some people's dinner or
other extracurricular commitments but I
do think that on the whole for students
who attended it it was helpful and we
made the recordings available to
students afterwards so even if you
weren't able to attend in person you
could review them later and I know some
students did that for preparation for
the exams and that type of thing yeah
and I do think I as you know was such a
fan of the idea of trying out zoom or
videoconferencing more generally for
this because if you decrease the
friction for students being able to
access something like this right they
don't on campus have to walk 15 minutes
to get there they don't have to walk 30
minutes after especially if they only
have one or a few questions you would
think that many students wouldn't be
availing themselves of resources if
there's too high of a operational cost
to them or too much friction between
them and the resource but I was
disappointed honestly like it is great
for the fifteen thirty or so students
that that attended and it was often a
recurring group of students we saw some
familiar faces many weeks but I don't
know I mean out of 800 students on
campus a hundred plus through Harvard
Extension school is fifteen to thirty
out of 900 ish a success or a failure do
you think for something like this I
think it's probably too small to really
call it a success because it was a fair
amount of time investment that the
production staff had to be there in
order to make it work and to post
produce the video and make it available
online and so I'm not sure now granted
part of it may be that there were just
other avenues that students had to ask
questions it's not one of our intended
changes that we originally reported here
but we started working with
a new question-and-answer software tool
called Edie this fall for the first time
that just I think was a much nicer
interface for students to be able to
quickly ask questions and get feedback
from the staff and I feel like a lot of
students ended up using that tool as a
way of asking questions too so maybe
that was another avenue that summer to
nate'd ask questions would you be
inclined to have us nix them the coming
year do you think it's a good question I
think right now I would probably lean
against doing the super sections at
least in the same form that we did them
this past fall if it didn't seem like
there were nearly high enough impact but
the super sections have evolved a little
bit over time like two years ago they
were in person but they weren't every
week and they were just on the weeks
when we thought the students would have
needed them and then attendance with a
little bit higher so maybe I wouldn't
keep them in their current form but
maybe there's some changes we can make
and I'd be curious to hear what like
feedback from students is later about
them yeah absolutely and we did survey
students throughout the semester so we
have some good data there but this is a
concept super section so larger than
life sections where we've tried for
years to get these off the ground
successfully and they always have been
this sort of middling success and I
remember some 10 years ago when the
super sections were only in person I
think by the end of the year we had one
student attending at the end but you
know fun fact she went on to be our head
Teaching Fellow for three years so oh
it's a wonderful recruiting mechanism it
seems to see who's really really
passionate about the material and
another one of the goals of the super
section at least this year was because
it was the first year that we had weekly
quizzes which I do want to ask you about
the super sections were designed to be
an opportunity to help get some question
and answer or get understanding in the
material in order to allow students to
be able to to work on those quizzes so I
did want to ask you about about those
quizzes like what was the motivation in
your mind behind them and how do you
feel like that went around a really good
question so cs50 for many years has had
a tradition of either quizzes or tests
or which are really just midterms in
some form and the the format of them has
evolved over time but in recent years
they've taken the form of sort of
thought based short answers questions
that aren't really regurgitant
Tyrael from the class they really are
meant to pluck up pick up some idea and
take it further than class time allowed
for or frankly just introduced some new
topic altogether and so this year's
quizzes which were weekly google form
based small assessments two or three
questions that we gave to students after
each lecture due the next morning the
idea being we wanted them to submit
these quizzes after they had attended or
if maybe watch lecture we wanted them to
submit them before they attended section
so that they were thereby with higher
probability arriving prepared for
section and awesome honestly we also
wanted to use them as a distribution
vector for the types of questions that
we didn't really have much time for in
other windows during the term we only
had time in terms of the academic
calendar this year for one test whereas
in some years we've had two such tests
or two quizzes or the like and we wanted
to be able to integrate some kind of
short answer questions throughout the
course that don't necessarily belong in
a coding based problem set these really
are different types of assessments and
they're not even just assessments they
were meant to be also instructive in
some form so they ended up I think by
design and I was complicit in this
probably being a little more evaluative
of the previous material I think we
tended to cherry-pick relatively
accessible topics in large part because
we knew students had only just seen the
material they hadn't yet had an
opportunity to review necessarily or
think about it or apply it in a problem
set so I don't think I was I was in love
with the types of questions we asked but
as a mechanism for incentivizing
preparation for section I do think they
were successful I mean students were
putting in the time and if there was any
design flaw besides the the ideal
questions I wish we had been able to ask
was really just the timeframe I think if
a student finishes cs50 at 5 p.m. even
if they're there in person then they go
off to dinner then maybe they have a
commitment or they certainly might have
work in other classes not much room in
the evening to actually fit it in before
the deadline but of course our section
started on Tuesdays around noontime and
so we wanted them to submit in advance
so I think my takeaway is they're worth
drawing I think they were sufficiently
successful but I think we need to give
students a slightly wider window
yeah it is tricky just how I wish there
was more time in the week to be able to
like have an extra day or something
where we could in order to make room for
this type of thing because I do think it
was a good thing that the sections were
earlier in the past last year you
mentioned that the lectures were on
Friday and because that was at the end
of the week it meant realistically most
sections didn't happen until Monday
which meant we sort of lost that weekend
of time this year I thought it was
actually a really big improvement that
right after lecture on Monday section
started on Tuesday which meant very
early on in this weekly cycle of lecture
than section and problem set students
were able to see some more examples get
some more practice and then really feel
like they had the ability to go forth
and actually start working on the
problem set earlier than they would have
been able to last year and I think that
was a really good thing but it also
meant less time for the quiz we really
need an extra day it leaves between
Monday and Tuesday so if anyone out
there is listening we need an 8 a week
and then cs50 would be would be better
well so sections 2 we have taken varying
approaches - over the past few years and
this year we've continued to expect
attendance which was something that we
rolled out a couple of years ago and we
also tried to make them more hands-on
because of a recurring bit of feedback
from students is really that they don't
have much time or much opportunity
between lectures and p-sets to really
apply what was in lecture but then is
expected of them in psets so how
accessible do you think this was overall
but the team yeah this is a really
common piece of feedback so every
semester after cs50 is over I go through
and read our Q evaluations the Q
evaluations or Harvard's student
evaluations of the courses where
students answer a whole bunch of
questions about things they liked about
the class things they didn't like and we
go through and we read this every
semester to see like what is the general
feedback and a big piece of feedback
about the sections was like I really
like my section leader but it feels like
section is just a rehash of lecture
feels like we're going over the same
topics from lecture and there were a
couple of reasons why that might have
been the case one is that last year a
lot of students came to section having
not watched the lecture and so as a
result of that more of section ended up
having to be like let's go through the
lecture material again to make sure that
no one feels totally lost or in the
course of the section so I think one
improvement this year is that because we
could assume with higher probability
that students were actually at section
the sections were able to be a little
bit more interesting we
have to rehash what happened in lecture
and we could actually go forward a
little bit
but another common piece of feedback
with students saying that they wished
they had more opportunity in section to
actually try things hands-on but it felt
like they understood things conceptually
in lecture but then struggled when it
came to working on the problem set to be
able to know how to actually write the
code to implement what they had in their
mind and I think that's a common
struggle in computer science you can
understand a concept or an algorithm in
theory and conceptually and when someone
draws a diagram about it but when it
comes time to actually start writing the
code like that can introduce new
challenges and it can be tricky in ways
that you might not have expected before
so one of my big goals with sections and
what I was trying to work with a lot of
the teaching fellows about was making
sections very hands-on to be able to
talk about topics but really to give
students opportunities to work through
practice problems to try approaching
problems in different ways to let them
work in pairs together as they tried to
write code to solve smaller problems not
as big as a problem set but still
related concepts that were going to come
up on the problem set just to give
students practice with taking a problem
formulating an idea in their head and
actually writing the code to be able to
solve it and I think that ended up
working really really well that for a
lot of students from cs50 is their first
exposure to computer science into
programming just that little bit of
extra practice with someone there to
help guide you and to to point you in
the right direction if you find yourself
lost or struggling that that adds a
little bit of confidence that you feel
like oh you can solve these smaller
problems and now this larger problem set
maybe is a little bit less intimidating
well I'm really glad to hear because
this is something we've been trying for
years and it's such a simple thing the
fact that students are more prepared
theoretically for class that that makes
a difference but hopefully we're getting
even the finer points just right now -
yeah and I feel like in just talking to
students they seem to anecdotally enjoy
their sections and feel like they were
getting things out of it our course
evaluations for this semester haven't
come out yet for the college I think
they're supposed to come out next week
or so but I'll be curious to read those
and just see what all the feedback is
about sections and lectures and all the
other changes that have been happening
this semester yeah absolutely one
another big one - was really the third
of the primary curricular support
structures for students No
his office hours which for us take the
form of one-on-one opportunities for
help with the courses problem set and
over the years we've offered these in
different forms usually several days or
evenings of the week we have reserved
some locations on campus whether it's a
computer lab or dining hall or library
for students to drop in with their
laptops and ask questions of the courses
staff and we usually have multiple
members of the staff Teaching Fellows
and course assistants there to answer
questions but you know for as long as
I've been teaching cs50 and frankly
probably taking cs50 back in the day
have office hours been one of the
greatest challenges of scale with a
class of cs50 size because even though
we throw as many resources as we can as
best we can with invariably there's a
line there's a weight and it's
significant sometimes I mean we have
quantitative data over the years where
some years students we're waiting at at
least an hour just to get their question
answered and if you liken this to the
real world where maybe you go to a store
and you're waiting for help from someone
or you call someone on the phone or open
a chat window to get technical support I
mean that's the same kind of experience
and these are students in an academic
environment just trying to ask questions
and clear up some confusions it's been
really hard to keep up with the load so
this year we tried to rethink these
fundamentally and we have tried for so
many years to gather students in one big
place like a dining hall or library and
a lot of members of the staff sometimes
10 or 20 or even 30 plus members of the
staff all at once but even then we've
never really achieved an optimal ratio
of staff to students and if you think
about it even if you have as good of a
ratio seemingly good is like one to five
one teaching fellow to five students
even if that teaching spell fellow
spends only 1012 minutes with each
student it's gonna take them an hour
just to get back around to the first
student again so that's the sort of
sense of the problem here so we rolled
out tutorials and tutorials for us were
opportunities to for students to sign up
for office hours quite simply by
appointment and we capped attendance at
these tutorials at five maybe six
students the idea of which was even
though that's not necessarily an ideal
ratio we removed students from the
stressors of from the pressures of from
the chaos of these larger space
is that frankly we're all too conducive
in a bad way to stress and rising stress
levels and we spread them out on campus
and offices and conference rooms and
dorm rooms and dining halls and what do
you think how did they go I think there
were an improvement and the word you
used before led to chaos I think that
was a good word to describe what office
hours were like in in past years that a
student shows up to office hours and
they're just so many other students that
are all trying to get help at the same
time and there's a limited number of
staff that are themselves stressed
because they're trying to balance
between everyone there was a staff
member who took the class last year and
when I was talking to her about what her
office hours experience was like as we
were talking about office hours this
year she said to me last year it felt
like going to office hours was like
going to war and he would like show up
and you would have to like fight against
all these of the students in order to
get a couple minutes of attention from
one of the staff members and that just
wasn't healthy for anyone nothing was
helping the students or the staff
members no I used to be terrified of
Thursday nights when problem sets were
due on Fridays yeah and I think the
tutorials just they feel so much calmer
like the ratio is 1 to 5 and we
eventually had to change it to 1 to 6
just because demand for them was so high
which i think is a testament to students
valuing the help that they got at these
tutorials but going to a room where
there's just four or five other students
just feels way healthier than going to a
room where there's a hundred people all
trying to get help at the same time that
was a small group of people that was
certainly our hope that you could the
fact that the Teaching Fellow was just a
few feet away and not a few tables away
just meant that you knew sort of
socially that they're gonna get to you
shortly even if it is maybe 10 or so
minutes it just doesn't feel the same
and yet tragically I mean there was
motivation for our larger scale office
hours so to speak especially in the
dining halls and office hours one we
originally hoped that they would just be
much more social and much more
interesting than the basement of Harvard
Science Center where they once were back
in yesteryear but to there's just an
economy of scale right with a 1 to 6
ratio in some corner of campus there's
arguably an inefficiency and if there's
some quiet time or lull time that
Teaching Fellow is just kind of standing
there waiting they're not actually
helping anyone whereas in the library or
dining hall if there were a student at
the other table that's EF could wander
off and go help them so I do
think in theory the larger-scale office
hours were a better allocation of
resources but it came with a lot of
negative externalities yeah and there
was also the benefit of having many
staff members in the same place like one
common thing it would struggle with the
tutorial says there's one staff member
with five students and as a result of
that if a staff member is struggling
with something in the larger office
hours where there's a lot of staff there
the staff member can turn to another TF
and ask them for their input or multiple
lives can be on the same piece of code
to try and help things together it was I
think a greater challenge for the staff
to lead these tutorial so I think
they're harder to leave than office
hours because there's a lot more
independence involved in them that you
need to be able to solve problems on
your own
because there isn't always going to be a
whole bunch of other staff that you can
turn to at that particular time well we
did try to mitigate that by co-locating
some of the tutorials so that in the
same room or same building there were
two groups of five to six students in
one or two TFS do you have a sense of
whether that indeed helped by just
having two smart people in the same room
instead of just one I think that does
help and I think as much as possible
trying to pair people up is useful for
just being able to bounce ideas off
someone else and to do a little bit of
that efficient allocation of resources
where if there's a lull in one area a
staff member can go to another area in
order to help students there too but
it's not quite the same as having a lot
of staff in the same place and being
able to brainstorm things together
because there are definitely issues that
are tricky enough that I remember like
standing around a computer with two or
three other staff members on a
particularly tricky bug or something
that none of us could quite figure out
for a little bit of time until we really
thought about it together and so I think
there are definitely trade-offs but I
think on the whole these tutorials have
been much healthier than the office
hours and just a better experience for
students yeah no this one's a keeper in
my mind this is really the best yeah
changes for sure and yet I think it's
only fair that we admit we were a little
nervous at best terrified at worse that
these were going to blow up on us in
that we knew that even though we have a
teaching team of some 80 Teaching
Fellows and course assistants for our
eight hundred some-odd students there
was not going to be enough slots during
a week for everyone to sign up for a
tutorial if they want we were kind of
banking on the fact that numerically
based on the number of people we wanted
to have maximum leader
tutorial and based on the number of
tutorials maximally that we wanted to
expect of our teaching staff for each
week so that they balance their other
responsibilities and their other classes
we couldn't handle a hundred percent
uptake of tutorials we could only handle
roughly 50% but we week by and I think
you noted that we increase the capacity
from five to six not a huge problem but
that definitely speaks to I do think the
the uptake in the success of them yeah
I'm looking at the numbers here and just
we recorded the attendance at all of
these tutorials so we could look back at
the end of the semester and see like how
many people were actually showing up to
tutorials as compared to office hours
and in week one it looks like about 200
students signed up for office at or for
tutorials rather in the second week it
was like 380 or so in the third week it
was 500 and I think it was 650 so they
became more and more popular week after
week after week and so around the
midpoint of the semester I was
definitely starting to get a little bit
nervous about how many signups we were
having and we only have so many staff
that can only work so many hours and so
there's only so much help we can provide
at these tutorials but ultimately I
think it ended up working out well well
and thanks to you we had a special tool
for this too cuz we needed some
mechanism via which students could sign
up for these tutorials so Brian kindly
whipped up a web-based tool that
students log into and then they click a
button next to the day and time and
location that they want to sign up for
and it also had the benefit of capping
attendance so that only five or six
people could sign up for a given
tutorial but there's interesting
sociological things that we discussed
over time too because it was not
uncommon for students just to be
no-shows or to email us at the last
minute and say they couldn't make it and
this was actually a hard problem because
here we had a scarce resource that we
were trying desperately to ensure was
being efficiently allocated to as many
students as possible and the fact that a
student might sign up for one of say
five or one of six slots and then not
show or cancel at the last minute such
that there's no time for someone else to
take it was really frustrating and it
was hard to sort of deal with and
students were often apologetic but I'm
not sure some of them appreciated that
this was a resource you're effectively
taking away from someone else because we
were at capacity certainly mid semester
yeah I don't know if it was is totally
apparent of students just how scarce
the resource necessarily was that if
you're signing up for something where
there is no cap maybe you feel like oh
it doesn't matter if I don't show up
that's fair but yeah with the tutorials
that is it is different and it was
always unfortunate when a student didn't
show up and another student probably
wanted that time and it were only
available at that particular hour and so
weren't able to attend as a result well
and here I dish say was an opportunity
for us a teachable moment and what we
actually tried doing in software helped
with this was we prevented the student
who was a no-show and who did not give
us enough time to reschedule someone
else we prevented them from signing up
from a future tutorial until they took a
moment to email us the courses heads
just to sort of ask that that hold be
lifted the idea being to sort of send a
social message and also hopefully an
educational one that like hey okay
something must have come up life happens
that's fine but you need to handle this
differently and I think it would be
interesting to look retrospectively at
the data to see just how frequently was
the same person a no-show and to see if
we actually had some positive impact
here on expectations yeah I haven't yet
looked at that data but I have a feeling
like I can't remember too many multiple
no-shows where we had to remove the
ability to sign up for tutorials
multiple times for the same person I can
take another look at the data but that
would be an interesting question that
I'd like to get the answer to yeah
well shall we move on to one of our
biggest failures all right go for it so
I was such a fan in June of 2019 and for
the months after that of what we called
code reviews ultimately the idea being
not only in academia but the real world
sitting down with someone or digitally
corresponding with someone about the
quality of your code and getting typed
or verbal feedback about what you could
have done better or differently or
otherwise with your code we've in cs50
have of course experimented with
different forms of feedback over the
years correctness which is now automated
style which is now automated and design
which historically was done by humans
but this year we deliberately eliminated
the quote unquote design feedback from
students scores in large part because
one analysis of past data that we've
collected say in fall 2018 suggested
that the numbers we were assigning to
students design scores really weren't
influencing their final grades as a
result but the
cost operationally to providing that
design feedback was huge of the 12 plus
hours a week that one of our teaching
fellows might work on cs50 and work with
their students they were spending
upwards of six or more hours alone on
grading this is an isolated activity for
many of them it's heads down in front of
the laptop spending hours on end
providing feedback to students and then
we knew moreover quantitatively for Mom
that students were barely engaging with
this con with this feedback and
sometimes weren't even looking at it
which really called into question so
while undoubtedly I do think there is
value to providing very precise feedback
when it comes to the design of some
students code what could they do better
and why the reality is I think we made
the right call reallocating the teaching
fellows time to more tutorials or more
office hours or more in human time a
human time in person not in human time
is was probably a net positive given a
finite number of hours in the day yeah I
think so and I remember the frustration
of like having to do hours and hours of
grading and then students not reading
any of that feedback that I spent a
whole lot of time working on it just
felt like an inefficient use of my time
if I'm spending all this time and
students aren't actually reading any of
that feedback and so this year fall 2019
we introduced code reviews the idea was
that alongside the courses tutorials for
which students could sign up you could
also via the website sign up for one or
more code reviews for feedback from a
live human via videoconferencing to make
things easy and in-and-out to get
feedback on your most recent submission
of a problem set the idea being to still
provide on an opt-in basis as much
design feedback as students would like
on their code but more tailored to the
students who actually did want it
unfortunately
roughly how many students would you say
wanted it I think we got maybe two or
three signups and any given week out of
a class of 800 plus students yeah this
was a big fail I think it was a
wonderful idea in spirit but I can only
surmise that conflicting demands on
students time the fact that this is
optional the fact that this is opt-in
maybe the fact that it was by design on
video was just a bit strange even though
that meant they could pop in spend a few
time a few minutes talking with the
staff member and pop
was theoretically a compelling thing but
in practice it just flopped yeah I think
part of it is that it was totally
optional and much like the super
sections were optional when it's ops no
there's just much less engagement and
the other piece of it I think if I just
had to guess I don't have any data to
back this up but I think it's just a
matter of the context switching that
when a week ends a lot of students you
go to another lecture you want to focus
on the next week and the code reviews
are the only part of the week that
really aren't focused on that week
they're focused on the previous week
it's looking back at the work you did
last week which is a valuable thing to
do to learn how to do things better but
in a academic semester that just moves
so quickly week after week after week
not only in cs50 but in other classes I
think it's a challenge to get students
to opt into engaging with stuff in the
past yeah I agree and here too if
anyone's listening if we could have
twice as many weeks we could then have
an on week in an off week because truly
I think pedagogically that would be the
idea after submitting some work you have
an evaluative process a feedback loop
even an opportunity maybe for students
to work on further and resubmit the
previous week's problem set so as to
just get better at it and then actually
take into account actively that feedback
as opposed to just passively looking at
or hearing it so we'll try again but
rest in peace code reviews they won't
return next year in this forum shall we
turn finally to perhaps the other
biggest change this year which was the
courses notion of tracks in which you
and a few others were instrumental yeah
so this was a really big change to the
real structure of the curriculum of the
course the course for a long time has
had the last couple of weeks be focused
on web programming where students learn
HTML to build web pages and they use
Python in order to build web
applications and we had a little bit of
JavaScript and CSS on top of that as
well and one of the effects of that we
found was that a lot of students final
projects when it came time for the cs50
fair were all very very similar there
were a lot of web applications that all
looked fairly similar and a couple of
students would take it upon themselves
to try to teach themselves how to do
something different how to do something
with hardware or how to build an iOS app
for example but it definitely wasn't a
lot and we thought that
maybe part of that was just the fact
that because we teach how to build web
applications that we sort of
inadvertently pushed students towards
building web applications for their
final project even though there are a
lot of other options that students could
pursue as interesting things that they
might find fun to build on their own or
in a group so the tracks were a change
we made to the curriculum this year
where we actually gave students a choice
of what they want to do with the last
two weeks of their semester instead of
just pushing everyone into web
programming we said you could pick what
you would like to study for the last two
weeks you could pick web programming you
can pick mobile app development with
either iOS or for Android you think Java
or you could pick game development if
that was something of interest to you
and we had separate problem sets for
each of those individual tracks we had
lecture videos so I led the web
programming videos Tommy MacWilliam the
former head Teaching Fellow for the
course led the mobile application tracks
for iOS and Android and Colton Ogden who
runs the games course at the Harvard
Extension school and on EDX he led the
game development track so we all
prepared lecture videos we all prepared
projects and let students pick what they
wanted to do with their last two weeks
but how would you assess the end result
the end result so there were it was a
good idea in theory I mean one thing we
definitely found was that a lot of
students just skewed towards web
programming web programming was like I
think 80% plus I think you Denny 7 / 87
% yeah that sounds about right
ended up choosing web programming there
was a smaller percentage for iOS and
games and then I think there was like
one percent of students ended up
pursuing Android in part maybe because I
think just based on statistics at
Harvard almost every 99 percent of
students have I phones yeah 99 percent
of students we have iPhones Android is
for whatever reason not super popular
around here no but to be fair in the edx
version of the class in the
opencourseware version of the class for
which these tracks are now available as
of january 1st i do think we'll see a
lot more uptake of the android track to
be fair yeah i definitely agree with
that and i think it's a good thing that
they exist for students who want to
pursue them but it was interesting the
things skewed so heavily towards web
programming this past year and maybe
that was just students from prior years
that were encouraging
this year's students to do web
programming because that's what they had
seen before but I'm not entirely sure
all the factors that might have
contributed to that skew yeah and I'm
not sure I'll admit that we should keep
these next year
I mean 87% toward web is significant and
there was a huge cost and time
commitment not only view but of Tommy of
Colton cs50 s production team and the
like because we essentially mushroomed
the last two weeks of the class into
four times as many lectures and videos
and problem sets and and trainings - I
mean we spent a good part of October
asking that the staff ready themselves
for not only being prepared to help
students on the web track but also one
other track and our TAS at Yale did the
same thing and that too is not without
cost and even then though the TFS were
moderately prepared to help students the
reality is that just by having done the
web and the mobile and the games tracks
themselves the problem sets therefore
doesn't necessarily mean they're
comfortable with ass answering questions
that they themselves didn't stumble with
or come across on their own so support
wise this was actually quite hard I mean
this is true of any change we make to
cs50 whereby if we don't have the luxury
of 80 staff members who previously
talked took the material the previous
year it's hard no matter what but we
definitely exacerbated that problem this
year was harder to help students and we
imposed more of a burden on a few of the
staff who wonderfully stepped up and
effort as many of the questions as they
could but it was tough keeping up with
this yeah definitely and I remember a
similar scale thing they're not quite as
large-scale when we made the switch from
PHP to Python 2016 that that was a
challenge for the staff - that had never
seen Python before to learn Python in
just a couple of months in order to get
ready to work with students but the
difference there I think is the whole
staff were all focused on the same
material yeah they're all collectively
learning Python could all turn on each
other whereas for the tracks we thought
it'd be unreasonable to expect that a
staff member in a month learn all three
of the tracks to be able to work with
students so we had each TF specialize in
one of the three additional new tracks
and the result of that is that if you
had iOS questions there were fewer staff
members that had done the track and were
able to answer questions about it so
that was definitely a challenge - when I
think we really felt it at the CS
the hackathon this all-night event that
we run each year during which students
work on their final projects because we
really I think noticed how many iOS and
Android and games questions there were
and how few of us during the evening let
alone at 2 a.m. 3 a.m. 4 a.m. were
qualified and awake to actually help
students I mean I was actually surprised
to that 87 percent was as high as it was
because the whole motivation for the
introduction of the tracks was the
hypothesis that there are a non-trivial
number of students who take a class like
cs50 for whom the course isn't
everything they want it to be because
it's trying to keep so many students
such a broad demographic of students on
the same page and so the tracks were an
opportunity for students we thought and
we hoped to sort of customize the course
and tailor it to their own personal
interest so they get more out of the
course that's of personal interest to
them as opposed to it being
one-size-fits-all so in theory we did
put that structure in place but
surprisingly did 87 percent of the
students lean toward the version of the
class we long had so seems we somehow
latched on to the the most popular track
certainly some 13 years ago when we
introduced it yeah and I may be part of
this too is that like when I was doing
the videos for the web programming track
I was basing it off of all of the work
that you had done in past years on
designing the lecture materials and the
examples for demonstration in that track
so this is a track that's been around
for a while longer and so maybe that
just attracted more people to it as well
whereas you're very attractive yeah but
I do think it's worth noting too that
the other challenge that we would have
to wrestle with even if we decided to
keep these tracks this coming year is
that especially in the mobile world the
software stack and the technologist
change is so darn fast
oh yeah changes really quickly like when
Tommy recorded those lectures about iOS
I remember reviewing that maybe a couple
weeks later and in those couple of weeks
Apple released a new version of Xcode
the tool you use in order to build all
these iOS apps and the user interface to
change from what was in the video and so
the button that Tommy was saying I
should click on it just wasn't there and
so I'm sure within a couple of months
especially a year from now things are
going to change even more yeah and not
only the user interface but some of the
paradigms and the techniques or the
function calls or the like which means
the materials got to be refreshed and so
there - it's a vicious cycle where as
much of cs50 score the first
the first eight nine weeks as well as
the web track are less sensitive to I
think changes but that's testament to
the fact that it's meant to be an
introduction to computer science not an
introduction to iOS or to Android or to
game development or the like
so that at least is a core worth keeping
yeah and I think a lot of the other
topics of cs50 touches have been around
long enough that they change much more
slowly like python undergoes changes but
they tend to be backwards compatible for
the most part and so python might have
new features but the examples from
python from last year or two years ago
those aren't going to break even though
a new version of python that's out
indeed and I am so glad a few years ago
we decided because we discussed this and
debated this internally should we go at
the time with Python 2 or Python 3 since
python 2 was still pretty much
omnipresent on people's computers but we
went from three we went with three from
the get-go and thank God because January
1st 2020 came around which is when
python 2 was officially deprecated even
though it's certainly still around we
didn't have to stumble yeah that's
really I fit everything that's just
already Python 3 so makes it easier for
us indeed so that was 2019 a mix of
successes and failures and in variably
Brian and I and the team will be talking
about these very same topics and ideas
and new topics for the coming year by
all means feel free to write us as
always at podcast at cs50 dot harbor edu
if you have suggestions for future
topics or feedback on this particular
podcast and some of the interventions
and ideas by all means please share some
thoughts of your own definitely hoping
to hear from you
this was the cs50 podcast my name is
David Malan my name is brand-new and
we'll see you next time bye Ron