Well, we may be seeing more of this, right?
The future is now Harvard is hiring a new
teacher for its introductory level coding course this fall.
But it's not a teacher in the sense that you would think it's an A I chatbot.
The university plans to incorporate chat bots to
help students find errors in their coding,
answer questions,
offer feedback and help students learn more about the coding process, right?
A computer helping you learn about computers. Essentially
joining me now, David Malan,
he is a Gordon mckay professor of Computer Science at Harvard University
and helping to oversee this new course in the chat bot.
Thank you for being with us. Um So
how did this come about initially?
Sure. Thank you for having me.
So we have a long tradition in this class CS 50 at Harvard of using software to
better support the TAS to help teach the class
and also the students who take the class.
So for us,
this A I check bot is really just an evolution of the
sorts of tools that we provided for students to for some time.
Uh It's sufficing to say when you have hundreds of students on campus.
And in our case, by Open course Square and platforms like X,
we have thousands of students online.
Uh we get quite a few frequently asked questions.
So among the goal of the chat bot is to answer
as many of those questions correctly for students as it can.
So the chat bot is just like we're seeing on screen,
basically a texting back and forth.
It's not like an actual A I helping to teach and advise students
like, like in the sense of like a face, a person, a name, right?
No, it doesn't have any physical form just yet.
It is indeed very similar in spirit to chat GP T like you're seeing on the screen.
But unlike chat GP T and tools like it right now that tend to be too helpful by
literally just telling you anything and everything that you
might want to know about out some subject.
What we've been doing over the past few months is trying to
teach the bot to be a little more like a teacher,
being a good teacher leading you to an answer, but not just handing it to a student,
right? Because the risk would be, students would become too reliant on it. Right.
Turning to the chat bot to do their work.
Exactly. Ok.
So what are some checks and balances that you all
are putting in place um to mitigate those potential risks?
Yeah, so much of what's happening right now.
In the space of chat,
G BT and large language models in general is
the ability to feed these types of programs,
prompts uh instructions in English that essentially give the bot
a personality or guide it to a certain behavior.
So the sort of prompt that we're providing to our own bot is akin to teaching it.
Assume that you are a teacher of an introductory computer science course,
answer students questions in simple terms,
do not provide them with more than just a few lines of code.
The goal being really to put downward pressure on just how helpful
chat GP T and other tools might otherwise be by default.
So the goal is to make it much more akin to a
good teacher and not just a tutor handing you the answers,
right?
Just a way to support. So it hasn't been put in action with the students yet.
It will soon.
But what has been some of the feedback um
as you've developed it and watched people use it.
Sure.
So we actually have been beta testing it,
so to speak with our summer school students at Harvard summer school.
So we have about 70 students here on campus and off campus using it thus far.
And it's been pretty darn good so far.
Our goal has been to have this bot answer as many as 80% of
students questions and I dare say it's probably approaching 90 or so percent.
It's proving very good.
When it comes to curricular content, questions about computer science,
about programming.
Some of the early bumps in the road that we ran
into were that it was less good at answering administrative questions,
questions about deadlines or policies because those
things tend to change semester by semester.
So insofar as these large language models have
really been trained primarily on historical data.
We've been uh improving it over the past few weeks to know
a little something more about present day policies and dates as well.
What about the risk of getting hacked and what are some
safeguards that you all put in place to prevent that?
Yeah,
so this is uh integrated into our own software
infrastructure that students already use for writing their code.
Um The bot itself is proxy, so to speak through one of our own servers,
so that we have an ability to keep an eye on things centrally in that way.
And we also have an ability to analyze the
quality of the responses that are coming through.
And indeed,
we pretty much have been doing this every day such that if
we see the bot being too helpful or even too inaccurate,
we can then put in some safeguards and measures for the next day onward,
that will guide it toward giving better solutions.
But it's very much access restricted to only being able to take input from uh a
few different modes of uh technology and just
a couple of seconds here left professor,
but um this is being used for coding.
Could it be used in other classes at Harvard as well?
Absolutely.
And that's our hope actually for the fall semester in the
spring is to collaborate with some colleagues in the humanities,
the arts, social sciences, physical sciences.
It really is not something that's restricted to computer
science that just happens to be our own domain.
All right. Well, we'll check back and see how it's going here in a couple of months.
I appreciate your time, Professor David Malan.
Thank you for having me. Thank you.