DAVID J. MALAN: Hello, world.
This is CS50, and this is an introduction
to how CS50 has been teaching using Artificial Intelligence, or AI,
over the past several months.
My name is David Malan, and I teach this here course called
CS50, which is Harvard's introduction to the intellectual enterprises of computer
science and the art of programming.
If you would like to follow along with these same slides
that you'll see over my shoulder, please feel
free to take a photo or screenshot or pause
at this point to use this QR code, which will lead you
to a copy of today's slides.
Now, CS50 is among Harvard's largest classes here on campus.
We have some 600 undergraduates over the course of the fall, the spring,
and the summer semester nowadays.
We also have some 200 students each year through Harvard's Extension School,
which is our continuing education program.
And for the past nine years, we've been collaborating with our friends
down the road at Yale University in New Haven, Connecticut,
where we have some 250 students taking the course as well.
They primarily watch the course's lectures online,
but with our own teaching assistants in New Haven and faculty
do they have sections or recitations, office hours, CS50 events, and more.
For the past several years, we've also offered variations of the class
through Harvard's Business School and Law School, CS50 for MBAs, so to speak,
CS50 for lawyers, so to speak, which are focused more specifically
on those particular demographics.
And then, of course, as many of you know, the course since 2007
has been freely available as OpenCourseWare,
which is to say anyone is free to take or to teach the class, even
using the freely available materials as well as technology.
And indeed through platforms like YouTube
do we have some 1.8 million subscribers.
Through platforms like EdX do we have some 5.7 million registrants now
to date.
Now, CS50 itself, as many of you know, is
quite substantive in terms of how much content
and how much technology it covers.
We begin the semester with a language called Scratch, which
is a very friendly graphical language.
We transition thereafter to roughly half the class
in C, a more traditional text-based language.
And then in the latter half of the course
do we transition to languages like Python and SQL
and JavaScript in the context of HTML and CSS so that at the end of the course
students are not only prepared for further studies in computer science
foundationally, but also, if they never again take another computer science
course, we hope that they're well equipped
to go back to their own fields of interest,
be it in the arts, humanities, social sciences, natural sciences,
physical sciences, or beyond, to apply lessons learned from computer science
and programming to problems of interest in their own domain.
But with this many students and with this many teachers,
suffice it to say it's been a challenge and a goal to provide as much
of a support structure as we can, not only for CS50 itself, which many of you
know by its OpenCourseWare name, CS50x, but over the past few years
has CS50 really evolved into a whole ecosystem of courses,
an entire curriculum, particularly focused on introductions
to various technologies and tools.
But here are some of our screenshots of some of CS50's most recent courses,
all of which, too, are freely available.
In fact, if you're new to the community, feel free to go to edx.org/cs50,
where all those courses and eventually more await.
The course we'll focus most of our attention on today in this talk
is CS50 itself, otherwise known as CS50x, which is available at this URL
here.
And for students and teachers in K-12, so to speak,
middle schools, high schools, and so forth
do we also offer a version of the class called CS50 AP,
which high school and middle school teachers are welcome to adopt or adapt
whether or not participating in an advanced placement
program for their own classes and students around the world.
In fact, here's one of our favorite photos from a few years
ago in New York City, where we brought together some three public and two
private schools to have a CS50 AP hackathon, where
students worked on their homework assignments or final projects.
Pictured here is a visitation we had just a few weeks ago in Jakarta,
Indonesia, where we worked with nearly 300 middle school and high school
teachers there who spent the past six months taking CS50,
perhaps like you, online, then to culminate in a professional development
workshop where CS50's team went to Jakarta to meet all of these teachers,
and in the coming weeks and months will they return to their own classrooms
to teach CS50 in some form itself.
And pictured here, lastly, are some of our students in Nicaragua holding
proudly some of their CS50 certificates and Harvard pennants, which
is a community too that we've gotten to know all too well additionally.
So indeed, if you'd like to join those and more communities,
please feel free to reach out after today
at any time via outreach@cs50.harvard.edu.
So the premise for today's talk and really
the motivation behind a lot of CS50's work over the past year
has been this premise that ChatGPT and tools like it,
Bing Chat, GitHub Copilot, and the like, are just too helpful.
They are all too willing and all too deliberately designed
to try to answer your questions outright.
Now, in terms of the real world and in terms of solving real-world problems
and producing code, for instance, in industry, that's a great thing.
I think we've seen evidence already between tools
like GitHub Copilot and Bing and ChatGPT that there is potentially and invariably
is going to be even more of a productivity boost.
It will save humans time, and it will amplify the impact of individual humans,
much more so, for instance, than replacing them outright.
It will amplify one's impact.
But in the context of education, as most of you might know or remember,
it might be nice to have someone hand you the answers to all of the questions
you have on some homework assignment, exam, or the like,
but that's certainly contrary to the whole point of learning
in the first place.
And so our premise and presupposition is that tools like ChatGPT out of the box,
because they're either on or off, there's no settings
really for these tools right now, are all too willing
to just answer any and all of your questions
instead of, like a good teacher or tutor, leading you to an answer.
Rather, they tend to just hand it to you outright.
And so what CS50's team set out to do over the past year
was try to implement our own version of ChatGPT, if you will,
building on top of the shoulders of these other giants, OpenAI, Microsoft,
Google, and others, but imposing some pedagogical guardrails, so to speak,
ironically trying to make these tools less helpful,
putting a little bit of downward pressure
on their default behavior of handing you all of the answers, all of the code
that you might ask for.
What we instead wanted to really do is implement
more of a tutor, a good teacher.
And so, at least with our on-campus students initially
and now all of our online students as well, in CS50 syllabi,
it is not reasonable, that is, it is not allowed
to use right now AI-based software like ChatGPT, GitHub Copilot, Bing
Chat, or the like that suggests or completes answers
to questions or lines of code.
So through policy, we simply disallow usage thereof.
Technologically, there's nothing stopping,
of course, students and teachers alike from using these tools.
So we've tried to walk this line sort of ethically and educationally
within the class to message where these lines are
and which lines should not be crossed.
But rather than just take away this very nascent technology, which is undoubtedly
going to be useful and probably with us here on out,
we do deem it reasonable by contrast for CS50 students
to use CS50's own AI-based software, including the so-called CS50
Duck or Duck Debugger, DDB, which is a riff on GDB,
the GNU Debugger for short, which is now built into two of CS50's tools
called CS50.ai and CS50.dev, which is to say students
are asked not to use off-the-shelf tools like ChatGPT, GitHub Copilot, Bing
Chat, and the like.
But they may and are encouraged to use CS50's own tools that ideally do have
those pedagogical guardrails in place.
So how do we go about implementing this so-called CS50 Duck and why?
Well, in computing circles and in the world of programming, as some of you
know, it's been a thing for quite some time
to use something called rubber duck debugging, or rubberducking for short,
whereby, if you don't have a colleague, a teacher, a friend, a family member who
knows more about programming than you, you should at least keep a rubber
duck on your desk near your laptop or desktop or phone
so that when you do have a question or confusion
or you have some bug or mistake in your code,
you can at least hold up this rubber duck and talk to it,
walking through verbally whatever confusion or problem you're having.
And even though in the real world, the duck really
shouldn't be quacking back at all, squeaking maybe, but not quacking back,
at least in that process of airing your confusion verbally, for so many of us,
myself included, that proverbial light bulb eventually goes off atop your head,
and you realize, oh, that's what I'm doing wrong,
because you realize in verbalizing the problem where you've perhaps
gone astray.
So years ago, we in CS50 actually implemented a virtual version
of this rubber duck because even though we hand the ducks out here on campus
to students residentially, we have so many more students and teachers online.
So we implemented rubber duck debugging or rubberducking
in the context of not only the real world--
and pictured here is an unnecessarily large eight-foot duck that is often
beside me on stage, in addition to the tiny little rubber
ducks that we do give out-- but we also implemented a virtual version thereof.
Within CS50, for the past few years, we've
used a free open-source tool called Visual Studio Code for Microsoft,
very popular in industry and also usable in educational contexts.
And we implemented what's called an extension, like a plug-in using
some JavaScript code that CS50 wrote to create a virtual chatbot that
takes on the same personality of a rubber duck.
And so, for instance, for the past few years,
if a student were to ask a question like this textually at their keyboard,
"I'm hoping you can help me solve some problem," well, up until recently,
as many of you might recall, all this virtual rubber duck would do would be
to respond with one, two, or three quacks.
And that's it.
And the extent of the complexity here was
that we had a little bit of randomness involved
so that we would programmatically generate one, two, or three quacks.
Now, this is not a scientific measure, but we
have anecdotal evidence that just having the duck quack back
at you textually actually helped solve a nonzero number of problems in the world.
And a nonzero number of students were appreciative that just
by going through the process of typing out their confusion, even
though the duck was only going to quack back at them,
they realized through that process of verbalizing or textualizing
their thoughts where they had gone astray.
And honestly, this actually resonates with me.
On occasion, I've posted on websites like Stack Overflow,
or Stack Exchange more generally, which has
been a very popular tool for questions and answers online.
And I have probably not posted on Stack Overflow
far more frequently than I have posted on Stack Overflow
because I'm so worried about looking like a dummy to everyone on the internet
by asking a question that maybe I shouldn't in retrospect that I carefully
write down all of my thoughts, I try to express
what I have tried, what is not working, what the symptoms are.
And so darn often does that proverbial light bulb go off for me as well.
So if you, too, have ever sat down to write a post or maybe an email or a text
message but not sent it because you realized, wait a minute,
I don't need to, that then is the beauty of something like a rubber duck.
Now, some of our students online just a few months ago
were sort of shocked to discover, literally overnight,
that when they began to ask their questions in English
or some other human language, literally overnight,
a few months ago did the CS50 Duck start responding
to them in English and, in some cases, other human languages as well.
And this is all thanks to the work of CS50's team
over the past several months, and certainly the work
over the past many years of the OpenAIs, Microsofts, Googles,
GitHubs of the world, to make this kind of chatbot, so to speak, now possible.
And so thanks to one of our teaching fellow's daughters in New Zealand
has the CS50 Duck really been brought to life here, not only graphically
but textually as well.
And so pedagogically what our goals have been over these past few months
and moving forward is really to provide with AI students with virtual office
hours 24/7, so to speak.
Office hours, for those unfamiliar, at least in college campuses,
is an opportunity for a student to go meet with a professor or a teaching
assistant or a TA one-on-one or in small groups
to just ask questions about the week's material or homework assignments
or the like.
24/7, of course, is referring to the number of hours in the day and days
in the week.
And the goal then through AI for us has been to provide really
an approximation of a teacher being available to students
throughout the day, throughout the week, to ideally
help get them over certain hurdles.
And we're fortunate, frankly, in places like Harvard and Yale
and college campuses and university campuses more generally,
to have a lot of human resources and lots of teaching assistants,
in some cases, lots of humans.
But so many places around the world and so many students around the world
don't have access to those same resources potentially.
And so the goal here, too, has really been
to try to uplift as much of CS50's community as possible and ideally level
the playing field at least in terms of the support that is available to someone
whether it's on campus or now very much off.
And here is sort of the holy grail for us, so to speak,
whereby the goal technologically and pedagogically for us
has been to ideally approximate a one-to-one teacher-to-student ratio.
Because even in places like here on campus,
we might have 10, 20, 30 students assigned
to one teaching assistant or TA.
We have weekly sections or recitations where those relatively smaller groups
of students get together, not just for CS50,
but to discuss other courses as well.
But if you start to do the math and you think about an office hour, 60 minutes,
if you have only six students attending an office hour, be it the professor
or with the TA, that's only 10 minutes per student among those six.
And for so many students for whom CS, programming, even STEM studies, Science,
Technology, Engineering, and Math, are new,
that's just not very much time at all.
And so being able with software to approximate the idea of one of me
for every of you, one TA for every of you,
is a very exciting thing educationally.
And even though admittedly there's a lot of potential downsides
on the horizon for AI, the applications we are so excited about
are indeed those within education.
All of this work that you're about to see
would not have been possible indeed without our friends
at OpenAI, Microsoft, GitHub, and beyond,
and particularly CS50's own team here in Cambridge and beyond,
including Rongxin, Andrew, Patrick, Charlie, Carter, and more,
over these past several months in particular.
So allow me to share on the whole team's behalf what it is we've been up to.
So underneath the hood is this website, really this web service, called CS50.ai.
It's a domain name unto itself, but it describes really this architecture here
whereby over the past several months we've been building out this technology
stack, if you will.
Now, at the end of the day, a lot of this stack is built on top of services
like Microsoft Azure, which is their cloud service, or OpenAI Zone.
Indeed, companies like these, as some of you know,
offer generally what are called APIs, Application Programming Interfaces,
which is like a tool that you can sign up for, sometimes for free,
sometimes for money, to actually use that company's technology or services
or data in your own software.
So really, OpenAI, Microsoft, and others have done a lot of the hard work here,
and we have done our best to layer on top of their work a pedagogical layer,
if you will, including those guardrails.
Architecturally, then, our system looks a bit like this.
But for today, we'll focus really on the human component, the user
interfaces that have really been now presented to students and teachers
alike.
So here for the unfamiliar is a link to CS50's programming environment nowadays.
It's a web-based integrated development environment, or really a text editor
with lots of extensions built in, called Visual Studio Code.
It's built on top of something called GitHub Codespaces, which
is a cloud version of what are called Docker containers, which
is to say when a CS50 student or teacher logs into a website, CS50.dev,
we use GitHub's APIs to automatically create
for you something called a repository, something called a codespace, a.k.a.
container, in the cloud on GitHub's infrastructure so that
what you see in your browser is a real-world programming environment.
But what you don't have to do is install any software yet on your own Mac or PC.
It just works in terms of day one.
At the end of the semester, what's nice about
VS Code nowadays is that you can, if you want, install it on your own Mac or PC.
It sometimes requires a bit of technical difficulty
and solving thereof but much better to do that,
we think, at the end of the course than, for instance,
on day one at the beginning.
So once you visit this website, you see a landing page not unlike this.
And built into this same tool is this CS50 Duck.
And what we set out to do some months ago was try to put our toes in the water
with Artificial Intelligence, or AI, trying to think about, all right, what
would be a helpful but also relatively easy feature
to implement using AI by writing software of our own
on top of OpenAI's APIs?
And again, OpenAI is the company behind today's ChatGPT.
And so the first thing we did to put our toes in the water, so to speak, was,
could we write code to explain lines of code to students?
So whether they've written some code or they've downloaded some code
or copied some code from class, could we explain it to them line by line
as any good human could if they were to raise their hand or ask someone online?
So we did this.
Here is a screenshot of VS Code.
Here is a screenshot of some code written in that language called C.
And this code relatively simply prompts the human for their name
and then prints out hello so-and-so, based on what they've typed in.
For a student new to programming and certainly C,
there's a lot of non-obviousness going on here.
There's a lot of complexity and a lot of syntax.
But what students can now do, as of the introduction of AI to CS50,
they can highlight one or more lines of code.
They can right-click or control-click.
They can then select this option here, Explain Highlighted
Code, which doesn't come with VS Code.
In fact, Rongxin wrote an extension to contribute this menu option to this here
menu.
And as soon as the student clicks that, within three seconds or so
do they have a ChatGPT-like explanation of exactly those lines of code.
Now, to be fair, this example itself is not very complicated.
A human, a teacher could certainly answer this same question
and explain these eight lines of code line by line.
But that would take a few seconds, a few minutes.
They might not be awake at the hour the student has this same question.
And so the power here is that all of this
was automated and customized for this particular context or code.
So that was actually relatively straightforward.
And so the next feature we set out to do was a little more sophisticated.
Could we advise students with AI on how to improve their code style,
the aesthetics, the formatting thereof?
Within CS50, because we're an introductory course,
we have actually consciously disabled a lot
of features that normally come with IDEs, or Integrated Development
Environments, or text editors like VS Code.
So we disable autocomplete.
We disable autoformatting, things that in the real world
and once you've taken one or more courses, yes, totally reasonable
to turn on.
But in CS50 as an intro class, we really want
students to develop some muscle memory and actually understand
why their code should look this way, how to make their code look this way.
And then, once that becomes boring and uninteresting,
then they can automate the process, for instance, later in the course or after.
So when it comes to style, we implemented a new-and-improved version
of a command line tool that many of you might know as style50,
but we made it more graphical.
And so pictured here is another screenshot of VS Code.
Here is some more C code at top left.
And for today's purposes, I'll stipulate it's pretty messy.
It's all left-aligned, and it's not very pretty printed or well formatted.
So what students now can do, in the top-right corner of VS Code,
is they can click a button called style50.
And what this is going to show them now on left to right
is on left, what their code currently looks like, to right, what
their code should look like.
And this built-in diff editor, so to speak,
uses some red and green color coding to make clear what you should remove
or what you should add.
But even this, frankly, to new programmers might be nonobvious.
What am I supposed to do?
Why?
And so also at top right now, there's this Explain Changes button.
And if students click that, they similarly
get a ChatGPT-like explanation of how and/or
why to make their code look from left to right as we've proposed.
So again, something that any teacher who's available or awake could do,
but here again is this approximation of a one-to-one teacher/student ratio
available 24/7.
But the next feature that we set out to implement, which is now
generalizable away from CS50, away from programming code,
and really I do think representative educationally
of what teachers and administrators in schools
will be able to do in the arts, humanities,
social sciences, physical sciences, and beyond, not just in CS soon,
is can we answer at least most of the questions
that students are asking, for instance, online?
Now, for many years, CS50 has used various tools and various social media
to enable students to ask questions and get answers,
either from classmates or from teaching assistants or from myself,
throughout the day.
Of course, we are not always available.
Of course, not all of those answers from classmates,
who themselves are only learning the material, are necessarily correct.
And so there's really been an opportunity
here to try to answer all the more at scale and all the more
correctly those students' questions online, both on campus and off.
So what we set out to do here was implement an existing third-party tool
called Ed, which some of you might have used, and we use it very heavily here
on campus.
It's a question-and-answer tool that allows
you to post questions asynchronously.
You ask now, and some number of seconds, minutes, hours, days later, hopefully
a TA or a professor will reply.
So here, for instance, is a screenshot of a question
that a CS50 student might ask.
And we've anonymized them here as John Harvard,
but it's representative of a student question.
And this question is kind of a softball.
It's kind of an easy question because it's very definitional.
Quote, unquote, "What is flask exactly?"
I mean, honestly, Google could answer this, Bing could answer this.
So this itself was not hard, but it was our first attempt
to see how well AI could do.
Here now is a screenshot of how the duck actually
responded to that specific question.
So Flask is a micro web framework written in Python.
It is classified as a microframework because dot dot dot.
And let me stipulate, especially if you're new to CS50,
this is a topic we introduce at the end of the class itself.
This is a pretty darn good answer.
Indeed, this is something that I as a human would ideally have written myself.
But of course, the AI, the duck, was able to implement--
was able to answer this question within seconds itself.
And now for the technically curious, what's really happening
is when a student asks a question via this tool, called Ed,
and they hit submit to post the question to the website, what
we have done over the past few months as CS50 is we wrote some code,
also coincidentally in a language called JavaScript,
that intercepts the student's question, adds some formatting
and some additional phrasing to it.
We then send it to our own server via HTTP,
and that server is CS50.ai, as mentioned before.
We then relay that question in some form to OpenAI's server or Microsoft's
servers to actually use their underlying large-language model, the technology
that underlies tools like ChatGPT.
CS50.ai quickly gets a response from OpenAI or Microsoft.
We might validate the response or make some slight tweaks to it.
Then CS50.ai passes the response back to the Ed tool,
which itself is a different website.
And all that happens within three seconds.
And voila, the student gets an answer from the so-called CS50 Duck
here and now.
Now, many of you have heard, of course, that AI is fallible.
It's not always correct.
I do think this problem will go further and further away
over time as the technology gets better.
But what we do do right now is provide ourselves
with opportunities to at least remind students
that this technology is imperfect.
So for instance, here's another, more sophisticated,
question about the Caesar problem set with which some of you
might be familiar.
Students have to write code in C to implement
what's called a rotational cipher to encrypt or decrypt information.
In this case, though, I'll highlight it, the student
has asked a more nuanced question, is there a more efficient way
to write this code?
So this isn't quite as simple as answering a definitional question like,
"What is Flask exactly?"
This is now more nuanced, where the AI has
to understand the context of the problem and the code
that the student has written, including this here error message.
And so in this case, here's a screenshot from the same CS50
Duck in this same tool called Ed, which, for today's purposes, I'll stipulate
is a pretty darn good teacher-like response.
Not only does the duck acknowledge what it thinks the student is trying to do.
The duck also provides a few lines of code,
but really just starter code, nothing that spoils the answer to the question
outright.
But the duck also, note, reminds the student at the bottom
here that this is very much experimental.
I mean, even our software, as you might have seen at CS50.ai itself,
is still in beta, which is to say still being developed.
And we remind the student, "Quack.
Do not assume that my reply is accurate unless that you see it's been
'endorsed' by human staff.
Quack."
Now, where is that referring to?
Well, it turns out that this particular tool
has a graphical button that humans can click to endorse other people's answers.
Until recently, this was meant to be used so that a teacher or a TA
could endorse a student's response to another student,
to just validate that, yes, the staff approve of this answer.
It is good and correct.
What we have done is co-opt this same button
to now signal to students that if they see that a duck's reply has been
endorsed per this icon at top right by a human,
then they should trust that it indeed is valid.
So we might not endorse it within three seconds
but usually within minutes or maximally hours, where we're doing the same.
Frankly, I think this is a short-term mechanism to avoid what are generally
called "hallucinations," where AI just sometimes makes things up by chance.
But I think this is a problem that it too
will start to go away and away as the technology only gets better.
But for now, this is how we are at least partly mitigating this.
So this then is the URL, CS50.ai, that anyone
in the world with a free github.com account can access.
In fact, you're welcome to try it now or later.
But at this URL lives not only that architecture,
that service that I referred to earlier that's
talking to the graphical user interface and talking to OpenAI or Microsoft.
Also, there is a full-fledged chat interface.
So CS50.ai itself has its own front end, its own graphical user interface,
that is very, very similar to ChatGPT by design
so that students no longer need to even post asynchronously their questions
and wait some number of seconds for the duck or a human to reply.
They can have full-fledged conversations with the duck,
much like you and I can have the same with ChatGPT, Bing Chat, or the like.
This chatbot, though, as before, reminds the students from the
get go that their response should be--
its response should be taken with a grain of salt.
This here response reminds students to always
think critically so that for at least for now, they're
at least not in the habit of just assuming as outright facts something
a computer is telling them.
Here, though, is a question that will look
even more familiar to those of you who have some programming
experience, especially with Python.
And it's representative of not only an interesting code question but also
representative of the amount of detail or, dare I say,
lack thereof that students often include when asking questions of us
humans or these ducks.
So here is a bit of Python code that is trying to prompt the user for two
integers, x and y.
It is then trying to calculate the sum of x plus y,
but the problem is not obvious from the student's post here because all
the student has asked, notice, is "My code is not working as expected,
any ideas?"
So here's an example of a question, and it's barely
that, that really the AI needs to understand from context,
almost understand the code, it would seem, or at least recognize
it as familiar to some other code the AI has been trained on,
so to speak, thanks to lots and lots of input from the internet and beyond.
And in this case, I'll tell you the symptom would be this.
If the human typed in 1 for x and 2 for y,
you would hope that this code would print out
1 plus 2 equals 3 as the answer.
Unfortunately, this code, as some of you might be realizing,
actually prints out 12, that is, what looks like 12.
And that's because, as the duck notices, it
seems that you're trying to add two integers,
but the input function in Python returns a string,
that is to say text, not an actual number
that you can perform mathematics on.
And so when you try to add x and y, you're
actually trying to concatenate the two strings not add two integers.
So the duck here provides students with just a couple of lines of guidance,
but indeed lines that include Python's int function, which
will indeed convert a string that looks like a number to an actual number.
And so what we've seen now behaviorally among students
is that most students are now interacting
with the duck via this here synchronous chat conversational interface,
some of them a little too much.
And in fact, we adopted midway through the past few months
this here heart system, a sort of HP system or energy
system, whereby you can only ask so many questions now per unit of time.
And this is a knob we can turn to allow for more or fewer questions.
But this is meant to really kind of chop off
the tail end of excessive use educationally we think of the duck.
That is to say, we've seen some students online ask hundreds of questions
per day, which, honestly, I don't know where exactly the line is pedagogically
between too few and too many questions.
But in the real world, if a student were to ask a teacher hundreds of questions
per day, it feels like that's the time, if you think back to high school
or middle school, a good teacher would probably say, David,
why don't you go back to your desk and think about this a little bit?
And so what we try to do with these hearts
is prevent students from asking too many questions too quickly at once
to virtually send them back to give it some thought.
And frankly, nowadays, too, this architecture is not free.
APIs generally cost money.
And even though it's free to students and teachers, our friends at Microsoft
and OpenAI and GitHub and others are kindly
covering through educational grants use of this system,
we also just try to minimize the operational costs
with this same mechanism as well.
So to speak to the scale of this whole system, if curious, as of today,
we're up to some 115,000 students and teachers who have used the duck over
the past few months alone.
That's roughly 20,000 prompts or questions really being asked per day
for a total of roughly 4 million questions so far.
So even if you've just asked a few questions,
you are in very good company among all of these others.
We thought that we'd share now a little bit of the technical insight of how
all of this works because, actually, CS50's architecture is now
retrospectively pretty representative of what other people in the world
are now doing as well.
Underlying a lot of today's uses of AI is a technical term
known as a system prompt.
That is to say, companies like OpenAI, Microsoft, Google,
and others have made available to the world Large Language Models, or LLMs,
which have been trained on massive amounts of input, English text, code,
text in other human languages, to try to recognize patterns
so that when it is asked a question, it knows with high probability
how to respond.
The AI doesn't necessarily understand the question
but at least recognizes it in some form.
And using tools like--
using tools like neural networks and other components of machine learning,
so to speak nowadays, AI is trying through these large language models
to finish our thoughts for us or answer questions more concretely.
But you can personalize them.
You can customize their behavior by giving them what's
called a system prompt, for instance.
So out of the box, these large language models
sort of speak English and some other human languages.
They speak Python and some other programming languages.
But you can give these AIs a personality or a scope of reference.
So for instance, and this is abbreviated,
here is what CS50's system prompt maybe in day one
looked like, though now it's much, much longer and more detailed.
We tell the AI, you are a friendly and supportive teaching assistant for CS50.
You are also a rubber duck.
And that phrase alone is sufficient instruction
to get the generic AI to start quacking really and behaving like a rubber duck.
We tell the duck further, answer student questions only about CS50
and the field of computer science.
Do not answer questions about unrelated topics.
Do not provide full answers to problem sets,
as this would violate academic honesty.
And so we're effectively "programming" the AI, if you will, through English.
And so this is what has become known as prompt engineering, which
is trying to come up with the English or the human language description of how
you want the AI to behave.
I do think this is a technique that's going to evolve quite quickly over time
as these things get more featureful.
But for now, you use English or your own human language
to tell the AI how to behave.
And in our case, after this so-called system prompt,
we tell it, answer this question.
And if you, the student or teacher, ask this thing a question,
we essentially copy/paste your question at the bottom of this system prompt,
and your question is what the world would generally call a user prompt.
That's what's coming from you.
The system prompt is what's coming from us.
Now, just a few weeks ago in the US and a lot of other countries
was April Fool's Day, whereby lots of people tried to make funny jokes.
And thanks to my colleague Rongxin did we
modify the system prompt of the rubber duck for just over 24 hours,
as some of you might have seen, to behave a little differently.
And so what Rongxin kindly did was edit our system prompt and still start with
"you're a friendly and supportive teaching assistant for CS50."
But he then added this text on April 1.
You are also a rubber duck in Rick Astley's band.
Importantly, you should always cheer up the student at the end by incorporating
"Never Gonna Give You Up" in your response.
Now, for the unfamiliar, Rick Astley and this particular song
have become known as a meme on the internet.
And the goal is to trick people into watching a bit of its music video.
So Rongxin kindly integrated this language into the duck's programming,
if you will, so much so that if you, the student or teacher,
were to ask a question that day with this system prompt in place, like,
"What is recursion?" you would get wonderfully an answer that we did not
hardcode-- this was dynamically generated by the AI using just that
system prompt alone--
an answer like this.
"It's a powerful tool.
And remember, I'm never going to give you up,
so keep practicing and answering questions,"
an allusion to the lyrics in that same song.
So what are the results now pedagogically, operationally
that we've seen in just the past few months
alone because we really are just beginning?
Indeed, this is very much a beta, very much experimental,
but already impactful, we dare say, in the real world.
So in terms of usage, at least among our own undergraduates,
whom we survey weekly throughout the course with questions,
we know that 17% of the students in blue were using the tools, CS50.ai
and the duck inside of CS50.dev, more than 10 times per week.
32%, in green, we're using them 5 to 10 times per week.
26% of them were using them two to five times per week
and so forth, which is to say a supermajority of our undergraduates
and daresay now our online students really dove into this use of AI
quite quickly as well.
In terms of the helpfulness, this is now anecdotal and measured only
based on students' own responses, not a more rigid measure,
but 47% of students in blue this past fall
found the duck-based tools very helpful.
26% more, in green, found them helpful.
21% found them somewhat helpful and so forth.
So again, not 100% across the board, but a supermajority of students
were already finding these tools--
in what we'd call like version 0.9; I mean, it's a beta;
it's not necessarily done or finished-- were already finding them useful.
Now, more quantitatively, we looked last summer,
and this is work we'll continue in the coming months-- we looked
at a relatively small sample set of our first questions
and smaller version of CS50 of students over the summer last year.
And when we analyzed the questions being asked
by students via that asynchronous tool called
Ed and we analyzed with human eyes the quality of the duck's response,
we came up with these measures that in the very first version of the duck
last summer, the duck answered 88% of curricular questions,
content questions, correctly and 77% of administrative questions correctly.
So not as good administratively, but by administrative,
we mean policy questions, deadlines, things like that,
that frankly change every semester or every year.
So it stands to reason that the duck was just outdated because especially,
at the time, OpenAI's data set had been trained only as far as 2021,
not 2023 data.
But even 88% already out of the gate was incredibly strong.
Curiously though, and to share a more scientific finding here,
in fall of 2023, when we had even more students taking the class on campus,
it appeared at first glance that the duck's answers were only
correct 39% of the time, a marked decrease.
But this just didn't line up with reality as we perceive students' usage.
There was no actual evidence to suggest really
that the duck was erring nearly 60% of the time
but rather that students' usage of the duck was changing.
These numbers here were entirely based on that asynchronous tool
called Ed that, again, posted a response asynchronously for the duck or a human--
from the duck or a human.
But what we found was that students' behavior was very quickly transitioning
to CS50.ai itself or a synchronous conversational version thereof built
into CS50.dev, that is to say VS Code.
So in fact, we noticed this as follows.
When we looked at the data in ED, this Q&A
tool that we use both on campus and off, among our on-campus students
over a year ago, in fall of 2022, students
were asking, across 500 or so students in total, roughly 0.89 questions
or one question per student online during the semester-- not
very high per student, but with 500 students it certainly added up.
The next summer, summer of 2023 that we looked closely at,
they were asking roughly the same, 1.1 questions per student.
But this past fall, when we deployed the CS50 Duck in this conversational form
as well as in VS Code, the number of questions students
were asking on campus asynchronously via that Ed tool
for questions and answers dropped by 75% to just 0.28 questions per student.
And so our working hypothesis is that students
were asking most of their questions now conversationally, synchronously,
so to speak, via the duck's own UI.
And far fewer questions were being sent to Ed.
Indeed, the questions that were going to Ed, we do think,
were those more difficult questions or questions
that maybe the duck did err on that were being escalated,
so to speak, to Ed because humans were keeping an eye, per the Endorse button
and so forth, the Ed tool, but we were not
keeping as close of an eye on the conversational bot.
So we think the types of questions being asked by students
were indeed becoming much more administrative, much more particular
that the duck itself wasn't answering conversationally.
We looked to what other impacts.
On campus, we're fortunate to have lots of humans and lots of human support
both at Harvard and Yale.
And so through office hours numbers, we also saw a difference.
In fall of 2020 and in fall of 2022, when we had roughly 500 students or so
per semester, we had some 50% of students attending in-person office
hours, signing up by appointment for small-group questions and answers
with our teaching assistants.
In fall of 2023, that dropped by 40-some percent to just 30% attendance.
That is to say, fewer students were taking advantage, funny enough,
of those same human resources.
We've omitted fall of 2021 because that was impacted particularly by COVID.
But otherwise, we think that this has been a marked change this year
with the duck live and in students hands vis-a-vis prior years instead.
And some of our favorite anecdotal evidence that we're now presenting,
that we've presented recently in a paper that we'll link to ultimately hereafter,
is some of students' own comments about the duck
when surveyed on campus at term's end.
One student noted that it "felt like having a personal tutor.
I love how AI bots will answer the questions without ego
and without judgment, generally entertaining
even the stupidest of questions without treating them like they're stupid.
It has an, as one could expect, an inhuman level of patience."
And that recognition by a student really resonates with me
because I think back even to my own college days, graduate school days,
when not infrequently I would go into a professor's office hours one on one.
I would ask my questions, several of them sometimes.
But I feel like more often than not, I would leave the professor's office hours
nodding my head, saying thank you, yes, that cleared everything up,
but thinking to myself, that did not clear every thing up.
I'm still confused, but I kind of overstayed my social welcome.
Not because the professor didn't want me there,
not because they ushered me out the door,
but because I felt that there was some limit where I was kind of thinking I'm
the dummy.
Imposter syndrome is a thing.
Maybe I didn't really feel like I belonged there.
And so the fact now that we have technology via which students can still
access humans in many cases but can alternatively
have a much longer, much more paced conversation with an AI
that's pretty akin to a teaching assistant is pretty enabling,
I think, for clearing up all of our confusion,
so much so that even before CS50's lectures nowadays,
be it on campus or on camera, even I find myself using tools like this
or ChatGPT more generally to go down intellectual rabbit holes,
ask questions of the AI that I could google or I could look up on Bing.
But AI'S getting pretty darn good at just hitting the nail on the head,
so to speak, giving me the answer I want, therefore
amplifying my productivity, my efficiency, so I can then
keep my head in the space of preparing for class in that way.
Another student this past fall wrote, "The AI tools gave me enough hints
to try on my own and also help me decipher errors and possible errors I
might encounter."
And a third student wrote, "I also appreciated that CS50 implemented its
own version of AI because I think just directly using something like ChatGPT
would have definitely detracted from learning."
And that, too, was our whole working premise
and indeed a lot of the motivation behind building CS50's own duck on top
of these open platforms.
Now, what future work and future impacts do we think lies ahead?
Well, I think one on grades, both on campus and off.
For instance, those of you who have taken CS50
are generally familiar with how we grade our programming assignments, at least,
on three axes, correctness, design, and style.
And for many years now, since 2012, have we automated correctness
grading by way of a tool at the command line called check50,
which runs what are effectively unit tests or functional
tests of a student's code.
So we automated feedback on correctness years ago.
Similarly, years ago, we automated feedback on style
through the command line version of style50
and more recently now the graphical version.
And so we've already seen over the past 10-plus years that students' grades have
been going up and up and up, certainly on campus as well as online,
and this is independent of the sort of grade inflation you hear or read about
in higher education more generally.
This is a direct side effect of providing students
throughout the week with iterative, if not immediate, feedback.
And so through AI what we do suspect is about to happen
and is probably already happening though we've not yet measured
it is that the quality of the design, third and last most, of students' code
will probably begin to increase all the more before they officially
submit as well.
Because if and when we use this AI to provide students
with a design50 tool, which they effectively already have
because you could just copy/paste your code into the CS50 Duck and ask it,
like one student, How could I improve this code
and get iterative feedback throughout the week?
it stands to reason logically that if the AI or really a teacher in general
is giving you iterative, incremental feedback back hour by hour or day
by day, that hopefully your code, or your homework more generally,
will be pretty darn good, if not close to 100%, by the time
you actually submit.
So if we then think ahead a few months, a few years, if everyone's getting 100%,
like what does that really mean for assessment?
Is there still an opportunity to help students distinguish themselves
from some baseline measure of where they should be at a certain point?
Is there any way we can distinguish just how
strong or weak a student is in some topic after some number of months?
Well, where we this is going is going to be an opportunity
to apply AI in other ways.
We have already CS50 here on campus use CS50's Duck, reprogrammed it
with a different system prompt to try to train our teachers how to answer
students' questions in person.
We get the duck to behave, for instance, like a student who
has some confusion in office hours.
And they do role play with the human TAs.
I think moving forward what we'll likely do with students, meanwhile,
is program the duck with really a different system
prompt to have a conversation with students at the end
or maybe even weekly during the course, akin to yesteryear's oral examinations,
which if you've never experienced, might be one or more faculty sitting
down, very stressfully, with a student in front of them, asking them questions
about their field of study, about their thesis or dissertation, or some topic
more generally.
That's hard to do with humans because you really
do need a one-to-one ratio, if not a multiteacher-to-one student ratio
to achieve that readily.
But through AI, perhaps we can start asking our students here, all of you,
or your teachers, you, questions about the material being learned,
be it CS50 or anything else, have you type your answers as quickly as you can,
and then maybe in version 1 of this vision,
we use humans to evaluate the quality of the conversation
you just had to see how good your answers seem to have been.
We can then sort of ignore questions that we think, ah, the duck or the AI
shouldn't have asked that.
It wasn't a good question.
And frankly, in version 2 of that vision,
I would conjecture is going to be just having the AI evaluate
the quality of the conversation for you automatically and for us
so that you get this feedback loop, ideally week to week during the term
and probably at the end of the course as well,
that really gives you a better sense of, one,
where there are holes in your understanding, what you should focus
your own time and learning on, and, two, just how strongly you
are exiting the course, whether indeed you've performed at the level of an A,
so to speak, or a B or a C but not necessarily worrying, therefore,
about these more marginal numbers that we accumulate during the week now based
on these tools like check50, design50, and style50,
which themselves can evolve into more teaching tools
than they are for assessment alone.
So that then, over the past few months alone,
is how CS50 has been taught using AI thanks to CS50's own team
and so many of our friends in industry around the world.
This here is the title officially of a paper
that we presented at the ACM SIGCSE, Special Interest Group
on Computer Science Education, conference most recently.
So if you'd like to dive into more of the details of these here talk,
if you look up this paper's title online,
that should lead you to it as well.
So this then was AI and CS50.
And this, of course, is CS50.