[MUSIC PLAYING] DAVID MALAN: This is CS50. 

[MUSIC PLAYING] 

DAVID MALAN: Hello, world. This is the CS50 podcast. My name is David Malan. 

BRIAN YU: And my name is Brian Yu. And today, we thought we'd discuss academic honesty in CS50. And so every year in CS50, we always have some number of cases of academic dishonesty where some number of students submit work that isn't their own, either by copying homework from a friend or by looking something up online and using a solution they find online as part of their solution. And so this is something that CS50 has had to deal with for years now in terms of how best to address this type of situation, and how best to prevent academic dishonesty in general. 

DAVID MALAN: Indeed this was-- when I first took over the course myself back in 2007, it was really an end of semester process. After the teaching Fellows would evaluate student's work and provide feedback throughout the semester, I would finally, all too often by semester end, carve out some time in order to then cross compare all of the submissions from that semester looking for statistically unlikely similarities between students work. Indeed, what a student might sometimes unfortunately do is copy the work of another student, lean too heavily on some resource online, copying more than a reasonable number of lines of code. 

And so by cross comparing all submissions with software itself, do we then notice which lines of code are in both student A student B's work, and then conclude ultimately, that statistically this was unlikely to happen. 

BRIAN YU: Now, how exactly do you draw those conclusions. Because I'm thinking about a programming language like C, there are only so many parts of the language. Their for loops and their conditions. And probably everyone's solutions to similar problems probably have these sorts of elements. So what exactly do you look for in this process? 

DAVID MALAN: Yeah, it's quite fair. If we relied on this kind of cross comparison for programs like Hello, World, everyone would appear to have written exactly the same code. But as soon as we get into CS50's second and third weeks where the programs they write in C tend to get a little longer, there does end up being more opportunity for creativity, for different stylized actions by students. And so students code does start to drift. Even though at the end of the day the solutions might still be using for loops and while loops and conditions and so forth, students might format their code slightly differently. 

They might write slightly different comments. And so what tends to happen over time, as the programs exceed maybe 10, 20, 30 lines of code, is there enough variation? And indeed, unfortunately, what we often notice is not even necessarily that the code is identical, because as you know, that in and of itself might just be a coincidence. Especially, when nowadays we have 800 students, it is absolutely going to be the case that two students write, by chance, very similar code. 

But unfortunately, the kinds of things we tend to notice is when students have the same typographical errors, or they use precisely the same variable names, or they make precisely the same mistake in precisely the same location. And at that point, our instincts start to kick in and we look at code like this and start to realize, while this may have happened by chance, on scale the odds that had happened in this line and in this line and in this line between two students code is just more likely than not better explained by some deliberate act. 

BRIAN YU: So at Harvard at least, when there are cases of academic dishonesty, they're usually referred to some administrative body, which now is called the Honor Council here at Harvard. And I think you've pointed out and a couple other people have pointed out that CS50, though it is the largest course that the university, does refer far more people to the Honor Council like any other class on campus. Do you think that has to do with something about computer science or introduction to computer science? Or why do you think that might be? 

DAVID MALAN: No, I don't. And that's certainly an unfortunate distinction that we've long had, say for, one or two years where there are issues in other departments. No, I don't think that computer science students are any less honest than their classmates in other fields. I don't think students in CS50 or any less honest than students in other computer science courses. I think it really boils down to one, you and I and educators in computer science are perhaps somewhat uniquely positioned with tools-- with software tools via which to detect it. 

And in a large introductory course like CS50, I think it's important not only out of fairness to those students who are behaving honestly throughout the term, but also because one of our goals should be in this course, to teach students the ethical application of computer science. That we should be holding students to those same expectations as are prescribed in great detail in the courses syllabus. And so I think it's really a function of our one, looking for it. And to two, through on it that really ends up explaining the large numbers. 

BRIAN YU: Yeah, so I'm looking here at the data from past years in CS50, and it does seem that there's also a fair amount of fluctuation in terms of what percentage of students in the course end up being referred to the Honor Council. Like, in 2009 for example, it looks like nobody was referred to the Honor Council. And in other years like 2010, 2012, there's like 1% or 2% of students. But in other years like 2015, it's up to 5%, 2016 is up to 10%. What do you think accounts for that fluctuation because that's a pretty big difference between one year and another? 

DAVID MALAN: Yeah, there really has been as you say, from 0% to 10% depending on the year. I think it's a few things. Part of it I think is just a function of how much time I or we put into the process. I think the year in 2009 when there were 0%, I did look for worrisome instances at that particular year, but admittedly in retrospect, I probably spent less time that year than the subsequent year. Because the subsequent year it went up to 2%. 

With that said, it might have been by chance, just a group of students who exhibited this pattern of behavior with far less frequency than others. So I think that's certainly possible as well. But I think the uptick in more recent years for instance, 10% in 2016 and roughly 4% or 5% then, which is where we've been rather in equilibrium the past few years, I think is also a function of just how much time we invest in it. So back in 2008, and for a few years there after, it was only me who is engaged in this process. I would run the software by myself. I would look at students submissions side by side. And I would ultimately decide which to refer forward to Harvard's Honor Council. And then ultimately, document all those cases. 

But in more recent years have we involved more of CS50s senior staff in the process. The upside of which is that we can now one, analyze the submissions roughly on a week to week basis. The upside of which is that we can provide the Honor Council with the tails far more quickly. Students themselves, while though, never a pleasant process at least no sooner rather than later, rather than getting to the entire end of the semester and then realizing just how many or how often they cross some line. 

But two, the fact that we have multiple human eyes on it means that we do allocate more time week to week on each of the individual submissions and the crossways comparisons thereof. The upside though of those multiple humans, we now have two or three of us who ultimately vote on whether or not a case should move forward to the Honor Council is that I at least, and hopefully all of us, have much more comfort in sending a case to the Honor Council because not one pair of eyes, but two or three have all adjudicated it to be a clear indication of a line having been crossed. 

BRIAN YU: Can you tell me a little more about that process? You've talked about now that there are now a couple of eyes that are all looking at the submissions, but you've also talked about software being involved too. So what is the interplay there between the role that software plays in trying to detect this sort of thing and the role that people play in trying to detect academic dishonesty? 

DAVID MALAN: Yeah, I should first emphasize that it is not software that is ultimately disciplining students or referring them to Harvard's Honor Council. It is rather just a tool that we use as a first pass. Given that we have some, nowadays, 800 students, each of whom are submitting 10 homework problems over the course of the semester. This is a big O of-- n squared problem times 10 or so. So it's a huge number of comparisons that need to be made, and it just wouldn't be practically done by hand or by eye alone. 

So what we do is run software that literally cross compares every submission against every other submission sometimes, within the current year or even, based on our archives, against recent prior years as well which explodes the problem even more. And what we get out of that software based process is a list from top to bottom of pairs of submissions that the software considers worrisome least similar. And then we, the humans, typically go through the top 50 or the top 100 matches on that list and use our human eyes and our own experience and our instincts to decide, ah, this just happened by chance or, oh, as you said, this is a relatively short program like Hello, World or Mario. This is just bound to happen at that point in the semester. 

But certainly as the problems get more sophisticated and the code gets longer is it more clear to multiple humans that, hmm, looks like something's awry here, especially when it is again, the same variable names or the same comments or worse, the same comments with typographical or grammatical errors in exactly the same place, odds are that's much more likely to indicate copy paste than it is two students independently in their own rooms, on their own laptops literally writing in the same place the same errors. 

BRIAN YU: Makes sense. And it's also interesting that depending on the type of software that you use, in the same way that a compiler can take a C program and figure out what is the structure of the program and compare the structure of a program to another, that these sorts of comparison programs can do the same thing. They can take two pieces of code, and even if they might use slightly different variable names, can still look at the structure of the program as a whole and try and compare them against each other to do some more sophisticated comparisons. 

DAVID MALAN: Yeah, and thanks to some of CS50s team members, Chad and [? Yella ?] and Kareem, we now have our own tools, Compare50, which automates this process for us. And you can perhaps, given your experience in the space, speak a little more perhaps to the algorithmics underneath the hood? 

BRIAN YU: Yeah, it is really Chad and [? Yella ?] and Kareem that were doing a lot of the work there. But algorithmically, it's sort of an interesting challenge to figure out how to do these sorts of comparisons. Because even though it might seem like a computer is obviously going to be able to do it faster than people are going to be able to do it, it's still a lot of work even for a computer. Especially, if you consider like 800 students in the class being compared against all of the other students, plus all of the students who have ever taken CS50 before, not only for one problem, but for all of the problems in the course. 

That's a lot of work for any computer to do. And so there is a lot of interesting algorithmic efficiencies that have been put into the software in order to make it work a little bit better. Trying to take advantage of things you actually learn about in CS50. Things like hashing in order to store data inside of a hash table so you can very quickly look up whether or not you've seen a particular pattern of characters in a file before. Those sort of data structures all come into play if you start to think about, how do you try and solve this problem in a way that's efficient? 

DAVID MALAN: Yeah. And besides software, certainly our own policies have evolved over time. So you know for instance, that in a few weeks time, we'll be presenting at a computer science education conference called CSEIT a recent paper that a few of us worked on based on our experience with issues of academic dishonesty over the past few years. And it's perhaps worth noting that software aside, I think one of the more noteworthy policy changes we introduced some years ago was CS50s so-called Regret Clause. Which was just a single sentence that we added to the courses syllabus that encourage students to come forward if within 72 hours of submitting some work, they realized that, oh, they had indeed crossed some line. 

They had copied unduly from some resource online. They had copied some portion of code from a classmate or otherwise, somehow other across the line that was prescribed in the course of syllabus as being not reasonable. And what we committed to doing in writing in the courses syllabus was there would still be penalty and there would still be consequence, but it would be limited for instance, to our zeroing the problem or the problem set that the student had submitted. And we committed not to escalating the matter to Harvard's Honor Council. 

The hope was that we could actually turn what had historically been purely punitive processes whereby we detect some transgression, we refer it to the Honor Council, and there after the student is penalized in some way, the most extreme outcome of which might actually be required time off from Harvard University itself. We wanted to create a window of opportunity where students after some sleep, some thought, some reflection, can actually own up to a mistake. Because for so many years, so many of our cases were truly involving students who at 2:00 AM 3:00 AM 4:00 AM are under very little sleep, under significant amount of stress, and with a deadline not only in CS50, but perhaps some other course looming, made some poor decision to take the quick way out to just copy and paste someone else's work and submit it on their own. 

And even if they've decided or realized a day or two later, wow, really didn't mean to do that. Really shouldn't have done that, we had never described a well-documented process for how they should handle that and how they could own up. And so this Regret Clause was meant to help ideally chip away at the total number of cases we were seeing. But ultimately, help students meet us halfway so that it becomes more of a teachable moment if you will and not just punitive. 

BRIAN YU: So I remember when I first took CS50 in fall 2015 it was, I remember seeing the Regret Clause in the syllabus. And I remember being a little surprised. Because it wasn't something I had seen before. It's not something that many other classes do. Not really anything that I was familiar with. So I'm curious about where the policy came from? Was it inspired by any other policy? Or where did you start to find your way to this idea? And what was the process like for bringing this into the course? 

DAVID MALAN: Yeah, it was really inspired by having, for almost 10 years, watched the number of cases come through CS50 and watching the circumstances that ultimately explain them. Again, these late night poor decisions under a great stress. And it just felt like we, the teachers of the course, should be doing or could be doing a more proactive job at trying to tackle this problem. And not just looking to detect it, but looking to teach students how to one, ideally avoid it altogether. 

But two, even if they do cross some line how to address the situation then. And yet, it was not with great ease that we rolled this out. There were absolutely some sensitivities on campus among administrators, among the universities Honor Council, who had long standing processes when it came to issues of academic dishonesty, not only for CS50, but all courses at Harvard. The upside of course, is that by having a central body, Harvard's Honor Council, adjudicate all of these cases, you have uniform processes. You hopefully have more equitable outcomes overall. 

And there was great concern initially in some circles that we were now doing something more on our own internally. And so it only debuted after quite a few conversations with Harvard's Honor Council and administration so that we can ultimately get folks comfortable with what, at the time, was an experiment, but now is an ongoing six year policy for us at least. 

BRIAN YU: All right so now six years in, policy has been around for a little while. Do you feel like it's done what you expected it to do? How does it compare to what your original goals and objectives were for what the policy would do for the class and for students? 

DAVID MALAN: Yeah, so we hoped that it would actually chip away at the total number of cases that we were referring to Harvard's Honor Council, but it did not in fact, do that. Interestingly enough, the number of cases we have referred to the Honor Council since have been roughly the same or even higher in some years than prior to the Regret Clauses introduction. We had the wonderfully successfully and nontrivial number of students avail themselves of this clause. Most years so in the court clauses first year, 2014, we had 19 students come forward under this clause, reach out to me in the courses hedge, generally by way of an email first. 

After which we would then schedule time to chat with me. And I would chat with these 19 students one on one and better understand what had happened and what had they done. Better understand what circumstances had led to them having made whatever decision it was we were then discussing. And then ultimately, explicitly tell them, all right, let's consider the matter behind us. After zeroing the particular work in question to reassure them that this was indeed the end of that process. But the beginning, hopefully, of a healthier approach to future problems sets. 

And we would then encourage them to-- and discuss with them ways for better managed managing their time, better managing their stress. In some cases, too, it came to light that there were extenuating circumstances. Students struggling with issues at home, with their family, with relationships, with other courses, issues of mental health. And so what was a pleasant revelation to us was that we were able more proactively than had been possible in the past to connect students with support resources on campus, whether academic in the case of tutoring, or perhaps health in the way of mental health. 

So that too seemed to be a positive outcome and the experience that we were able to connect up to 19 students that first year with other resources on campus. And there after it fluctuated. In 2015, we had 26 students. In 2016, we had seven students. Then it went back up in 2017 to 18 students. And I think this variation is partly just a function of messaging on our part, on my part. How much time we spend in lectures and in emails during the semester reminding students of the policy's availability. I also suspect that there's some ebb and flow based on the current-- the given year. 

If more students in this class know that a student in the previous year might have invoked this clause there just might be broader awareness of it. But it's been a good number of students, I think every semester. However, the fact that we didn't see a downturn in the number of cases we referred too was also a surprise. In fact, in the first year of the Regret Clauses existence, it turned out that most, if not all of the students that invoke the Regret Clause did not even appear on our radar when we ran our software based cross comparisons of their work. 

Which suggested that had they not come forward, we actually would not have noticed and they would not have been connected ideally with these resources. And so that too was a bit of a surprise. These students invoking the Regret Clause dare say composed a different demographic of students that we hadn't yet previously identified. Students who had indeed crossed some lines in many cases, but that had not been connected with or been offered some teachable moment that might actually help them course correct. 

And I should note too, that of the 19 students, 26 students, and so forth, not all of them it had indeed crossed some lines. In several cases each year, were students unnecessarily worried. And so I would simply reassure them and thank them for coming forward, but not to worry, you've navigated the waters properly. 

BRIAN YU: Yeah, it's really interesting that now by reaching this other demographic, you've been able to have these sorts of chats that otherwise may not have been able to happen and connect them with other kinds of resources. I'm curious as to what are the kinds of advice you give to students that find difficulty with time management and stress? Because I think this is not a unique problem to CS50 that and other computer science classes are just in school in general or even outside of school. 

Like, time management, stress, managing these things and making good decisions is-- it's challenging. And something that I'm sure many students and other people face. 

DAVID MALAN: Yeah, absolutely. To be honest, it's fairly straightforward things. It's things that we even put in the courses syllabus or FAQs often. For instance, in a programming class like ours, start early. You have nearly seven days from start to finish for each programming assignment. And the key to avoiding a lot of the stress is to just start early, so that when you do invariably hit a wall or encounter some bug that you just can't quite see, you can go to sleep, you can go for a run, you can take a shower. You can take a break from it and come back to it some hours or even a couple of days later and have that perspective. 

I mean even I found in the real world that I do not produce good code when I, myself am under stress. It's no fun. It doesn't yield correct results. And so really helping students realize that, it is a relatively simple fix. They just really need to take charge and commit themselves to that. Besides that, it's often a matter of referring students and reminding them of the many resources that the course offers on campus, whether it's the courses lectures, or sections, or office hours, or notes or tutorials, or any number of online and in-person resources. 

And just reminding themselves that you need to meet the course halfway and take advantage of these resources. And it's no surprise that you are struggling if you're not availing yourself of at least some of these resources. 

BRIAN YU: Yeah, actually it's always incredible to me when on our problems at forums, we always ask students like, on what day did you start the problems set? And so many students respond like the day of the deadline or the day before the deadline for a project that we wrote with the expectation that it will take students a week to complete it. And students are trying to do it like day of or day before. It always amazes me the number of cases where that ends up happening. 

DAVID MALAN: Yeah, so I think the more we can send that message even before we get to the point of a student having regret clause this conversation, the better. I should note though too, that another surprise effect of the regret clause was not even that we-- or the number of cases we referred didn't go down, but rather at least in at least one year they went significantly up. In 2016, and as you noted, is when we had 10% of the courses student body. So this is 10% of the students taking CS50 referred to the courses-- to the university's Honor Council. 

But to be honest that too was in part. And I think our numbers since have been partly a reflection of our feeling that when we do detect what appears to be a straightforward case of academic dishonesty, plagiarism of some sort, duplication of code, these days, I think I personally am even more comfortable referring the case than I was in years past because we have given students an opportunity to meet us halfway and reach out. And indeed, as you know, in every one of the courses problem sets this year on the form via which they submitted their work, we asked them to check a checkbox to acknowledge their understanding of the clauses availability. 

And so at that point, if we are not only reminding students each week that it's available and they are not thereafter taking advantage of it, it seems quite reasonable, I think, for the course to move forward with the more traditional punitive process involving the Honor Council to investigate whether indeed the line had been crossed. 

BRIAN YU: I'm curious. So we often talk now about like the line being crossed and what it means to cross the line. I'm curious about how you see this in the context of programming assignments in particular. like if you're writing an essay and you copy a sentence, that seems like very clearly copying. But in the case of code if you copy a line of code you see from Stack Overflow for example, if you're looking up like, how do I solve this particular problem, and you incorporate a line of code, that that might not be crossing a line. So how do you think about where the line is in the context of a programming assignment? And how to teach that kind of thing? 

DAVID MALAN: Yeah, it's a really good question. And it's a common question, because I think there's a perception among folks both in the software world and non-software world that this notion of academic dishonesty in a programming class itself is incompatible with the idea of programming. And I do very much disagree with that. The lines that we prescribe to students, both in broad strokes and in very precise bullets in the courses syllabus, essentially try to teach students to be reasonable so to speak. And what might that mean? 

Well, early in the semester in CS50, we of course, have students in C, and later in Python implement Mario's Pyramid. So a sort of pyramid-like structure just using some ASCII art to paint that picture. And it involves ultimately like a couple of for loops. It would be unreasonable for students to go off and Google or look on Stack Overflow for something like, how print Mario's Pyramid. That would be a search for the outright solution to the problem. 

And surely it is not our intent to assess you on your ability to Google a solution like that as opposed to crafting it yourself. However, it would be very reasonable for instance, to Google something like, how write nested for loops in C. Or how print spaces in C. Because it's actually not obvious to students one, how you can actually have two loops and one nested inside of the other using different counting variables. And two, how to print would appear to be blank spaces on the screen, not quite appreciating that it's actually just the SPACEBAR. 

So I think it's very reasonable for students and it is allowed in the course syllabus to look for short snippets so to speak of code. Where a snippet itself is one line, few lines, but it is not the essence of the problem. And so indeed when we do find that students have crossed the line, what has happened is we notice some curiosity about their code. It's maybe very similar to another student's code or it suggests a technique that we haven't taught in the class or some syntax that's not consistent with what we know students have seen in the class. And so we ourselves might Google certain key phrases or portions of code or comments that we see in their code. 

And sure enough, it too often leads us to the very same GitHub repository or Reddit post, wherein someone else has posted exactly that same code that the student has copy pasted. And so there too, the kinds of cases we are referring are not the many, many, many students code who very reasonably use these kinds of digital resources. But the ones who use these resources, and then take shortcuts to submission as by just copying and pasting many lines of code that they see. 

BRIAN YU: So other than the Regret Clause now, which we've talked about for a little while, have there been any other things you've thought about doing or things you have done to the course in terms of thinking about how to either address academic dishonesty when it happens or to try to prevent it beforehand? 

DAVID MALAN: We have. So couple of years ago, we introduced the courses Brink Clauses, so to speak. Which was a couple of sentences inspired by a colleague of ours at Princeton, Chris Moretti, who gave us some really inspiring language that encouraged students in the courses syllabus to write us late at night just as they felt themselves being on the brink of making a poor decision. That is to say, even when you and I and most of the courses staff might be asleep and a student might be working late at night on their work, it would be reasonable to assume that they could get a response to a request for an extension for instance. 

And so with this brink clause prescribed was a mechanism for students to send that note to say, listen, I really feel like I'm in a bad place. And I worry I'm about to make a poor decision as by copying and pasting too many lines of code online. I'd like to discuss this tomorrow and indeed that's what the syllabus asked them to do. Go to sleep, don't submit your work. We'll figure it out in the morning. And just writing students to write us and meet us halfway under that sort of duress was the intent of the clause. 

Unfortunately, when it was invoked some number of times that first year, based on the wording of the emails, based on the conversations we had with students, it really devolved into a backdoor to just extensions. We did not believe, ironically, that most of the students who were invoking this clause were actually on the brink of doing something academic dishonest. They were simply on the brink of not meeting the deadline. And so we ended up removing the clause from the courses syllabus, . Ultimately 

But I'm glad we did try it, but this was one example of a measure that, at least for us, in our context, in our implementation failed. But I do think more compelling has been what we introduced a few years ago in the spirit of the Regret Clause, but whereby we actually initiate the conversations. So it's not infrequently been the case that when we've crossed compared so many students submissions that there's a few cases that seem a little worrisome, but it definitely doesn't seem like it's over the line. We certainly wouldn't refer them to the Honor Council on that basis. 

But we realized that this then would is an opportunity for us to maybe go chat with those students now and say, hey, listen, you appeared on our radar. We think it's because of the similarities between your code and maybe some other students. And we would leave the other student anonymously out of it. But we would then ask the student, how did you get your code to this point? Walk us through the process and let's figure out how you came so close to what we worried was crossing a line, so that you can just avoid it moving forward. 

And so these interventional conversations, as we describe them internally, I hope has actually gone a long way to just helping students navigate the waters. Even if they don't cross those lines, they at least now are being more conscious and thoughtful about what it is they're doing. 

BRIAN YU: And what do you usually gather from those sort of interventional chats? Like what sort of actions you find that students are taking? Does is seem like there's some teachable moment there that you're helping students with? 

DAVID MALAN: I think so because not infrequently would it be the case that two students were indeed working reasonably on the homework assignment together. But they were perhaps asking each other a few too many questions about code. It wasn't necessarily entirely in pseudocode or in English, their conversations. And maybe one was being shown the other's code, which is allowed within some circumstances per the syllabus. But maybe a little too frequently. 

And so as such, their work was just sort of over time, converging to become one in the same. And so given that we would have these chats within a week of them having done that, it was usually pretty obvious to students like, oh, let's not do that again. And recalibrate their approach. 

BRIAN YU: So it seems like all in all, CS50 has tried a lot with the Regret Clause, with the Brink Clause, with these interventional chats that you've had with students. A lot that CS50 has done with regards to the issue of academic dishonesty and trying to create teachable moments out of that. And trying to work within the university and with students on how to improve that situation. What do you think are the lessons to be taken away for other courses? What can other classes do, either in computer science or outside of computer science that they can do based on the lessons that you and the course overall has learned from these years of working with these issues of academic dishonesty? 

DAVID MALAN: I think one takeaway has been just clarity. Our policy in the courses syllabus is not short, but it is detailed. And that's the result of a lot of situations having arisen over the years, a lot of conversations having happened over the years. And so I am glad that we do documents so clearly for students, where the lines are and what our expectations of students are. Toward that end too, I think it has been a good thing that we've introduced these interventional conversations. 

Even if a course is not as involved in the mechanics of the process as we are, they're not necessarily running software across compare your submission. But when something does appear on the radar, if a teaching fellow or teaching assistant does notice some curiosity in the student's code, it's dissimilar to their code last week or it's a little too similar to another student's, I think just being comfortable reaching out proactively to those students, not to impugn them, but rather to say, listen, we have some concerns. 

We don't feel you've crossed a line, but we'd like to better understand what you've done and how you did this. So that we can steer you in the right direction moving forward. That too seems a very straightforward, healthy and teachable opportunity. And as for the Regret Clause, I certainly think it's worth trying in other classes. I think it certainly is completely reasonable that a course, whether ours or anyone else's, just clearly defines what steps students should take when they find themselves in certain situations. 

And prior to the forgot clause it was ill-defined. What should a student do if they make a poor decision, especially late at night and then they do actually regret it the next day or some number of hours later? There was no well-defined process. And while technically, there was nothing stopping a student from coming forward and turning themselves in, I can certainly appreciate the trepidation that a student might have with taking that on not knowing what the outcome might be. Especially, if they assume it might even be time off from the University itself. 

So I think the fact that we've sort of clarified how to conduct oneself before you get to that point, after you get to that point, and after we have detected as much, is just only fair to students in the class. 

BRIAN YU: I think there are a lot of very useful lessons there in terms of what classes can start to do about this sort of issue. Certainly, if any of you are interested in learning more about this, we've actually written a paper, the two of us along with Doug Lloyd on CS50s teams about economic honesty in CS50. So we can provide a link to that if you're interested in reading more about the policy and about the Regret Clause and about other interventions that CS50 is made on these sorts of issues. 

DAVID MALAN: Indeed. The title is Teaching Academic Honesty In CS50. If you want to Google something like that. And if you're more interested in the software side of things and the cost comparison of submissions, if you go to github.com/cs50/compare50 you'll be able to play around with the open source software there as well. 

BRIAN YU: Certainly, if you have any feedback about today's podcast or suggestions for future podcast ideas, you can always reach us at cs50.harvard.edu. 

DAVID MALAN: This was CS50.