DOUGLAS KIANG: All right, great! So I wanted to talk today about just some strategies that we've come up with for managing the CS50 classroom. You know, I've been teaching for, probably, about 30 years, I think. So I started first teaching CS50 when I was able to join a small group of educators in Seattle. 

We got together and were trained, and David's vision for CS50 and his-- just a way of sequencing the concepts really kind of had a huge impact on how I teach computer science. And so I'll always be really grateful to him and to his team for that. And a lot of what I'm going to present today-- the resources are in that folder. 

And a lot of it has just come from trial and error. We just try lots of different things. I'm very fortunate now to be at a school that offers four years of computer science, and CS50 is the intro course. So you have to-- it's a prerequisite for everything else. 

So if you want to take CS at Menlo, you have to come through CS50 first. And it's a rigorous course, but we provide lots of scaffolding and support, and we get everybody over the hump. And by the time they've had a year of CS50, they're ready to tackle all kinds of challenges in AI or app design or independent projects. 

And so we really love the foundation that CS50 gives us for this. So I think where we're going to start-- we're going to cover-- I'm hoping to cover these six things, and then have some time to take some questions at the end. 

But I'm going to talk about how we structure a lesson. I'm going to talk about this concept of belonging. If we want to create an inclusive environment for learners of computer science, what does belonging look like? 

I want to go over a quick game that we use to help prepare kids for the AP exam. I'm going to talk about a system for late work that has really changed-- solved a lot of our problems, and actually given a kind of a flexible framework for kids to actually get work in that doesn't become a pain for me and my grade book. 

I want to share some tools that we use for scoring the CS50 problem sets. And finally, I want to talk a little bit about this big experiment we've been doing this year, which has been essentially to really try to focus our feedback, on competencies, and get kids to master what we think all those qualities are that make you a good computer scientist without focusing overly much on a single grade describing how you've done. 

So with that said, I want to talk a little bit about lesson structure. So we typically-- I think of a lesson as spanning one or more days. And we usually start with an Unplugged Activity, then we go to what we call a birdhouse activity, which are guided activities, where everybody kind of has to come up with the same solution or similar solution. 

And then, where we can, we really try to push kids to do an independent project or an application of each concept. And I want to start by providing sort of a quick example, within the context of CS50, of what that might look like. 

So for the Unplugged Activity, kids come in to class. They have a paper-and-pencil activity we usually start with, so we give kids a bunch of scraps of paper on which we've got 1 through 9. And we ask them to keep track of the first number in that list. Let's call it j. 

That location-- we want you to swap that location with the last one. And then, we look at the next number, and we swap that with its counterpart. And we look at the next number, and we swap that with the counterpart. 

And we ask them, notice what's happening to this array, essentially, of numbers. And they'll notice that at some point, those numbers are entirely reversed. And then, if they continue swapping-- they move j, and then swap it with its counterpart-- look at what happens. 

By the time j gets to the end of the array, that list is in the same order again. And so we actually have them do this with index cards or pieces of paper. And then, we introduce the problem set, and the problem set is called Filter. 

It's one of the CS50 AP problem sets that involves working on an image. And essentially, that is the algorithm that they're using to reflect or reverse that image. But this is often-- if kids come up with a solution, this is often a solution that they come up with. 

And they say, hey, filter's not working. It's not changing anything. The image looks the same. And that's because there's a pretty common error in this code. This is something that lots of kids used to do, because they just sort of-- when they went through each row in the j loop, they started at the beginning, and they just went all the way to the end. 

And when we do the Unplugged Activity, when we started before, we hit Filter, if we start with the Unplugged Activity, we get kids to realize that actually, you only need to go halfway. Because if you go more than halfway, it ends up the same. So actually, Filter was working. It was reflecting it. 

It got halfway. It reflected it entirely, and then it continued and reflected it back again. And so it's not that nothing was happening. Something was happening, but it was happening twice, and you only really want it to happen once. 

And so this is an example of an Unplugged Activity. And so we start every class with a pencil-and-paper activity. Because we actually find that for emphasizing thinking, it's actually a distraction to have them working on the laptops and working in-- even working in the Sandbox or working with check 50 or everything else. 

We want it to really focus on the thinking. And so we actually will make this paper thing, and we'll say, work with a partner in this activity. You've got your cards. Go through and write down what is the number order, and what's the index of each number, as you go. 

And we have them see, what does that pattern look like, and we make sure that they can articulate what that is. And that's what we call a Daily Check. And when they do that Daily Check, we want to make sure that they're actually going through and thinking and articulating their thinking as they go. 

Now, we don't collect these. We don't collect them. We just ask the kids to do them in class. They're actually-- they only spend about 10 minutes on it at the beginning of class when I'm taking attendance. 

I've put, in the resources folder, Daily Checks for pretty much every problem set that we do. And there are multiple-- for many of them, there are multiple daily checks that we use. So if you look at filter, these are all the checks. 

They are intended to be printed out. They're worksheets, essentially. But they hit on everything, and they help to review and assess kids' knowledge for all of these. 

And we find that asking kids to articulate their thinking and solve these little problems at the beginning of every class in a very low stakes way-- because we're not collecting them. It's not-- we find that it really helps kids sort of fill in those gaps. 

And we also will give quizzes. And we'll draw questions for the quiz. Some of them will come from the Daily Checks. So if the kids are blowing off the Daily Checks-- or the other way to look at it, I guess, is if they're doing them faithfully and they're checking the solutions, they will do better on the quiz, because the quiz will contain stuff that they've already done. So that's in that Resources folder. 

I also have a guide that pretty much goes through all the concepts that are covered for each Daily Check, and what the relevant problem set is. And so we have them in here. And I love the fact that we have this community. I've learned so much from the different people in this community-- from Bill, from Margaret, lots of people. 

And you know, I've learned just as much or more, just from the contributions that other people have had. So if you-- it would be great to get people contributing more of these Daily Checks. If you find these useful and you want to add to them, please go right ahead, and let's add more of them and get this to be a really useful community resource. So I'm very excited to kind of introduce this-- hopefully, this, along with the practice problems, will be useful for people. 

So those are the Unplugged Activities. Then, we go to what we call the Birdhouse Activity. And a Birdhouse Activity is kind of like a woodshop. You know how everybody makes a birdhouse as, like, their first activity? 

And birdhouses are great, because you can quickly assess them. You can look at the birdhouse, you know, what it's supposed to look like at the end, and you say, OK, this one-- the roof's a little crooked. It means you need to work on your measuring, or you need to work on your cutting. 

But everybody works together. They do the same project. That's a birdhouse project. And then, I think of birdhouse projects here as Sandbox Activities. Because after we do the Unplugged Activity, then we throw-- I might start the next 10 minutes of class. 

I'll start with the Sandbox. I'll say, OK, everybody go to this URL. I'm going to teach you something. This is how we do a swap. And I literally will walk them through on the board how to do this, and they kind of type it in. 

When they hit that URL that I give them, they open up their own copy of it. So the work that each student does in the Sandbox is different from everybody else's, but everybody starts from the same source. And then, the next thing we'll do is, we'll say, OK, now start writing the code. Practice reversing the number of orders-- the order of numbers. 

I'm going to give you a series of numbers in an array. I want you to reverse it. I don't want you to worry about how to print it out or anything else. I've already coded all that for you. But just within the context of this little Birdhouse Activity, I want you to talk with other people at your table and figure out how you go about incrementing the loop variable and swapping at each location. 

And then, once they've done that, OK, now, do it with a two-dimensional array. And we have them do that. And by the end of the first class, they've actually got working code in their Sandbox. And they can go to their recent Sandboxes, and they can pull it up and they can see it, so that when they actually start working on the filter problem set, they have working code. 

Whether they realize it or not, this is the basis of the code they're going to need to use to do reflection. And so we find for reflection and blur, and some of the, I think, the more difficult problem sets for a lot of students, by scaffolding it in this way with the Birdhouse Activity, with an Unplugged Activity, kind of helps them when they actually go to do the actual problem sets. 

And when you click Share, it'll bring up a bit.ly link, and then you can-- excuse me-- you can share that with students. So I find the Sandbox to be really, really helpful when I teach kids concepts. 

OK. Now, the problem with birdhouses is that most schools stop after the birdhouse, right? They say, OK, we've taught the skills, we've assessed the skills, let's move on. But everybody pretty much has-- it's very hard to differentiate with those birdhouses. 

And so where we can, we try to offer the chance for students to do an independent project. Now, Birdhouse Activities are really high on new skills, because you're teaching the new skills, and you're assessing them. But they're very low on creativity, because everybody's coming up with the same thing. 

An independent project, on the other hand, is fairly low on new skills, because I'm not asking you to demonstrate anything new, but what I'm asking you to do is apply what you've learned in some way that is more creative. And so that's where I think these Birdhouse Activities and independent projects go hand in hand. 

And so here's an example. If we've gone through Filter, the students have done Filter. The problem sets, really, are Birdhouse Activities, really. What do we do with that skill? What do we do now that we know how to do a blur, we know how to do a reflection? 

What would that look like? And so we give them a challenge. Your challenge is to make art. Your challenge is to modify an original photo, and that's the problem that we give people. 

And so the example is, take a photo. And you're going to apply some filter that you've created to that photo. And you can make it a combination of reflection or blur, or look up how other kinds of filters are created. 

And this became one of the favorite problem sets of our students this year. It wasn't one of the ones that are the normal curriculum. It was this thing that we asked them to do that didn't teach a lot of new concepts, but in fact, it just had them apply it. 

And we had each student write, in their own words, an artist statement that talked about, what kind of Filter did you use? How did you change it? And we posted it. We actually printed up a bunch of them. We printed it like an art gallery, with their artist statement along with the artwork. 

It was amazing. So the differences that students came up with-- they came up with all different kinds of examples. And ultimately, we had the students post them to a Padlet. 

And Padlet's a great bulletin board where kids can post all of these. Some kids did Minecraft-inspired artwork. And what I love about it is, they're all different. And even students who really struggled with Filter felt successful by the end of our artwork problem set, because they really had a chance to undertake that challenge in the context of something that they cared deeply about. 

So-- and it's something that was personal to them. So I guess my challenge to you would be to go beyond the birdhouse, you know. Let's introduce these things with the Unplugged Activities and the Birdhouse Activities and the problem set. But where you can, during the year, find ways to take a bit of a breath, and have them apply those same concepts in the scope of something that asks them to demonstrate it in a creative way. 

So here's the next thing. I wanted to talk a little bit about this idea of belonging. And the idea about belonging is that if we want to be more inclusive, we have to move beyond the stereotype of a computer science person. 

As long as students can say, oh, the type of person who takes CS50 is, then I think we're sending the wrong message. It should be for everybody. Everybody can do it. Everybody should take it. 

It should not be an identifiable thing that you're this kind of person if you take this course. And so probably, the biggest message that we give to all of our students is this idea of, you belong here. That's the message that we weave throughout the entire year. 

We actually print up laptop stickers that-- we actually call our course CS1, but it's CS50. We actually got fortune cookies made up. It turns out the fortune cookie was invented in San Francisco. I didn't realize that. So it's a short drive for me, and fairly inexpensive. 

But we actually got these fortune cookies that were made up with different fortunes. And one of them is this idea of, you belong here. Because that goes along with what we really try to emphasize with our kids throughout the year. This was a tough year for kids coming back. 

Social, emotional development is one of the top priorities of our school-- is really taking care of kids. And when you put them in a course like CS50, which is a tough course, I tell them, even for me, when I started taking it, when I did Mario, there's a little voice in my head a lot of the time, when I really get stuck, that tells me, you're not smart enough. 

I have no idea what to do. You don't belong here. I shouldn't have taken this course. And to counteract that voice, we get the kids-- we lead them, actually, in saying this to themselves. When you get stuck, say this to yourself, I am smart, I am resourceful, I belong here. I am smart, I'm resourceful, I belong here. 

And we actually will walk kids through. We'll have them practice it as long as-- until they don't feel silly saying it, or at least feel less silly saying it. Because that is the message in our course. You are smart. You are resourceful, and you belong here. Every one of you belongs here. 

And it's a tough course. You're going to get stuck at different points. But there are people here who can help you. You do belong here. You're in the right place. That's the message we're really trying to send to them throughout. 

Now, people have studied the idea of belonging. It really has two components to it. It is really this idea that students have-- it's a perception of their own ability. One part of it is, do I fit in with the others, and one of them is, do I have what it takes to do the work. 

As much as we embrace-- popular culture embraces this idea of the geek, the nerd, the outsider, these stereotypes about who does computer science. That sends the opposite message to people about, do I fit in. In terms of, do I have what it takes, people are concerned about the rigor. 

We actually changed our course to give students-- we actually defaulted our course to be pass/no pass, or pass/fail. We call it pass/no pass, or credit/no credit. Students can petition to take it for a grade if they want, but we find that by removing the grade and substituting other ways of providing feedback, we really help remove a lot of that stress that students have about, do I have what it takes. 

And so this idea of belonging-- these are some strategies that we use for fostering a strong sense of belonging in our course. We want to provide positive role models. We want to share student success as much as possible-- so the art gallery idea with the artwork. 

Creating more adult advocates, promoting the context over the tools-- why are we doing this? And when David talked about-- in his first presentation this morning, he talked about taking these concepts that are fairly common in computer science, are fairly central, but providing them in a context like decoding a license plate or figuring out something that people see in real life all the time. 

That's the context. That's really important. Building community is super important, just as important as it is here-- building a community of teachers teaching this stuff. You want to build community in your classroom of students who are trying to learn this stuff, and foster this growth mindset-- this idea that this is hard, yes, but if I work hard at it, and if I take advantage of the resources that I have, I can learn this, I can do this, I can get better at it. 

So one of them-- this idea of promoting positive role models and sharing student successes-- students need to see other students who look like them. And so we're very fortunate at our school. We have a number of students who come back after having taken CS50, and they want to be teaching assistants. 

And so we have a whole program where students actually come back, and they work with other students. Primarily, they do office hours over Zoom, and they also help with scoring submissions. So we have, probably-- I think we have 91 students in our CS50 across five different sections. And so each one of our TAs works with one section and takes primary ownership of working with those students, and then helping to score and give feedback as well. 

The idea of, do I have what it takes? So this idea of, can I do the work? We really have to look hard at, what are the things that we were doing at Menlo that potentially were giving the wrong message. So for example, here's a sample of course description. 

So this is a JavaScript course. This is supposed to be for students who don't have any prior experience with JavaScript, but if you read the course description, does this coarse description speak to its intended audience? When we looked at something like this, we thought to ourselves, this is kind of like if you're going to take an Intro to Japanese course, and the course description was all in Japanese. 

Right? Like, this doesn't send the message like, this is safe. This is a safe place for you. You don't already have some experience with JavaScript. And so what we did is, we took-- we tried to take some of these other courses that we teach, and we tried to translate them into what's the context. 

The course title is not Intro to JavaScript anymore. Now, it's Building Websites for Nonprofits. Here's what we're doing. We're working closely with a community organization and getting programming experience. 

Here are some of the different places that we've done outreach for-- things that you'll recognize. These are the people that we're helping. And in order to do that, you need to learn JavaScript, but we'll cross that bridge when we come to it. 

Students signed up in droves for this, where before, that course description sent people running the other way. The other thing to do is think about creating more adult advocates in your community. We're trying to do a lot to change this perception of what computer science is to students. 

What we didn't realize for a number of years was that students were getting turned away by the course counselors, by the guidance counselors that were signing kids up for courses. A lot of them had inherent biases about computer science or about who does well in computer science. 

So they would say, oh, yeah, you've gotten B's and C's in math. You probably don't want to take CS50. You might want to try a different course. And that's the wrong message. We want to be open to everybody. 

A lot of those students could do just fine, math-wise, with it. For a lot of cases, CS50 was the one place where they had success in mathematics, because they had the context. So we didn't realize that some of our girls were getting turned away by counselors who-- bless their heart, they had the best of intentions, but they just were thinking, gosh, you know, you're already taking five courses. 

That's really hard. You might want to look at maybe taking an AP Environmental Science instead, or something like that. So we created these flyers and circulated them to the teachers and really talked to them and said, this is what a computer science course today looks like. It's real-world context. It's a whole TA staff that we have available. It's office hours regularly. It's flexing on deadlines. 

There's all kinds of stuff that we do. It's a pass/fail course. It's not the typical computer science course that maybe you took 20 or 30 years ago. That's the message we're trying to get. So think about, in your own school, who are your gatekeepers? Who potentially could encourage or discourage students before they even get to you and try to send that message? 

Awareness is really important. So we actually produced-- we just created this simple video-- one-minute video. We've gotten so much mileage out of this video to talk about what the course is, because we used students' own voices in it. 

[VIDEO PLAYBACK] 

- What would you tell people about taking the course? 

- I would say it's a really fun and enjoyable class, and I think that you learn a lot and get a lot out of the class. 

- What three words I would say-- number one would be "fast-paced." Number two would be "surprising." And number three-- I guess, "exciting." I know "surprising" and "exciting" are similar, but yeah, those would be my three words. 

- "Challenging." I would say, "intriguing," and I'd say, "hard," but it's good. 

- Gratifying, rewarding, and challenging. 

- Challenging, engaging, and rewarding. 

- Do you think you have to have a lot of experience to take this course? 

- I don't think you need to have a lot of experience to take this course. Because you don't need any skills to go in, and you'll come out with everything that you need to know. 

- I think I knew [INAUDIBLE] I think I knew how to do two things on Scratch, and that was about it 

[END PLAYBACK] 

[END PLAYBACK] 

- Stressful, challenging, satisfying. 

- Fun. 

- Fun and rewarding. 

- Exciting and worthwhile. 

- I think that the collaboration is probably the most fun. I really enjoy getting to meet new people and work with them and struggle through problem sets. And I definitely have made friends that I'll keep for a long time. 

And again, it's consistent with our message throughout, but said in the voices of students. And we shot this in, like, a day on an iPhone, just went through and asked most of the kids just what three words come to mind when you think of CS50, and talk about it. 

So we've found this to be much more useful than anything I could tell them-- things that other students say. So I want to talk about this game, the Explore Game. So I actually was on the development committee for AP Computer Science Principles. 

So I helped with the revamp of that course when we decided to get rid of assessing the Explore task and we moved the assessment of those principles, the impact of computing, into what we're calling Stimulus Questions. And so when students take the exam, they have about five to seven multiple-choice questions that are based on a reading comprehension passage about a fictional computing innovation. 

And so here's an example that I just kind of quickly wrote. This is not a real example, but it's representative of what kids will find. And the idea of this is that we come up with some fictional innovation, and then we're asking students to identify what was the beneficial impact to society, what was a harmful effect to society, what are the effects on data security, privacy storage. 

All the things that we used to ask them to articulate in the written explore task, we're now trying to do through the stimulus questions. And the answer to this one by the way, oops, is B. It's-- you can share your bird call with other people, who can then turn around and share it with their contact. 

So that means that your content could potentially be shared with people who are not in your contacts list. So it's stuff like that, that we're asking kids to be able to infer from reading through these. Now, how do we go about practicing this? 

I actually came up with this game called Mission Feature Feature. And we have been able to use it for app design, but this, in many ways, is kind of what we were doing. We're coming up with the stimulus questions. 

It was like, OK, let's think about some. It can't be a real app. So what's some app that has a combination of features that have some impact on technology and society? And so I created this deck called Mission Feature Feature. 

You can just use one deck of cards, and it works for the entire class. And the cards, basically, are divided into Mission cards and Feature cards. And so we'll actually just play this in this group here. 

The idea behind it is, if you have a mission-- so for instance, let's say the mission you draw is finding a lost item. You're going to draw two features at random. One feature's a gyroscope, and one is a touch screen. 

So now, you have about five minutes to brainstorm, OK, what is an app that uses the gyroscope and the touch screen to find a lost item? And then, we'll ask kids to just speak, talk maybe a few sentences, give me a prototype. What's your app look like? 

And so a student might say, OK, I've got an app that uses a radio tag that broadcasts the location of your lost item. And then, as you tilt the device, you see an overhead map, and then you touch the screen to show where the signal is the strongest. And what is a harmful effect to that? 

OK, well, maybe, you know, maybe it is broadcasting-- well, this doesn't say it's broadcasting your location. But maybe potentially, it can leak the location of the item to other people, so somebody else can find it before you find it. 

But again, they don't-- and they're not coding this app. They're not creating it. But we'll run this in class, like, every few weeks. We'll just say, OK, it's Mission Feature Feature time. It takes 10 minutes. 

And it gets kids in the practice of quickly thinking through, what are these beneficial and harmful effects of all of these different technologies. And so let's just try this for fun here. I'm going to put a bunch of numbers on the screen. 

What I want you to do-- we're going to do a quick and dirty sort of random number generator. So pick one number. Focus on that number. Think about that number. Everybody pick one number. Keep it to yourself. 

And then, once everybody's got a number, you're going to-- on the next screen, you'll see a bunch of missions. The mission number-- that's your mission, OK? Here we go. All right. So just remember what your mission is. That's your mission. 

OK. Now, we're going to scramble it. So don't forget your mission, OK? You got your mission, everybody? All right. Now, you're going to pick two numbers. All right? Pick two numbers to focus on, and remember those two numbers. 

All right, you got two numbers? OK, here's two features. Pick the two features that match with your two numbers, in other words. And then, I want you to think about, what's an app that accomplishes your mission, using those two features. 

And I will say, if you want to pick a different feature, that's OK. If you want to pick three features, like, pick a third one on your own, that's OK, too. But it has to use at least two of these features and your mission. 

And I want to give you a little bit of time to think about it. And then, when you have an idea, go ahead, and if you go to the Gestures at the bottom of the screen, you can go ahead and pick Gestures. And raise your hand. I'd love to just quickly hear some ideas. 

What was your mission? What were your two features? And what does it do? All right, Jason, I'm going to-- let's see-- have you unmute. Let me see if I can-- I'm gonna unmute you, Jason. And what's your mission and your two features? What's it do? 

SPEAKER 1: I have to answer a question with a speaker and augmented reality. 

DOUGLAS KIANG: OK. Great. So my mission is answer a question, and the two features are speaker and augmented reality. Great. 

SPEAKER 1: So this is the app-- is that you point it at anything, and using your finger, you can, like, circle exactly what the object is. And it figures out exactly what color it is by RGB, and then, like, says it all out. It tells you exactly what the color is. 

DOUGLAS KIANG: Wow, that's awesome. I love that. And who would use this? Who would the audience be? 

SPEAKER 1: Maybe painters? If you're, like, trying to match a color at Home Depot. 

DOUGLAS KIANG: I love it. And then, for accessibility, it kind of reads it out loud, right? Uses the speaker to read out the colors? I love it. Thank you. That's awesome. Super. 

Amir, I'm going to unmute you. What's your mission and your two features? 

SPEAKER 2: OK. First of all, nice to meet you. Then, my mission was to teach and learn the-- I think it was the fourth number. And my two features were camera and accelerometer. 

So it's one of the most basic things, I think. For using the camera as we are doing right now, there would be an application, you know, like Zoom or any other application. People could join together and use the camera to talk with each other on. 

So we may use the microphone feature also. And there may be some sort of presentation, which as you are sharing, for example, the app could share the presentation. And by accelerometer-- for example, by shaking the phone, you could change the slide or something like that. It could be something like this. 

DOUGLAS KIANG: I love it. Thank you. Thank you. All right, and then one last one. [INAUDIBLE]? Yeah. What was your mission? What were your two features? Oh, sorry, let me mute you. OK. 

SPEAKER 3: Hi. OK, my mission was to answer a question, and my two features were using microphone and voice recognition, and calendar. So I think my app can ask you for a date, and uses the voice recognition and finds out which data you're looking for, and it looks at your Google Calendar, and finds out what things you have to do in that day, and gives you the schedule of your work and says, if you have any special date that day or not. 

DOUGLAS KIANG: I love it. Thank you. All right. Great. I want to-- in the interest of time, I'm going to move on and ask you to consider one more thing. These three additional cards-- so once we get kids in the habit of quickly pitching, here's what the app does, we want them to think about what's the beneficial effect on society, what are the data, privacy, storage, and security features that might come into play, and then what's the harmful effect on society. 

So these are Effect cards that you would apply to each one of these. Maybe kids can pick one at random, or you might give each kid a set of three and say, you need to do each of these three. So this is how I would adapt this to principles, the CS principles. 

We ended up-- it turns out it's really popular as a way to do app design. Because a lot of kids, when you ask them to do an app, they'll come up with an app that really is more like a web app. Like, it just pretty much works on a browser. 

If you're going to create a mobile app, typically, you want to push kids towards actually using the mobile features of the phone, which-- GPS, Bluetooth, that kind of thing. And so Mission Feature Feature has those other features, and it sort of-- we find that it helps loosen up the thinking, in terms of thinking about different kinds of aspects. 

OK. So great. So I want to go on and talk a little bit about how we handle late work with students. So basically, we always would have this problem where kids would not turn stuff in, and then they get farther and farther behind. And then, like, right before the grading deadline, they'll turn in, like, a dozen things that I've got to grade, and some of them are, like, four or five problems sets ago. 

We'd have problems where I'd want to go over the answer, but not all the kids have finished it. So by the time I wait for everybody to have turned it in, it's two or three weeks later. Everybody's forgotten, and they've kind of moved on already. 

So we had this problem. Other things that I would do is, like, I would take points away if they turned it in late, but then in my grade book, I'd have something that somebody turned in that was kind of lousy, and it was a B, and something else that somebody turned in that was a brilliant solution, but they turned it in five days late, so it was also a B. 

And so these are not equivalent, in terms of quality, but they both show up the same in my grade book. So we came up with a system that doesn't adversely affect the grade book. And the kids love it, because it makes it really clear about when things are due. 

So the idea is, every problem set has a deadline. Now, kids start the year with three tardies and three lates. And the difference is, a tardy is if it was due, say, on a Thursday at midnight, and you missed that deadline, you have one day to turn it in. 

All right? You have one school day. And if you turn it in by midnight on Friday, we take away one of your tardy, so now you have two left. But it's an automatic extension, and we will just grade it as if you turned it in on time. And you can do that three times during the semester, actually. 

And it's the next school day. So if it was due at midnight on Friday, they have until midnight on Monday to turn it in. So those are tardies. Now, if it's more than a day late, they can't use a tardy anymore. They have to use what we call a late. 

And a late is a one-week extension. They have a seven-day extension for a late. But late-- in order to qualify for a late, you actually have to meet me face-to-face. You have to meet with me, like, after class or before class, or on Zoom or in office hours, and we need to-- you need to explain. Why is it late? 

And the reason I'm doing that is because when-- before we ask kids to meet with us, if kids get really stuck and they can't finish it on time, and then they look at it a day later, and they're like, oh, hell, no, I still can't figure this out, then they let it go, and they go, I got a week. They'll wait six days, and then they'll look at it again, and they'll be in exactly the same place, and they'll still be stuck. 

And so it's just kicking the can down the road a week later. So what we ask them to do is, they have to meet with us and explain why. I'm stuck on-- I still don't understand how this function is supposed to work. 

Or-- and then, I'll say, OK, well, let's sit down, and I can explain this to you, and you can get unstuck. Or they'll say, I was traveling for a soccer tournament, and I know what to do, but I just need to sit down and do it. I'm going to get it in. 

Great, you don't need any help from me then. Just get it done. Your late is approved. But I have to approve the late before they can actually turn it in for credit seven days later. And that's actually a big incentive for kids to make it by the tardy deadline. [CHUCKLES] Because they hate coming in to talk to me-- for whatever reason. I don't know why. 

And so they get three first semester of each one. And they don't stack. In other words, you can't turn something in two days late and use two tardies. We thought of everything, because kids come up with all kinds of questions. 

I keep track of it in Canvas, basically, as a 3-point assignment. So if you use Canvas, I just create an Assignment called Tardies. I give it 3 points. I create an Assignment called Lates, give it 3 points. 

Every time they use one, I take one of them away, and I leave a comment on that assignment, saying, OK, this was late on this date for this reason. And it's just a quick way that I can keep track of how many tardies and lates people have. If students completely run out of tardies, then they will have to use a late. 

So if they miss the deadline, they're going to have to come talk with me, and then they have a seven-day extension, which they may or may not need. If students run out of all of their lates, then no assignment they turn in can be more than one day tardy. Because then, it won't count for credit. 

So kids have to manage that. You'll notice some of these students that started with 3, they end up with 4 or 5. Why is that? Well, what we realized is, we give kids extra credit for doing things like showing up for Puzzle Day, for helping be, like, a guest helper at an office hours session on Zoom. 

They can do all of these things during the year that will give them extra credit for. But again, I don't want to mess up my grade book, because a lot of kids see extra credit as a way to make up for something that they were supposed to have done but they didn't do. So say, OK, I didn't do the Filter problem set, but is there extra credit? 

No, you need to do the actual curriculum. Extra credit is extra. So what we do is, we say we're going to give you an extra tardy. And if it's really special, we'll give him an extra late. 

And what's interesting is, many of the kids who will do the extra things, like coming to Puzzle Day and getting involved-- they love getting an extra tardy, even though they never are going to use it, because they're always turning in their work on time anyway. But they like that recognition. 

And so it was kind of an epiphany for us to realize that a lot of the value of extra credit comes into recognition and not necessarily the points. But that's kind of how we do tardies and lates. But making it really clear has really been awesome. Because now, I don't have to worry about kids turning stuff in tardy anymore. 

And after seven days, you can't turn it in for credit anymore. So I'll still give them feedback, but it won't count. So we basically-- I don't have kids turning in anything that's more than a week late anymore, which is great. 

And usually, I only have a few people turning it in late. And pretty much, everybody's either taking a tardy on it, or they're turning it in on time. So this system works great. I actually have a link to it in the Resources folder. 

There's a whole summary. It's all written up there. So yeah, check it out, and try it out if it's something that you think might be useful in your own classroom. All right, so I want to talk about Auto-Commenters. 

So Auto-Commenters are basically these spreadsheets that we use. So we have TAs. And what' we'll have them use the Auto-Commenters for is to quickly go through and score a problem set. 

And the thing about the spreadsheet is, the spreadsheet will automatically generate comments for each, like, actual narrative comments that tell students in detail what they did right and what they could work on. And it used to be that I would keep track of these things in a separate folder, like in a text document, and just copy and paste all the time. 

But what I found is students are making a lot of the same errors, and I was looking for a lot of the same things. So I ended up basically just creating these spreadsheets that will kind of do it automatically. So we have a problem set that we created called Diamonds. 

Diamonds is very similar to Mario. The idea behind Diamonds is, you put in a number, and it draws a diamond of that shape. So if I put in 5, it's going to create a diamond, where the widest row is the middle row, and then it kind of goes down from there. 

If it's an even number, we need-- we ask them to repeat the middle row. This is an extra problem set. We actually use it in place of Mario, and it's in that Resources folder. So you can check it out there if this is something that interests you. 

But what we did is, we asked kids to do the Diamonds problem set. These are the things that we look for when we score it. And we pretty much use a binary sort of-- either they did, or they didn't do these things. 

And what we've done-- what I ask the-- so I ask the teaching assistants to look for these things. And when students turn things in, they turn it in on a site called Submit50. Or they use Submit50, and it submits it to a site that's located at submit.CS50.io. 

And that's where I manage all of my students' submissions. It's all run through GitHub, and they all show up in one place. And I go through, and I give kids feedback, based on what I see in the code. 

And I used to do this manually. But now, what we can do is, we share this spreadsheet with the teaching assistants. And let me-- I'm just pulling up my demo here. 

And for this, I'm going to go back to that CS50 Educators link that's got a bunch of stuff in it. And all of the auto-commenters that I've done so far are all here, and they're here for most of the problem sets, not all of them. I'm still working on getting all of them in there. 

But for instance, if you have something like Cache-- so if you open up Cache, I actually wrote guidelines for what we look for when we score these. And these are similar to, but they're not the same as, a lot of the grading guidelines that are on the TF site. Those are very valuable, too. 

And I put this out here just as a way for people to modify this as you see fit. It helps if you know a little bit about Excel or about how Google Sheets works. But again, these are the things that we're looking for. 

We want students to put their name on it. We want them to use variables instead of constants to represent the denominations. We want them to pass Check 50. 

And so you'll notice that I have a sheet for names. So these are the usernames-- the GitHub username, and the first and last name of each of my students. I go into Submit 50, and I download a CSV file that has the number of checks that there were, and the score that they got for Style 50. 

And this pulls it in automatically to construct a comment. So if you look here, these comments are actually written automatically, based on what students have done. And so I will go through to say, OK, I'm looking at this person, Mickey Mouse's work. His comments are nonexistent, so that's a negative 1. This one, I'll just hide. 

Did they use modulus and division? This person did not. They did define constants. They [INAUDIBLE] included a library that they didn't call any functions for. We don't really like that, so I'm putting in a 1 here. 

Did they use individual variables for the coins? So they declared dimes, nickels, pennies? Yeah, they did, so we'll put that in. And did they use a loop for the pennies? Yeah, they did. They didn't realize you could just add cents at the end. 

And overall, holistically, quality-- I'm going to say the quality here was a 1. Now, there are notes here about how you would grade each of these, how you would score these. And this whole comment is automatically written, based on what they've done. 

And it's basically concatenating these strings that come in from these stems. So for each one of these things, if they use the loop, if they didn't use a loop, I've already written it. You can certainly put it in your own language and write something for each of these. 

But what happens is, it all gets combined here. And so what I ask my TAs to do is, I just ask them to go through and just mark what they see. So if you look at Fahrenheit, this one's actually a good example, because I annotated this with instructions, such as they are. 

And so if you're interested, I invite you to make a copy of this and play around with it. Hopefully, it's something you can adapt for your own students. I actually have five sections in one spreadsheet. So I'll have all the students in there, and assign each one to different people. 

And in this annotated version, I've actually marked in green where you actually would be entering in ones or zeros, based on what you see. And again, the notes are here. So hopefully, this will save you a lot of time. It took me a lot of time to come up with these, to be honest. 

But you know, I'm putting them out there in the hopes that it's something that will be useful to you. And what we'll do-- we actually-- I'm going to talk a little bit more about this in a bit, but we actually give students scores for what we consider the big ideas of computing-- so abstraction, code fluency, their correctness and style, and then their documentation or comments. 

And I actually put this in Canvas. And then, we asked students to keep track of it in a separate Google Sheet that they shared with us. Because what we're interested in doing is looking at the trend of how students are doing in each of these areas. 

But again, the spreadsheets have totally made it possible for us to do all of this. Each of these is automatically calculated, based on those columns, whether kids did things or didn't do things that they were supposed to do. They get a four-point score for each of these. 

And so yeah, when I first started thinking about this, this idea was-- I spent a lot of time scoring, so why don't we write a spreadsheet that actually automates it? And I'm still working on it. And I'm actually spending probably more time tweaking it than I did scoring, so I'm not sure how much time it saved over the long run. 

But more recently, I'm spending less time scoring, which is nice. But I've actually managed to get the basics of something up there. I'm hoping, with our community here, that some of you will take this, and run with it, and do even more with it, and help jump in on this ongoing development. 

We can make something that will really be super helpful for everybody. I actually took this same idea, and I did this for my first quarter and third quarter comments. So when I give comments to students, it actually autofills from their scores for all the problem sets across the board. 

So yeah, so people are like, wow, your comments are really good. You spent so long writing them, and it actually is just, like, click and fill. [CHUCKLES] Anyway, so yeah, finally, I want to end with this idea of, if we're working towards mastery, what does mastery of computer science in CS50 AP look like for us? 

And so for us, we really went back to our core beliefs. And these are what we believe as a department about computer science. We think even though kids come in with some experience, it's really a skill that has to be learned over a period of time-- long period of time. 

And you have to take risks. And you have to be OK with failure and with a lot of red results in Check 50. We think that grades are really not a great way to measure this complex set of skills. 

We also think that students improve best when the feedback that they get is pretty quick. And that, honestly, was a place where I kind of fell down when I was trying to grade these things manually. It was taking me a week or 10 days to get students feedback. 

And by the time-- we work at such a pace-- kids are already almost done with the next problem, so by the time they were getting my comments, [INAUDIBLE] So that didn't work. So we wanted to give kids prompt, prompt feedback quickly, which is why we invented these spreadsheets that will quickly give feedback that we can go through and check. 

And finally, we noticed anytime we gave feedback and a grade, kids wouldn't read the feedback. They just look at the grade. So grades actually overshadow the feedback. For whatever reason, when the grade is there, they just kind of turn off on really reading carefully about, where do I get better. 

So we started with those core beliefs, and we said, OK, well, what's important for us, in terms of computer science? And these are things that we came up with. Basically, we want kids to reflect regularly. We want them to be resourceful. 

Code fluency is this idea of efficiency. We want them to be able to use functions and declare functions and break problems down into steps, so demonstrate a knowledge of abstraction in that way. And we want them to comment, not just the way a software engineer would comment, but comment as a form of metacognition. 

We want to know that they understand what they're doing when they come up with some long formula. That also helps with academic honesty as well. So those were the five things that we think are really important. 

And so I showed this to you a little bit earlier. Every student has this dashboard, and I have a master copy of that dashboard as an example in that Resources folder. Correctness and style are automatically assessed, so those automatically get filled in. 

But we want to know, how did they do for abstraction, code fluency, documentation? These three are the ones that we assess for most problem sets. And for others, we don't necessarily assess all of them. 

But what we're really looking at is, we want to see what are the trends. We also have kids-- every time we grade a problem set, we ask kids to enter those scores for each of those areas in their worksheet. And then, we ask them to insert a note that explains in their own words what they need to do to get a higher score next time-- or if they got a green, what did they do so well? 

And we try to encourage them not to frame it in the negative. In other words, don't do this, don't do that. We want to use positives like, define constants to eliminate magic numbers, or use an if-else to avoid doing an extra if check when it's not necessary-- those kinds of things. 

So we want them to put it in their own words, so that at a glance, when we conference with kids, we can say, you tell me. Which of these areas are you doing best in? Which of these areas are you improving in? 

And we remove the focus from just being on the last problem set, to looking at the trend across the year. What does learning look like over a period of time? And we try to provide lots of visual ways for kids to do it. 

This way, these cells actually color automatically, based on whether they put a 2, 3, or 4 in. We send this home with our report cards. We actually send home a radar plot that kind of shows, across all of these axes, how are they doing with respect to the rest of the class. 

We're trying to provide lots of different ways to visualize, how are they doing. So if we ask them, how are you doing in terms of your problem solving and thinking about computational thinking, we don't want them to say, I'm doing a B-plus. Because what does that even mean? 

So we want people to be able to say, well, I'm doing really well in terms of seeing places where I can use functions in abstraction. That's, like, my top. But I'm not doing so well in documenting my thinking, so I got to do better with comments. So we want kids to be able to articulate it at that level of detail. 

So these are the messages we tell parents. And this by the way, is something that we do in the spring. We actually hold a parents conference over Zoom, and it's optional. But parents who are seeing their kids struggle with CS50 throughout the year, and they want to come and ask questions-- we invite everybody to come to ask questions, but also to see their artwork, to see the work that the students have been doing, and see that we place a priority on students' social emotional wellness, on students collaborating and working with each other. 

These are the messages that we reinforce to students. And so we ask the parents to help us with reinforcing these same ideas. The first message we give them is, we spend a lot of time coding in class. I'll spend, like, up to, like, an hour, out of a 70-minute class, giving them time just to work in the problem sets at their tables with each other. 

But especially in the second semester, they have to do work outside of class. They cannot just do work in class. They have to be doing some work outside of class. And unlike a class that assigns daily homework, they've got a problem set that's due in a week. They have to do a little bit each day. And so we kind of try to emphasize that with kids and with the parents. 

The second one is to follow the 45-minute rule. The 45-minute rule basically is, if you've been stuck on the same problem or portion of a problem for 45 minutes, you need to stop. And either work on something else, or work on a different subject, or even just go to sleep, and wake up the next day and look at it again. 

Because-- you know, and I tell them, you're smart and you're resourceful. And if you've been stuck on this for 45 minutes, spending three hours on the same problem, you're probably not going to solve it. You need to change things up a little bit. Maybe talk to a TA or talk to me, but only spend 45 minutes on this, and then take a break. 

We remind them not to take ethical shortcuts. We actually do check everything. Everything kids turn in, we run through-- we use Moss and we use compare50. And we are concerned more with kids copying stuff off the web rather than copying from each other. 

If we see kids copying from each other, we'll usually have a talk with them and counsel them. But if they just wholesale copy and paste something that they found off the web, we will usually refer that to the academic deans. And every year, we have about 10% of the kids that do that. 

And I'm convinced we would have more if we did not emphasize so much with them what collaboration is, and provide lots of support for them to get help through office hours and through TFs, and through our online forums to get help. We also really cover time management, because a lot of the time, students are not cheating because they're fundamentally dishonest. They're cheating, because they're just stressed out, and out of time, and they haven't managed their time well. And so we're trying to help them with that. 

But for the majority of kids who do the work honestly, I see it as our responsibility to come down on the kids who are cheating and turning it in. Because I don't want to be the course where kids say, the easy way to pass this course is just to cheat, because they never check. 

And then, this is probably the hardest thing with our students. It's OK if it's not perfect. We have such a culture of perfectionism here. Kids will beat their heads against the same wall for three hours trying to be perfect. 

They'll be afraid to turn something in if it passes 19 out of 20 checks. We tell them, look, just turn it in. Turn it in, so I can give you feedback on your work. It's OK if it's not perfect. 

And that's really important to kind of give-- continually give that message. We just want you to do your best. And when the time is up, turn in what you got, and then move on to the next problem set. 

This is kind of our whole thing. This is kind of what this looks like for us. We start with CS1, which is CS50. Everybody has to take CS1. And then, from there, they have a choice. 

Once they finish a year of CS50, they can go on to do app design, which we teach in Swift. Or they can go on to our AP Computer Science A course, which is taught in Java, and it follows the AP Computer Science curriculum. From that, they can then go on to an advanced topics course, which is a rotating smorgasbord. Sometimes we do machine learning AIs, sometimes do web development, game design, that kind of thing. Or kids might choose to do independent study. 

We see kids following two branches. The kids who are pretty much just want to make stuff-- like, they're makers. They want to make apps. They want to change the world. They want to do stuff that has an immediate impact. 

Those kids, after CS50, will go on to take app design, and then off in an independent study to work on a project that is meaningful to them. Other students either want the AP credit, or they want a good, strong foundation in data structures. So they'll go on to take CS2, which is our AP course, and then they'll go on, and then they'll do advanced topics. 

And some kids who take app design, we pass on to advanced topics. But most of them go from app design to take CS2, and then on to advanced topics. Advanced topics is mostly taught in Python. So that is pretty much what I got. 

I do want to thank you for coming to this. I want to share all of those resources there, and encourage you to look at those. Hopefully, those will be helpful to you. I will pop my email in chat, so that if you have any questions at all throughout the year about anything, please feel free to contact me. 

And I am really excited. Again, like I said, the biggest benefit from me using CS50 as a curriculum has been the community that I have gotten along with it-- just, some really fantastic teachers who are doing incredibly creative things. And so I'd encourage us all to kind of reach out to all of us here in the community and take advantage of the vast amount of knowledge that we have here-- the global knowledge.