DAVID MALAN: All right welcome to the CS50x educator workshop. This is our session on grading programming assignments and handling academic honesty. We're joined today by CS50's own, Margaret Tanzoch, who's been teaching CS50 in her own classrooms for quite a few years and is here to share not only her experience, but also or advice on how to approach these challenges. 

Margaret, allow me to hand the floor to you. 

MARGARET TANZOCH: Hey, everybody. It's so nice to see you. As David mentioned-- thank you David, for that lovely introduction, I've been teaching 650 in high school for about five years. And what I'm going to go through today. I'm going to share with you what I do in terms of grading assignments and academic honesty. And I have to say, one of the things I love about the CS50 curriculum is that there's so much opportunity to kind of really run the classroom the way you want. There's no prescribed way that you have to do it. 

But let me just start. I'm going to share my screen. So to give you a little idea on my own background, in the high school I teach, called NEST+m New Explorations Into Technology, And Math, I have a lot of students. I teach generally about four sections of an intro computer science and one section of AP Computer Science principles. I have somewhere between 130 to 150 students. 

So again, the way I run my class might be a little different than if you just have a small group of students. I also teach CS50 at the Harvard Extension School and at Harvard Summer School. So I'm pretty familiar with a lot of the problem sets and the curriculum, as well. 

This is a picture of my high school. This was from a hackathon that we had last year, where David was kind enough to come and visit. We have, it's a K through 12 school. So although I usually teach juniors and seniors at my high school, we also invited middle schoolers. And everybody had an amazing time just using the day to have an extended time to really delve into some cool programming problems. So that's where I teach. 

So in terms of what we're going to talk about, hopefully, everybody had a chance pretty much to meet one of their neighbors. And by the way, I'm going to paste in, I think somebody's pasted it Google talk. I'm not sure if that's mine. I'll paste in the presentation that I'm doing, in case anybody wants to follow along. 

All right, so I'm going to talk a little bit about how I assign problems and what kind of deadlines I use, how I use the correctness and style scores, how I use design integrating process. And one little things that I do with my particular students, which are reflections. And then I'll talk a little bit about academic honesty and how I try to handle it, how I try to avoid that happening, and the culture that I create to kind of keep a lot of academic integrity in the classroom. 

And then there'll be plenty of opportunity for you to ask questions. I'm going to suggest that, as David had mentioned, if you do have a question, please raise your hands in the Raise hands option in the participants panel. And I'll pause and call at you at an appropriate time. 

All right, so starting are taking a look at grading programming assignments. Before we start, I want to talk a little bit about my own philosophy on grading. So in my high school, students take computer science as an elective. Many of them have no knowledge of what computer science is. Many of them are curious to get a sense of it to see if it's something they may want to major in in college or something they might have a talent for. Many of them feel a little bit nervous about coming into computer science because although they've used computers for so much of their life, what goes on inside that computer and how programs run is something that they have very, very little familiarity with. 

I don't really want students to feel afraid of taking my class because it's going to bring down their GPA. About 99 percent of the students at my school want to go on to college. And having a good GPA is very important to them. And so unlike in some colleges, students don't have the opportunity to take my class pass fail. So while I say this, I'm not a harsh grader. But the work is hard. I do give students a lot of opportunities to do well. I support them in their work. I allow them to collaborate. And I even offer like extra credit problems and writing assignments to try to even things out when students need a little bit of something extra to help out with their grade. 

Now I create a website for each of my classes. And I post weekly, pretty much, kind of the expectations for what we're working on that week. I post a program. Generally, one program a week, along with the program specs and tons of resources. So I'll post, for instance, the CS50 lecture. Even though I don't require that they watch the lecture, I will post it for them to have, if they choose to watch it. I will recommend that they watch it. I'll post many of the shorts that feature Doug Lloyd that delve in a little more detail into individual topics. 

I'll do a short mini lesson myself. And I'll give students most of the class time to work on their programming problems. I generally only assign writing problems for homework. Because again, some students, although the majority have computers and internet, I cannot assume that 100% of them do. So I want to keep things equitable. I don't want students to feel they have to stay up until 3:00 in the morning trying to debug a program, particularly they may have limited hours They could use their internet. They may have limited hours, they could use a home computer. 

So I give them lots of time to work on their problems in class. I'll allow them to work at their own pace. So I don't have harsh deadlines. I have suggested due dates. And students try very much to keep to those due dates, for the most part. But I do let students know that it's more important to me that they kind of work through the problem, that they understand what's going on with it. And maybe they do one less program or two less programs at the end of the marking period. Whereas I may give alternate assignment for some of the more difficult programs for students who need that than that try to kind of go crazy doing everything for homework. 

And again, we're in high school. So I try to keep it very reasonable amount of work. Students who finish early, and I do have some students that come into my class that already may have years of programming experience, or just really just fall in love with the material and end up working out the programs at home and working at a much faster pace than the typical student in the class. 

So for students who finish early, I allow them to become experts. I keep posters on the wall for each programming problem we're working on. And I allow those experts to help struggling students. In some schools, there are teaching Fellows, TFs or TAs, from some of the maybe past students or for students that have taken the course in previous years. We usually don't have an opportunity for that because my students are programmed pretty much for seven periods a day with only lunch as their break. 

So [INAUDIBLE] only one year ahead with TA. But in general, I depend on students in the classroom to help each other. I allow them to collaborate. And I will talk later about what that means to collaborate with academic integrity. So rather than have to have a program due and take points off if they can't get it in by the end of the week, I'll give students until the end of the unit to submit their programs. And for some students who are really struggling, who need some extra support, I'll even give them to the end of the marking period. 

So again, my students are a very diverse group. We're a considered a screen school, a gifted and talented school. Although we also have students with disabilities, students with special needs, IEP students. And the range of students in my classroom could be a student who is currently taking calculus as a senior, along with a student who failed algebra and is actually given to put in my class to get the math credit. 

So it's a very, very wide range of abilities, which is another reason why I really work carefully on creating a class culture where students understand that it's a process. It takes a while, sometimes, to get comfortable with program. And the mistakes are totally normal. It's part of the process. It's called debugging. 

I pose plenty of what CS50 calls some of the comfy versions of programs for extra credit for my advanced students. So my philosophy is that I never want anybody to get bored. There is always something for everybody to be doing at the level that you're able to handle it. And I do allow students to resubmit their programs as many times as they want to, up until pretty much the end of the say, for instance, the end of the unit, so that they can improve their grade. And the way that the tools work really make it very possible to do that. 

So I'll have my students submit all their programs, of course, using Submit 50. Submit 50, as you've seen in previous sessions this week, assign a corrected the score based on the Check50 for that problem. And also the style score based on indentation and spacing. Now you create whatever formula you want to create and get a grade for this. What I do is I use 75% of the correctness score and 25% of the style score. And that's how I assign grades. 

I actually automate it a little further by downloading, from these Submit50 Data [INAUDIBLE] platform, I'll download a CSV file and I'll run a little Python script to be able to calculate grades using this formula. And then I'll map students GitHub names to their student ID numbers and create a CSV file that I can upload to the school grade book. So that makes it a little easier for me to read grade programs, as well. 

The other thing that I do is that as I'm going through this, I'll take a look at some scores might sometimes look like a very low correctness score. And in that case, I do make sure to look at the code and tweak the score if the error was minor. Because sometimes, students will, at the last minute, try to fix the style on their code, even though the program is running and they seem to be passing all the correctness checks. But in the process of fixing the style, they might accidentally delete one of the forward slashes for a comment or possibly introduce a straight character into their code, which might cause the compilation to fail when the Check 50 is run during the Submit 50 process. 

So I will take a look at that because I knew what the grades to be fair. I'll usually mention it to the student if I could catch them and have them correct themselves. But if it's at the very last moment, there's no time, I will take that into account. And I'll adjust the grade on that. 

So this is an example. This is actually a real student that submitted Cash at the beginning of the school year last year. As you can see, the student submitted twice. And the first time they only got 4 out of 11 checks correctly and 74% of the style done correctly. And so a couple of weeks later, they were able to resubmit their program. They got a better Check 50 grade and a better style grade. And so the way that I assigned the grade for this student was to multiply that 10 over 11 by 75 plus the 90 times the 25 percent. So this student earned a 90.7 percent on this particular program. 

And I'll always use the highest score. So sometimes, the student may try to correct something. And occasionally, as they corrected, they may miss something else up. So I do have my Python script make sure to check it. And it'll only pick up the highest score when I'm creating the grades. 

Here's an example of the student, as well, that submitted Cash perfectly correctly. So as you can see, the check 50, since there were 11 checks for this program, the score came out to be 11 over 11. And the style came out to be 100%. And so the student earned 100% on this particular program. 

Now in terms of design, some teachers are very big on design. For my introductory students, for almost all of them, this is the first time they're doing any coding at all. I'll strictly-- and I also have a lot of them. I might have 100 or more introductory students. I'll use the correctness and the style score as a grade for that particular program. For my AP students, I will sometimes require additional criteria, depending on what the problem is. 

So for instance, when they are creating a program like Caesar or Vigenére, I do enforce that they use a function because functions are really important. A lot of my students do seem to struggle with functions. And so I will not give 100% of the program if they don't use a function. I'll make sure that they avoid any type of repetitive or unnecessary code, make sure they use good variable names. There always seems to be a student that wants to use like, Bob, as a variable name. So we'll have a discussion about that. And again, of course, I'll enforce that they use reasonable variable names and also comment their code. 

I do design feedback by looking at the students submissions and finding common less than optimal design choices that multiple students make. So for instance, from Cash this is the way that many students in my class have created the algorithm for the CASH problem set, right? 

So the idea here is you're trying to-- Cash is basically a greedy algorithm that's asking you to figure out the minimum number of coins that you can get from a certain number of change. So just to kind of open it up, what do you guys think? How could this design be improved? 

Justin, I see your hand's up would you like to respond to that? 

AUDIENCE: OK, I had raised my hand to ask a question. But I'm not prepped for this So sorry, I'll pass. 

MARGARET TANZOCH: Would you like to ask your question? 

AUDIENCE: Oh, sure, if that's appropriate. I would like to know-- because I heard you talk about how you manage the different level students and it sounds like a good strategy. And I heard David before mention to encourage students to just think about their progress when you start the class versus how to end it and so on. 

But there is a particular area that we struggle with here as a class discussion. And the students that are advanced tend to dominate because they are passionate about what they know and want to share it. And of course, there were other studies where like for example women, this particular about math, specifically, will do worse if men are just in the room. Or Malcolm Gladwell research that if you're a top student, you get more attention from the teacher and you do better. 

How do you manage that live discussion, the those students don't intimidate average and other students and everyone can work on their own progress? 

MARGARET TANZOCH: Well, I'll do different things. My students are organized in my classroom, at least before we work remotely, at tables. And so I'll often have students turn and talk with their small groups about a problem. And then I'll ask somebody from each table to share with that group talked about. 

And so in that way, hopefully, everybody has something that they can offer up, right? So even if a student didn't feel confident up front talking about maybe a design choice on a problem like this, in many cases, they got some information from somebody else at the table. And so they may feel a little bit more comfortable speaking up. 

The other thing is that I do fairly short lessons on a very small topic on a daily basis. So I might just do a little lesson on a for loop. And then I'll spend have the students spent most of their time doing coding during that period. And I'll walk around the room. I'll encourage them to work, sometimes, together, maybe have small groups sit together and psuedocode a problem before they start working on it. I'll talk to them individually. 

Yeah, I mean, that's an issue sometimes. I mean, there are some students that are very shy and quiet and just don't have a lot of confidence. And so I'll try to work with them and build up their confidence and really kind of stress how they could see how far they've come during the course of a few months in the class and certainly by the end of the first semester. 

I love to do-- there's an AP-- a cs50 AP program writing problem, actually, called, Analyze This, which I always do at the end of semester 1, where students are asked about their growth as a computer scientist. And I love to assign that and really encourage students to see for themselves how much they've learned during that particular semester, particularly students that often have no understanding of what coding even is to being able to complete programs like some of the cryptography programs, Caesar, Vigenére, and beyond. 

So it is a work in progress. I'm not going to say I have it perfect. But I do try to encourage a small group discussion. I'll also pull together small groups of students to sit around a table, if they're not quite competent, and to help them with their program, so that they feel a little bit confident, as well, in terms of understanding what the steps to solve a problem might be. 

So I hope that I answered your question. 

AUDIENCE: Another, sorry Margaret. Another question that came up in the chat from David [INAUDIBLE], is what is the average age of your students? 

MARGARET TANZOCH: My students tend to be juniors and seniors so they tend to be 16 and 17, for the most part. I have occasionally had one ninth grade student, who was 14 and [INAUDIBLE]. But mostly 16, 17, 18 something in that range. Good question, thank you. 

Other questions. I see a number of hands raised. [INAUDIBLE], do you have a question? 

AUDIENCE: Oh, no. I just to answer the question how the design can be improved. We can use some mathematics. Yeah, we can use integer division with remainder. And then it's very simple. We just divide by 25 [INAUDIBLE] remainder. And the integer quotient is the number of coins we will get. And then we pass to the next one, the remainder. 

So it will be much faster. And I implement it actually both versions with this while loop. And with integer remainder, integer remainder is so much faster. 

MARGARET TANZOCH: Excellent. that's excellent, great. What about this? If we even just looked at this code, if indeed we said, well, while loops are it acceptable approach. This is a new group of students, right? So what else if we were to use while loops might be something we could improve here? Anybody have any thoughts on that? 

[INAUDIBLE], are you raising your hand for this? Or did you have a different question? 

AUDIENCE: [INAUDIBLE] here from India. 

MARGARET TANZOCH: Did you want to respond to the design choices here? 

AUDIENCE: Yeah, a thought [INAUDIBLE]. We can have a one way loop that we can have a [INAUDIBLE] greater than or equal to 1. And all this can be placed in a speed statement. There I can say the percent is 25. Then we can have cents equal to cents minus 45. Similar for 10, 5, and 1. And I would say this [INAUDIBLE] statement, we can have clients equal to plus 1. Because it is common for all that while loop. So one while loop and one switch statement can work for this. 

MARGARET TANZOCH: OK, all right, thank you. Anybody else have anything to add? Is that [INAUDIBLE]? I'm not sure if I'm pronouncing that correctly. I apologize if I'm not. 

AUDIENCE: [INAUDIBLE] is my name, from Germany. Now I'm thinking have you talked about [INAUDIBLE]. So instead of saying greater than 25 and then cents minus 25, like x and an x, in both cases. So it doesn't matter what number is in there. And ask for input of the number that is used in this while statement. 

MARGARET TANZOCH: I'm not 100% sure I understand your comment that instead of the variable names cents, we would use-- 

AUDIENCE: No, instead of the 25, which is a fixed number which can be used only once. Instead of the 25, put the variable in and fill that with an input that is taken from the user. And then you can use it the while loop for every number that is given in. 

MARGARET TANZOCH: Got it, right. No, absolutely. That's another great approach. What about this while cents is greater than or equal to 1. Anybody have any thoughts about that part? 

AUDIENCE: Everybody, I just see repetitive tasks which can be used to the full function here. We can just create a function and substitute the number, which we should have substriked to do the function variable. 

MARGARET TANZOCH: OK, Ignatio? 

AUDIENCE: Well, I was talking about was functioning too but just about this sense equals 1. At the [INAUDIBLE] you just have the number of coins of 1 cents. So you'll not be in this loop. 

MARGARET TANZOCH: Exactly, this while loop at the end really is not necessary at all, right? Because as Ignatio said, that whatever number of cents is left is going to be the number of coins that are left, right? Because after you've taken out everything up to your nickels, the only thing you have left are pennies. 

So many different ways for improvement, maybe even using magic numbers instead of hard coding these values 25, 10, 5, and 1. So many different things to discuss about this. And so I will encourage those design choices to be more of a class discussion than enforcing that in the grading policy, for the most part. 

I also may have quizzes during the course of the year that ask students to improve on the design. So those are the ways that I will discuss, have those design discussions. Thank you for all of your suggestions. 

Another thing I do is that I have students do reflections after each programming problem. And I find this, for my class, to be a really important part of my own assessment in terms of what a student really understands. So a reflection would necessarily discuss their development process, perhaps an aha moment or a struggle that they've encountered while they were creating their program. 

I'll usually have a very specific question that asks them to explain a specific algorithm in that program. And in some cases, perhaps have them collaborate. If they were collaborating on that program, have them discuss that particular collaboration. And this really also helps reveal to me if some students maybe collaborated a little too much, maybe doesn't have a real understanding. And allows me to kind of zoom in and then have more clear discussion with that particular student. 

Semester grades, I use a combination of things. We use a grade book called IO classroom. It used to be called Schedula. And we have various different categories that are weighted. So for instance, the scores on the programming problems from the check50 and the style50 are one component of it. I'll also assign writing programs throughout the year. And those are the types of problems that I would tend to give for homework because it's not a problem that students need to struggle debugging in the middle of the night. 

And the writing programs, the cs50 AP curriculum does have some writing programs. And sometimes, I will have students write a short essay on some type of occurred event. So sometimes, I'll even have these essays substitute for maybe a more difficult problem that a student doesn't feel competent in solving, as an alternate assessment. 

So for instance, this year with the coronavirus as such a major topic in the news, I had students write a short essay on computer simulation and modeling and used a little bit of a discussion about the virus and how it replicates to kind of introduce that particular topic. 

So I like the idea of writing assignments to also bring what we're doing in class into a real world environment. And that allows us to have more timely discussions on various different topics. Also grade on classwork. And that would be partly involved in class discussion, how focused, engaged students are. And also allowing and knowing that some students are really shy you don't have the confidence. So I take them to account, as well. I do do quizzes throughout the year. And I'll be talking a little more about that tomorrow. And then if reflections, as I mentioned, which I find to be a really important part of students reflecting on their own program development and also informing me on what the development looks like. Also, it just helps me know were the problems too easy or too hard for the students, kind of where we are in assessing that. 

So I'd like to take a little break here. And let's take some more questions. 

Ahmad, do you have a question? 

AUDIENCE: Yes, Margaret. Thank you. I wanted to ask you about something regarding the grading or the comments, which is peer review. Have you tried? I remember that you said that you teach also at Harvard Summer School, maybe at the university level. Have you tried giving students the opportunity to peer review solutions from others students to maybe grading them? 

MARGARET TANZOCH: I think that's an excellent suggestion. I've done a little of that. I would like to do more of that. One of the reasons I haven't done that as much as I'd like is that students tend to finish the problems at different times because I have such a wide assortment of abilities in the classroom. 

And so some students will finish a program a day. Another student might take two weeks and really be struggling through it and working on that. So I haven't-- I do that occasionally. I have students, certainly, at the end of the year when I have them do final projects, I have them write up a proposal that they review each other's proposals. And they give each other suggestions, how to make it better, how to make it clearer. And I find that's a really great process to go through. 

I do have students sometimes work together on their problems, particularly if they're working at the same level. So I may have two students that are struggling a little bit. And I'll allow them to both develop. So their code will look the same at the end, but I know they've worked it through together. And so I'm OK with that. So they will kind of go back and forth. I'll see the two of them working together. And that can be really productive, as well. And then they will discuss each other's code. 

So I think there's a lot of ideas that could be implemented, depending on the profile of your classroom. So but that's a great question. 

[INAUDIBLE] I'm sorry if I mispronounced that. Do you have a question? 

AUDIENCE: Thank you, Professor Margaret. To shortly, the first one, do practice to share the best projects in class? 

MARGARET TANZOCH: I'm sorry. Could you say that again? 

AUDIENCE: Do you practice to share the project the best projects in class? 

MARGARET TANZOCH: Sometimes, I do. It depends on the class. I do that a little more in my AP class, where students are a little more at that point. So my AP students have generally already had computer science for a year with me. So they're a little more aware of what design means. And they're a little more aware of good choices. 

So I will do that sometimes with my AP students. We'll take a look at a really optimal solution to a problem and kind of walk through it together. I do like to do that. But for some of my introductory students, again, I'm, at this point, if they could get the program to work, that's a real success. And so I'm not always looking at the optimal solution yet. For AP students, we sometimes do do that. 

AUDIENCE: Do practice to share summaries after a number of lecture or seminars or labs in class? 

MARGARET TANZOCH: Again, sometimes, I do and sometimes, I don't. For my introductory students, because so many students are working at their own pace. I'll usually work with small groups of students. So I may not do a whole class. If I have a class of 34 students, sometimes, it's hard to keep everybody's attention, when everybody's not exactly at the same level. 

So what I'll usually do is, in that case, I'll work with small groups of students who are all working at the same level. And we'll go through it together. And so I find that that's a little easier to keep their attention, if I have four or five students around a table, and we go through it together, than if I'm lecturing at the front of the classroom and have 30 students kind of trying to pay attention to an overview, reviewing the coding a particular problem. 

So I will do that from time to time. And I usually do that with small groups of students after they've completed the event. 

AUDIENCE: And what about late submission? 

MARGARET TANZOCH: So again, because I have such a wide range of students. And I am-- as I talk about academic honesty. I'll talk a little about that, as well. I generally don't take off. I'm pretty flexible in terms of submissions because I know that students sometimes have five other AP classes they may be taking. They may be working a part time job to help support their family. They may have other family responsibilities, such as picking up siblings from school or babysitting. 

So I will let them usually submit with no lateness until the end of the marking period. Or not the end of the marking period, the end of the unit. And then after that, unless there's a special excuse, I'll usually take off a certain amount of credit. I'll still allow them to submit it. And they'll get some credit. But I will take off some credit, possibly five points for the first few days. It's like 10 points after that. It kind of depends year to year. Year to year I tend to have a different profile of students. So kind of, again, depending on the needs the students or the particular population I have, I kind of customize my policy to really work, to fit the particular students that I have. 

Some students, as well, some teachers, as we'll talk about, have very strict lateness policies. And it's certainly easier to keep the whole class going at the same pace when you're doing that. And then it does become easier to go over a problem solution because everybody is already finished with it at that point. So some teachers will have very specific tardies, like you can have. And I'll talk about that a little later, as well. You could have an extra week to submit something maybe once. And after that, you need to come to me to get permission to do that, something along those lines. And they'll have much more specific policies on that. 

But then again, given that my students are so all over the place, in terms of abilities. And I'm really stressing that I want them to do their own work on the problem. I want them to get a real understanding. Knowing that some students will really struggle with the material, I prefer to give them extra time to do it on their own than to feel pressured to copy a solution from somebody else. 

AUDIENCE: Oh, no. Thank you. Thank you very much. I really appreciate it. 

MARGARET TANZOCH: OK, other questions? Thiago? Did you have your hand up? 

AUDIENCE: You've talked a little bit about student engagement and how they will interact with classes. And [INAUDIBLE] about what types of exercises should they do like at home and by themselves? I'm sorry. And what types of kinds of exercises they should do at class and with the help of the teacher or professor? 

MARGARET TANZOCH: Well, I guess for my high school students, I generally don't require that they work on their programming problems for homework. And again, the reason is that some students don't have high quality Wi-Fi. They may have limited Wi-Fi minutes that they could use. And I know that debugging a programming problem can sometimes-- you can be stuck on it and take hours and hours and hours. So I don't generally require students to work on their programming problems for homework. I give them class time to do that with a lot of support. 

The kinds of things I would ask them to do more for homework might be a writing problem. So it's something that they usually can do in a more limited amount of time, maybe do a little research, or something like that. So I'll often have them do writing problems. And for students again, I will post the lecture videos. And I will suggest if it's possible that they watch certain chapters of that lecture video for homework, if they have the ability to do that. And the ones that do usually do find them extremely beneficial. 

Ignatio, did you have another question? 

AUDIENCE: Just about collaboration between the students. Have them some kind of collaboration or [INAUDIBLE] give them suggestions [INAUDIBLE] the student collaborate between themselves. And if yes, what kind of software environments you can use or suggest to use this approach. Because I was thinking like how researchers work with papers. I write my paper. I submit to our colleges that evaluate my work and give me back [INAUDIBLE]. 

I was thinking if that is some best way to [INAUDIBLE] in your class. 

MARGARET TANZOCH: What software do I are use for submitting assignments or returning assignments? 

AUDIENCE: [INAUDIBLE] kind of platform that you can use when a student submits [INAUDIBLE] and one, [INAUDIBLE] other student [INAUDIBLE] and suggest change and give back to the first student. And [INAUDIBLE] has a chance to improve their own answer. For. 

MARGARET TANZOCH: So for the programming problems, I do use the Submit50 for pretty much everything. And what I'll do is I will meet individually. I'll walk around and I'll look at their submissions. And I'll usually do one on one conversations with students about their particular codes. 

For some of my AP students, what I might do is then, I might print out what they've said that it's me electronically and then make comments on it, either handwritten or sometimes I might, if it's stories or Google Doc, in terms of some of the writing assignments, I might make comments and then also try to meet with them one on one to go over what those comments are and how they could improve their work. 

So right now, I do find that, for a lot of my students, having a one on one conversations tends to be a little bit more effective than using the comments and get help, though. We'll see next year if we start out remotely, I'll be looking for other electronic solutions. So Google classroom is something that we used when we went remote toward the end of the year. And so I did give students a lot of feedback on Google classroom, when they submitted a program or a project, as well. 

So they submit their programming projects the CS50, right? But then any writing assignments or sometimes even feedback for other programs, I would give them that feedback back on Google classroom. So really, quite a number of ways. 

But the ones that usually I find work best are when I would do like one on one meetings with students. [INAUDIBLE] 

AUDIENCE: Do you use any additional resources to supplement learning? Or do you find that everything from the cs50 is sufficient, even for students struggling? Are there an external sort of resources, books, that you might use to help learning? 

MARGARET TANZOCH: Well, what I'm going to show tomorrow is that I create a lot of labs for my students using the cs50 lab platform. And in the labs, which I'm going to go through tomorrow in my discussion on the assessments in labs, I have a lot of detailed instructions on how to do very specific constructs in programming. 

So for instance, I'll have, in lab, some students will be working on a program and they forget how to use command line arguments. They've seen the mini lesson. They've seen me do a little demo of it. But once they're working on their program, they forget or they don't totally understand how it works. So I'll have them do a lab. And the lab will have detailed instructions with a little practice problem on the right. 

And so for students that I'm trying to encourage to be self-sufficient, or because I may have a lot of students in the classroom and I can't help them with that very specific thing at that very specific time, I do find the labs can be really helpful for students to have the extra support. 

I don't require them to buy books. I do allow them to do research on the internet, as long as they're not looking up a solution. And sometimes, there's a little confusion on what that means, which I'll be talking about a little bit. But I do encourage them to use resources, the cs50 manual, depending on what programming language they're working on. If they're working on the second semester HTML or CSS, I'll have them use W3 schools. I'll introduce them to those resources. 

So the resources are pretty much online. And I'll try to find videos, sometimes, when I can. For my AP class, sometimes there are nice videos from code.org that talk about how the internet works and things like that. So sometimes, I'll introduce those. And even in my lesson, sometimes I'll try to use a video every now and then to be a little bit engaging, instead of being up there and talking too much in front of the class, which I find I can only do for a very limited amount of time, if I want to keep everybody's attention. 

So what I'm going to talk about next is handling academic honesty. So again, many different teachers do this in many different ways. What we do is at the beginning of the year, we'll read through the academic honesty policy that cs50 provides in their curriculum and in their program specs. And we actually have a class contract that students and their parents have to sign that also discuss the academic honesty policy. 

Several times during the course of the year, we'll talk about how students can collaborate and help each other while maintaining academic integrity. And some of the ways that might look, again, kind of consistent what you might have seen in the written part of the academic policy is that, if a student is a class expert, and they're helping somebody else, and they've already completed the program, that it wouldn't be academically honest for a student that's completed their code to show their finished code to a student who is struggling. So that's forbidden. 

I tell students, if they're helping another student, they're not allowed to put their hands on that other students keyboard. Because I've seen sometimes students tempted to kind of, oh, let me just type this in. But the student who's struggling has no idea what that student did. So they're not allowed to type on the other students keyboard. Everybody has to have hands on and work on their own keyboards. 

In terms of collaboration, again, students that collaborate, I make sure they're at the same level. I don't want to have students collaborating one that's so much more advanced or working with a student that's really struggling because then that collaboration often ends up going one way. So I try to have students, when they are collaborating, pairing up with another student that's pretty much at their same level and encourage them to kind of work out the solution together, think up the steps, working in pseudocode, maybe putting pseudocode the comments first, and then kind of working back and forth with each other, each in their own IDE, in most cases, to solve that particular problem. 

During the course of the year, I will do one or two pair programming projects. I don't usually do it until the second semester. And the reason for that is that, my concern is that, if students are paired programming when they're first learning how to manage using the coding environment, the text editor, the terminal that it's very possible that one student may end up not getting all that much out of it. So at the beginning of the year, I do want students to all be working in their own environments, submitting their own work. And again, with support when they need that. And really kind of stress alongside with that that it's really important that students struggle through the problems, if they need to. That struggle is how we learn. 

We need to have persistence. That failure is common. That there is so much failure when we're programming, we have a special word for it called debugging. That it's the nature of the way coding works. And even when I do all my demonstrations, I will usually make errors, sometimes on purpose, and sometimes just because I make errors. I forget my semicolon. And kind of see how students kind of understand the process programming is one that's often trial and error. 

We'll talk about the difference between using the internet for references and using the internet for outright solutions. And again, the reflection that students have to submit at the end of their program does give me a good window into how much of the code they actually developed on their own. 

Every now and then I'll get a reflection that looks like I finished a program. It was really easy. I had a problem, a friend helped me fix it, and now it's done. So that's usually a clue as to that that student really had no understanding of what went on with that problem. And then I'll go have a one to one conversation with them about that. 

So in order to try to avoid academic honesty violations, as I just mentioned, I really do try to stress that, if students don't put it the work to solve their own problems, if they're depending too much on help or from their collaborative partner, that they're really going to get nothing out of the class. And that, as we go into semester two, if they don't understand the basics and semester one, by the time we get to final projects, there's a pretty good chance they're going to be totally lost. 

So I stress it over and over, which is again one of the reasons why I will give students extra time. I'll let them know that I much prefer that they have the extra time to work through the problem on their own or with my support, where I'll be explaining as we maybe type in code together. For students that really struggle, we might type in some code together as I explain it and we write in comments. And then I have them explain to me what that code does. 

So my goal here is that students are learning from something from the class. Whether they learn it totally on their own because they could just pick it up, watch the shorts, watch the video, they got it. They got it. They're good. Or if it's through kind of an explanation for me as we go through it. My goal is that everybody comes out of the class having some understanding of what it looks like to program in the real world. 

And one of the things I love about cs50 is that we have an environment to code in with a real looking terminal, so that it is a real world look at what it looks like to actually create a program. And I think that students really appreciate that. 

But even though I go through all these things occasionally, there is a student that just doesn't really have the confidence to start a program on their own. My classroom is set up where we have about 30 computers around the outside of the room. The computers are literally two inches from each other. And students will sometimes just feel very intimidated about understanding where to even start typing in and might have their neighbor help them a little bit, have a student expert help them a little bit, to the point that they are not really doing the code on their own. 

And again, that's one of those things that the reflection will usually be reveal to me. I've occasionally had a student who thought they did a good thing by finding a live coding tutorial on YouTube because it taught them how to solve a problem. And I'll usually have a discussion with them after that. Because they are feeling, maybe, they didn't just find a get help repository. They didn't just find the code. Plus they, actually typed it in themselves. But we try to have these one on one conversations about that, when that happens. 

Ways that I sometimes discover academic honesty violations is that, in some cases, I might find a submission that solves an old version of a cs50 problem. So I'm lucky in the fact that I've taught this curriculum for five years. So sometimes, I'll notice that a student tries to solve Mario by looking at the height of the pyramid is between 1 and 23, which was the way the program was constructed a few years back. Now it's between 1 and 8. So that automatically raises a red flag. 

Or perhaps they start a pyramid with two hash tags on top instead of one, which is also an older version of the problem. Sometimes, I'll see students do their props with a Print f before they use a user input function, like getInt or getString, which again, is an older way that cs50 problems were solved because using a prompt inside the parentheses as an argument in one of the user input functions has only been around for about the last two years. 

So there are things, sometimes, that I'll see that I'll recognize are old ways of doing it, in which case we'll have that discussion. I also walk around the classroom. One of the reasons I like students programming in the classroom is that, honestly, at home, I have no idea if they have a parent who's a programmer. Many of them do. Not many, some of them do. Or if they have a friend that's willing to collaborate a little too much with them. I like to walk around as they're doing the coding and just randomly ask students to explain to me what they're working on, what they're doing, line by line, what their code is doing. 

Sometimes I'll notice in-code strange indentations that might indicate some kind of a copy and paste job. And that's also just a little bit of an alert. Again, as I've mentioned a few times, noticing that the reflection provides absolutely no detail about how their process work, what an aha moment was, or what kind of struggles they had, or how the algorithm operated, usually provides insight to me that they probably weren't very academically honest on that problem. 

Sometimes, I'll paste some suspicious looking code into a Google search. And I'll immediately see that whole solution come up somewhere on the internet. Some teachers do use Compare50. And I think it's a terrific tool. I have not used it up until now, for various reasons. I may try it out next year. I usually get so caught up kind of designing my class that I sometimes run out of time to try to use all the tools. And also, at the beginning of the school year, and since I do allow some collaboration, a lot of the programming solutions are going to look very similar. 

So again, I depend a little more on the reflections and on the personal relationships with students. But I think Compare50 is a great tool. And I know some teachers use it very, very effectively. 

I also know that some teachers will use the file revision history in the IDE to see if an entire program is pasted in like one step. And that's a great thing that you can see in the final revision history. Some cs50 teachers will have all of their students share their workspaces with them. And then they'll go through each of them and see how they indeed develop their program. Again, if I had a small class of students, I think that would be a great solution. But I just can't have 150 students share their workspace with me. And they get a sense of how to move forward with that. 

So what I will use file revision history for is, sometimes if I'm suspecting that a student copied and pasted code, I'll go over them in class. And we'll take a look at their file revision history together. And that might be very revealing. Usually, they'll try to have an excuse like, well, I developed my program in a Google doc first. And then I pasted it in the IDE. It's like, hmm, that doesn't sound like a very good way to go. 

Right, so what do I do when I find academic honesty violations? So I'll have a one on one conversation with the student. And honestly, in most cases, they'll usually admit it. If it's an introductory student, for their first offense, I don't necessarily tell them this at the beginning of the school year. But for the first offense, I'll often have them delete their entire program and write the whole program again on their own, or maybe with my help. And I'll watch them. I'll kind of keep a special eye on them as they're working on it. And in almost every case, when that happens, the code that they develop themselves is usually of code that looks like maybe simpler code, which I would expect that student to be developing the beginning of the year. 

Sometimes, I'll also just identify that a student uses very sophisticated code, or some kind of programming trick that we haven't even come close to talking about, like pointers for strings or things like that instead of a string data type, which usually indicates to me that it's probably not their code. 

If it's an AP student who I've already had in my class for a year, or if it's a second offense, I'll give them a 0 on that problem. And that if it ever happens again, I would call home and have a meeting with their student and their parents and an administrator. But honestly, after the student gets a 0 for the first time, it pretty much never really happens again. 

And I've also found that, in most cases, in having the discussion with a student, there's usually some underlying challenge that they're facing, which is why they just feel pressured to get a little too much help for their program. Again, which is why I don't have strict deadlines. I try to keep it open, so that students have the time to work on it and to actually develop their own code. 

AUDIENCE: Margaret? Sorry, my mic was open so maybe I was encouraged to ask a question. 

MARGARET TANZOCH: No, please do. 

AUDIENCE: Yeah, and I see, because you have high school students and you're in a formal setting and these things are critical. And they are also at the age where you want to teach them work ethic. For our students, they're a little older. And they're also self motivated. They really want to learn. And they have a project that they want to build later. 

So it's mostly about the skills for them. So at what point it reaches diminishing returns? By that, I mean, like if you're going to struggle with a problem trying to solve it, is it two hours, where you need to use another skill and learn how to phrase your question and ask for a code review on a form? Because I think that that's a great skill. When everything else fails, you need to know how to ask questions and how to build a community, also answer questions. That way, people respond to you really well. And your writing is really good. And you get this goodwill within the community. 

So that's one. And then for like Python, we use cheat sheets. So is there anything like that is useful for high school students, where they have a cheat that gives them the most used things. And it would have them. And then, at some point, they memorize it and not need it anymore. 

MARGARET TANZOCH: Yeah, I do create like a syntax sheet that I hand out or I sometimes just do it electronically now. I also have anchor charts all over the classroom with all of those, like here is a for loop. Here is a while loop. Here is how to use arguments and that type of thing. So I will have those kind of references available both on the website as a syntax reminder sheet, as well as anchor charts all over the classroom. Because I don't necessarily expect students to just automatically memorize syntax. 

The idea that I'm really pushing for is to memorize ideas-- not to memorize, but to understand ideas in programming, computational thinking, right? To understand how to break down a problem into steps and to think about how to solve it. So I do provide them plenty of assistance on syntax. And even when I give quizzes, I will allow them to use a syntax sheet and a page of notes that they've created on their own, which they can then use to help them with that quiz. 

And that page of notes actually is like a study guide, one page, two sides, has to be handwritten. It can't be printed out from online. They have to write it. The process of creating that is actually a great learning experience for them. So I will allow them to do that, as well. 

So yeah, I mean it's again, I have students of such a wide range of abilities. And I want to make it possible for everybody to, at the very least pass my class, if not do well, if they put in the effort. So what I'll do with students that really struggle is that I will hopefully have enough resources out there that most students can work on their own, or with an expert. And I'll pull together maybe four or five students at a table in the middle. And we'll work out some code together. 

And we'll literally go through the first step. Let's create a do while loop to validate user input. How do we do that? And I'll ask them, how do we start it? How do we write that? What's the condition going to be? And we'll do that together, interactively. And I'll type it up and project that on the board. And then they'll type that, usually, at the hub of the school laptops they'll grab. And they'll sit around the table and type that in with me. 

And so the goal, again, is to really hold their hands to support them when they really need it. And I also do walkthroughs on problems that go beyond kind of cs50 walkthrough that give them a lot more structure. And we might work out pseudocode together for a problem, so that students that struggle know where to get started. So I do try to provide a lot of support. 

So I see some more questions. Patricia? 

AUDIENCE: Hi, thank you, Margaret, for your time, for your presentation. I am just wondering if you can please explain to us a little how you organize your semester. Because you're explaining about how you grade and how you would handle some situations. But I am just wondering, when you have so many students, how many hours do you have in classes during the semester? If you have some labs, if you can tell us a little bit, so we can imagine could you apply the things we are discussing now. Thanks. 

MARGARET TANZOCH: Sure, no. I appreciate that. Thank you. It's a great question. 

So I meet my students five days a week for 45 minutes a day. There's no special section that's a lab. It's just a 45 minute class, five times a week. And what I do is that I do a very short-- I would always have a warm up on the board when they come in. And that might be on various different topics. So that they come in, they pull out their notebooks. And they write with pen and pencils. They don't start out with their computers. We have tables in the middle of a class. And the computers are around the outside of the classroom that they move to later on. 

So we'll start out with a problem. It might be the beginning of the year to just convert a binary number, maybe to refresh a concept from the day before. Sometimes, it's a discussion question, like talk to your partner about how we can collaborate with academic integrity. It might be a little code on the board. Determine what this code is doing, line by line. I like to give them snippets of code to see what code and good well-structured code looks like, that they have to interpret. So it might be something like that. 

So I'll always start out with like about a 5 minute warm up. And then I'll do a very short mini lesson and sometimes, a little demonstration of that particular topic. I keep it short. So my mini lesson probably never goes more than 10 minutes. And I might, for instance, talk about a loop, like why do we use a while loop, right? And how does it work. And then I might write a little bit of code and kind of interact because I think it is helpful for students to see actual life coding. 

They don't code with me because their computers are facing the outside of the room. When I was teaching remotely on Zoom, I did have students actually code along with me, which I actually really like. But I can't really do that in the classroom because there's no way I could get their attention when they're facing each other and facing the back of the room. So I kind of take that into consideration. 

So I'll do a short code along. And I honestly don't go on for too long because I'll lose most students. Their attention spans are not really that long. So I usually try to give them between the lesson and the code along, maybe 15 minutes tops. And then I'll have them spend about a 1/2 hour, 20 minutes to a 1/2 hour on the computers, working on their program. 

And during that time, what they'll do-- and I'll introduce them to the program spec. I'll show them where some of the resources are. And some of them will be sitting around the classroom, maybe watching the video walkthrough for that program. Or they might be sitting and they may be watching one of Brian's walkthroughs. Or they might be watching one of Doug's shorts. Or they might be watching a chapter of the lecture video. Some of them might be starting with their coding. Some of them might be sitting together with another student, working at pseudocode in advance. 

So it's a very diversified group and a lot of differentiation in terms of what students need. And yeah, then at the end of the class, I'll sometimes stop five minutes before the end. It kind of depends on how much time we have for coding that day. I might ask them to have a little share out, what did you accomplish today? Sometimes, I'll have them write a little reflection on a Post-It. And then that might go up on the poster that I have to correspond to that programming problem, so that I can also see what questions they have that maybe they need some extra help with. And then the next day I might ask one of the experts for that program to go over and help that particular student. 

And then I think, also, students see that other students have questions, too, that it's normal to have questions. It's normal to get stuck on stuff, that this is the part of the process. This is how it goes for everybody. So that they don't feel like, oh, I compiled my code and. It didn't work. I just can't do this. So I try very hard to make it a real culture community, where students could help each other. But again, constantly stressing that you're helping each other with academic integrity. I hope that answered your question. 

All right. 

AUDIENCE: Hi, Professor, I really appreciate your presentation. You are so clear. But what about any complaint after you have released scores to students? How do you proceed, based on your experience, please? 

MARGARET TANZOCH: If they complain about their scores? Well, in terms of the cs50, the Submit50, they can see very clearly what their score is and why they got it, right? They could look at the Check50. They could look at the Style50. So their program score is based on that. The rest of the score depends on the reflection. And so if they're not happy about the score with that reflection, I will have a one on one discussion with them that they have to provide more detail, or they didn't answer a question. 

So for the most part, students are not shocked at their scores. Sometimes, students will submit stuff, and they don't really check the score they get. And then a couple weeks later, it's like, oh, my gosh, I got such a bad. Score could I prove my grade? So it kind of depends where we are in terms of the unit and the marking period. Sometimes, I'll allow them. I will usually allow them to resubmit it if it's still within the unit. And they are allowed to improve their grade. And if it's already at the end of the marking period, it is what it is. 

Sometimes, I will allow them to do extra credit problems, like a writing assignment, as I mentioned earlier, on a particular topic that's relevant to us at that moment. And then I will maybe substitute that to their lowest score on their programming assignment. Because some students are going to struggle with it. Everybody is not going to get 100% on every program. But again, for students that are willing to put in the work and are willing to put in the effort, I will try to come up with a way of helping them improve their grade. And for the most part, it works out. 

Most students get either an A or B in the class. I don't have a whole lot of students that struggle to pass. I hope that answered your question. 

AUDIENCE: Can I ask questions now? 

In the previous line, you said that you use file revision history to check. I didn't quite understand how can we do that. Well, what kind of tool to use. Is it submit.cs50.io? 

MARGARET TANZOCH: So when you're in the IDE-- and I'll be doing a live demo in tomorrow. So if you're online tomorrow, I'll be happy to demonstrate what that looks like. In the IDE, there is, under the file tab, there's something called file revision history. And there's a slider that's provided. And it allows you to go back in time. You actually see timestamps for exactly the way you typed in your code from the very beginning at the time when you created your file. So if I slide back and I see a full program developed and then one minute later, nothing in the code, the only way that that code could get into that file is by a copy and paste, right? You can't type in code and have it all appear in one step. 

So that's one of the things that I might look at to see how the student, the process that they use to develop their code. So it's a really valuable tool. And it's great, too, for when you're programming, if you mess something up, you could go back and reverse your code to a previous place, so that you haven't lost old versions of it. And right now, that works in the IDE environment. Not in the labs, but I understand next year that may be integrated. So we'll see. That's a really terrific tool. 

And if you remind me tomorrow, I'll be happy to demonstrate it, when we do the lives, the kind of demo on how submissions work from a student and a teacher perspective. 

I see another hand raised. Is that [INAUDIBLE]? And again please correct me. 

AUDIENCE: Yes. [INAUDIBLE]. 

MARGARET TANZOCH: [INAUDIBLE]. Nice to meet you. 

AUDIENCE: Hi, I'm enjoying your presentation. I just have one question. You mentioned the warmup at the beginning of the class, and that your class takes about 45 minutes, five days per week. So I was wondering that a student who is struggling, when that student comes to class like we said, on Monday how much time would a struggling student would have to spend like in a night to come back the next day's class to continue with the followup? 

MARGARET TANZOCH: Well, again, I don't-- how much time would I have to spend working on their own for homework, you mean? 

AUDIENCE: Not really homework, but actually, it might be a great lesson, but that student may be struggling. And so it will need to go back and revise and recapture what was done. So your 45 minutes might be more like 2 hours for that student, talking about a struggling student. 

MARGARET TANZOCH: Yes, and I do have those type of students, absolutely. So what I'll do is, then, I will do a small group work with those students. So I'll pull together a group of students, three or four students, possibly. We'll sit around the table. And my computer will project to the screen, while other students are on their computers, facing like the sides of the back of the room. I'll sit through with those students. And I'll work through maybe a little bit more of a live coding example, for instance, on how to use operators, how do we assign values to variables, kind of break it down very, very step by step. 

I may actually start working on a program with them. So I may have them open up one of-- we have a couple of school laptops. They'll sit around. And they may type in the first couple lines of code with me. And I'll allow them, in a very small group environment, they're much more comfortable asking questions. 

It's true, in a full class environment, they may feel intimidated, particularly because sometimes, you'll have students that struggle to pass algebra sitting right next to a student that's taking calculus. And so they may feel very embarrassed and feel like, I just can't do this. Everybody has this other than me. So that's why I've tried very hard to make sure that we have a classroom culture of students being accepting of each other. And I'll work, again, with small groups of two or three students, so that they feel a little more comfortable. And I'll go over that lesson with them over and over again. 

And in some cases, they can do a few lines with the code by themselves. If they get a little something out of it, they'll get a little something out of it. And they'll kind of move on. And then I'll help them with the next program, as well. But sometimes, it's a very little more of a direct instruction that I'm kind of working with them. And also, again, I will encourage them to try out some of the labs, like some of the embedded videos in the program specs. And eventually, they may not get to complete Vigenére or even Caesar. They may just do the first few weeks of problems. And then I might substitute a writing problem for one of the more challenging programming problems for them, so that they could have an alternate assignment, basically. 

Keep them engaged and maybe on a topic, a current events topic or something that they'd be interested in. And then that will be a slightly different experience to have. But hopefully, they still get something out of the class. 

So what I was going to talk about, just some ideas here that other cs50 teachers have talked about in the past that I thought were really great ones, is that some students, some teachers, rather, will have students walk through their code to another student. So I think it's a great idea. It's not something I've done right now, but something I would like to figure out if I could kind pair students of similar-- that are working at similar paces on their programs, after they complete their program, to kind of walk through and explain their code to another student. 

Some teachers will have the students actually create a short video walking through their code, line by line, which I think is a really interesting idea. And that might be something that you could do instead of having students do a written reflection, kind of make sure that they're understanding what they're creating. 

Teachers have talked about bringing back students from past years to provide additional support as a TFA to the class. So that's really helpful. Again, I don't usually have the luxury of having that. So instead I'll have student experts that have completed a problem that will kind of walk around and help other students with a code. And a lot of students actually really enjoy that. I've had students comment that they were afraid they were going to be a little bored in the class because they've really been programming for years. But after finishing the cs50 problem, they really enjoyed helping your peers. So that could be another way to run the class. 

Again, I'll purposefully make mistakes. I'll stress a lot. And other teachers have talked about this, as well, that they'll stress a lot the idea of making mistakes, to really normalize failure, normalize errors, and make sure that the classroom is a very safe space, where students feel they can't fail. But that doesn't mean that they are failing. It just means that they haven't quite gotten there yet. That it's a process. And that perseverance is really key. 

Some teachers have talked about preventing last minute panic, when they do have like little more strict deadlines for programs to be due. And in order to prevent that, they might have something like a 24 tardy, where a student, with teacher approval, can have an extra 24 hours to complete a program. So they're not up at 3:00 in the morning feeling like, oh, my God, I'm going to fail this project. I have to get it in. So let me just copy from a friend. So again that's something that's really worked well for other teachers. 

Also, some teachers have had various different online discussion forums, where students can, together, share their thoughts and questions as they work through programs. And that's something that I'm going to be looking at this year, if we do start the year remote. Here in New York City, we have no idea yet how we're starting the school year, no idea at all. So I am trying to kind of think of new ideas for students to be able to collaborate and get to know each other in an online forum, one that I'm moderating and I'm keeping tabs on to make sure they're not sharing too much, might be a really interesting way to go. 

Some teachers also talked about white boarding and having students writing on the walls or on chart paper to kind of work out solutions that they then share with their classmates and with the rest of the class. 

So again, just so many different ways to run the class, which is one of the reasons I love this curriculum. I'm not boxed into having to do it a certain way. I prefer, rather than just having students watch the lecture, I do prefer to teach short segments of it myself. And I think a lot of my students do enjoy that kind of teacher to student relationship, in that sense. But then having all these resources available for backup. 

It's been a pleasure meeting all of you. I hope everybody got something out of it. I know many of you were teaching in many different environments, not necessarily in high school. But hopefully, it was helpful for you, as well. So thank you so much for being part of this.