MARGARET: Everybody, it's so really nice to meet you. I'm here from New York City, and I've been part of the kind of CS50 community f back from when they first made it available as high school curriculum, back in 2015. And, as was some of the presenters yesterday, we've been doing it for a while. And so I've been really thinking a lot over this last year, particularly as the AP assessments have changed for the AP CSP performance task and multiple choice exam. You know how to best fit in the CS50 curriculum to really support students. 

So, I'm going to share some slides and then feel free to ask questions as I go along. All right. But just to talk a little about who I am. I have been teaching computer science, actually, for the past six years, out of a school called NEST+m in New York City. And I teach, generally, four sections of computer science, like an intro class-- and I use CS50, actually, for all of my classes-- and one section of AP CSP. 

I have a lot of students. In New York, we generally have anywhere from, in my intro classes, usually about 33 students in the class, and in my AP class, around 25. So there's a lot of students. I've also been teaching CS50 at the Harvard Extension School, Summer School, and I've been AP CSP reader and table leader for the last many years. So although I still admit I still have a lot to learn, and I'm hoping that we can have some collaborative sessions here as well, where everybody can share some of their thoughts on all of this. 

So the school I'm at, NEST+m is short for New Explorations into Science, Technology, and Math. It's a New York City Public School. It's kind of unusual. It's a K-12, what they call gifted and talented public school. One of the only ones, really, attracts students from all five boroughs. And it's on the Lower East side of Manhattan. This is a picture of the courtyard in front of this school. 

And the next slide is actually my favorite picture. This is a picture from a hackathon that we hosted at our school a couple of years ago, before COVID. David came, we had our middle school and our high school students show up, and we just had a blast. And it was just a Saturday afternoon, as David talked about yesterday, not an all night hackathon, but during the daytime hours. The students had an opportunity to really spent more than their normal short classroom period working with their peers, learning new stuff, and experimenting. And so it was just a really fun day. 

So my students are really quite diverse. I have students that come into my classes that absolutely nothing about programming. They might know a little bit of HTML, from a web page they created in the past. Some of them are extremely experienced. Some of them want to major in CS, think they might want a major in CS, want to find out if they like CS. And the level of the students basically can be in the same classroom. Students who struggle with Algebra 1, all the way through to Calc BC students. So it's a very diverse group of students. 

And so I've been able to use the CS50 curriculum and all the support tools to really be able to differentiate, and I don't think I couldn't have done it without CS50, to keep everybody engaged and everybody working at the level exactly where they're at. My students are usually 11th and 12th graders. I occasionally have a ninth or 10th grader. In my school, basically, the computer science courses are electives, and so there are so many other courses they're taking. They're programmed pretty tight. From 9th and 10th grade, they don't even have room for electives. 

So 11th grade is the first opportunity for them to take an elective, and at my school, they usually want students to take an intro class before an AP class. So we structured it so that they take the introduction I'll show in a moment the first couple of weeks of the CS50 curriculum, as part of the intro curriculum. And then we take it on from there for the second year, when they do the full AP curriculum and take the AP assessments. 

So I adapt. I use a lot of it as is. I also add to it, tweak it, and change it around a lot. For the intro classes, I've been using the CS50 Labs for the entire first semester. And if you're watching recorded sessions from last year, last summer I presented a lot on how the labs could work. And if anybody has questions, I'm happy to share with you a little bit about that today as well. 

But basically I use the material from week 0, which is Scratch. Week 1, which is programs like Mario, sometimes in the AP curriculum there may be ISBN, or more sophisticated programs like credit. And then week 2, which is about strings, where we have Caesar, maybe Viginere, and some of the newer programs like substitution. 

And then, so that's semester one. Now the great thing is that, because of all materials I have available through CS50, for students that are more experienced, or just catch onto it, or fall in love with it-- there are some students who have never programmed, and once they start doing the problems they just they just want more and more. So I'll use the more comfortable versions of the programs, or give them materials and videos to either work at their own pace in weeks three, and sometimes even four. So nobody ever has to get bored. 

But that's kind of the minimum requirements I have students work through, and then for semester two there is a problem set called home page, where they make a website of four pages. I always use that is the introduction to HTML, CSS, and JavaScript. And then I introduce a little bit of game design using p5.js. And I've actually picked up a lot of the conceptual stuff from the CS50 game design class. I don't use the programming languages they, do LOVE and Lua, because I really need to use a web-based environment, because everybody doesn't have computers they can install software on. So that's some of the stuff. We've done Pong. We've done platformer games, getting a lot of the concepts from CS50. 

And then for the AP CSP class, I continue with the weeks, do a little summary again of week 2. Then go into week 3, 4. I skipped 5. I don't do Speller or memory allocation for my high school, as there's just too many students to support. Even though some of them could certainly handle it, I always felt I just have too many students to really be able to individually help students in the way they might need to get through that. And then 6, we do Python. And then we do a little bit of Flask and SQL. 

And basically I use a combination to help prepare students with various different resources, some resources from AP Classroom, et cetera. My classes are normally five days a week, 45 minutes a period. And this last year, we had only 30. We were totally remote for the instructional part of our day, and we only had 30 minute periods. So we weren't able to get quite as far, but it was still a very productive year. 

My own personal background. I've been teaching for 11 years, and I first took CS50 on edx.org, which was amazing. I was looking for a curriculum to use as I started teaching at the SQL, started teaching computer science, looking for something, and just happened to find CS50 at the right time. I did take CS courses in college. I worked as a programmer the first few years of my working life, and after that worked with various software vendors. I was involved with a graphic design startup. I taught myself some web design. Did some early HTML stuff when HTML was really ugly. But it was a real experience. 

And I was really interested, from that point on, to delving back in and learning more programming, particularly more modern languages. And so I often tried to take various CS courses online, but quite frankly, I just couldn't stay with them. I just found them to be not very exciting. I just couldn't stay with it. So when I found CS50, I just fell in love with it. It was so engaging. I just couldn't put it down. So I've always taken that excitement with me, I think, in the classroom. 

So what I wanted to talk a little about today were the AP CSP assessments, particularly some thoughts I had, some ideas. And I'm always interested in seeing who else down the road might want to collaborate on some of these different programs that I've been tweaking to help make them work with the AP curriculum. So I wanted to start with the Create Performance Task. 

So as those of you who are involved in the AP end of things, as you know, there's the new requirements this past year. Programs have to have an array or list, a dictionary, some of the kind of collection type, function with a parameter. But the function also has to include iteration and selection, and the iteration, or rather the selection, has to be involving the parameter. So there's some really specific requirements, and I found sometimes it wasn't really that easy to come up with an idea of a program that satisfied all those requirements. 

And then the written responses. There's only a rubric of 6 rows, as you've probably seen. And having been an AP reader this year, I mean, there are many, many, many, many things that students need to do just to get one point. So you can have everything perfect, and as we'll see in a moment, you talk about the function instead of the purpose of the program, you lose the entire point for row one. And I know somebody pointed that out of the chat window yesterday. 

So just please note, I'm going to try to open my chat window. It's hard for me to see all the chat messages. Oh nice, I see somebody is in New York City. That would be great. I'd love to connect with people and just stay in touch throughout the course of the year, so please feel free to do that. And please note it's not always easy for me to see the chat window when I'm sharing my screen. OK. 

So some of the thoughts I had was that it would be really nice, as I'm going through the CS50 curriculum, to have students work on a program that satisfied the Create requirements in Scratch, in C, and in Python, with the written responses. So Doug kind of touched on that a little bit yesterday, in having students work together with him in his building a birdhouse, I guess, part of his curriculum, to do a Scratch program. And then he had them write up a response to it, which I think is a great idea. 

I came up with a very-- not a very good, very simple Scratch program, which barely meets the requirements, but just to be able to demonstrate. I had one student who really struggled this year, and wanted to do his Create test in Scratch. And so I came up with it quickly, and I'll share it with you, but you all probably have much better ideas to really have students, maybe, even have a program specification, where they're actually creating a program in Scratch, and then writing up the sample Create test responses, which are, as Doug mentioned yesterday, the part they're really being graded on. 

But then I thought would be really interesting if, once they get to C, maybe in Unit 1 one of the programs that they complete has also that requirement, particularly a list and a function. And maybe when they get to Unit 2, and they're working the strings, one of the programs that they work on has a list or some kind of a dictionary, and a function with the parameter et cetera, so that by the time they get to the Create test they really know what they need to do. 

So I started to kind of come up with some things I'll share with you in a few moments, but as I mentioned the program specs can be very confusing. I found them to be-- or rather, the requirements for Create I found to be pretty confusing. I spent a good amount of time on the AP discussion board just trying to really clarify exactly what they were looking for. 

So I thought it would be really interesting if we had actual program specs, maybe something I would write up. I don't know if Carter or Bernie would be involved, or maybe some of you, but we had program specs to complete programs that have that function with the parameter and a function call, that the function has a loop that a conditional, and that different branches of that conditional would execute depending on the argument being passed. And also that the program, the array or the list, has to be purposeful. And again, the written response would be included as part of that problem, so that by the time they actually get to the Create test they're really familiar with it. 

And I guess I was thinking this could be-- because I had them do a practice Create test this year, where they created their own program from scratch. It took forever. It just took so much time. So I thought, maybe it would be more productive to have them actually work on a problem specification where they now have gone through it, we go through the solution, they write everything up, and we could be very, very clear. 

So the types of possible sample programs I came up with. I came up with for instance, in Scratch and C And Python, just a grade calculator, where you put in a number of grades, and you would then, for instance, be told, did that equate to an A or B or a C or D or whatever? So this is the simple little Scratch program I put together. And again, I did this for the one student who was trying to figure out what exactly he needed to do to have the requirements. And very simple. 

And I have a Google Drive folder that I'm going to share with you all in a bit that has all this in it, that you can see the link to this. But basically, you know, I have an input here. The input is a list. The list has all the grades, and depending on the average you generate an A, B, C, D, or an F. Again, not very exciting, but kind of a minimal program that satisfies the requirements. 

So I guess what I was thinking is that maybe it would be nice to have samples, like before students even start on the problem spec to solve the problem, they could actually look at existing code, to really understand it and break it down so that they really have a good idea of what they need to do. This one is an example of a C program that I created that does the same thing, basically. 

So we have a list, and we have-- or an array in C-- and then we have a calculate function that, similar to the Scratch program, calculates an A, B, C, D or F. And it should have all the requirements, because you have different branches that execute depending on what the input is. The input is a list. It has to be a list. Why does it have to be a list? Because you can't really have a variable number of grades without having a list. So in my estimation, it kind of satisfies what you need to do. And then the same thing in Python. 

And What I did this year is that I actually went over this with the students. I created an exemplar using this Python program for them to use, with an exemplar a written response-- which, quite honestly, after being an AP reader, I probably wouldn't got 100, or 6 out of 6 anyway, because of the difference between function and purpose. 

But, so those what I was thinking in terms of the samples. So we have a program. We can show it to the student. We could go through it, and explain what parts of this would be necessary in order to have the function and the list that satisfy the requirements. And then I was thinking, it might be nice to have problem sets that the students do, or maybe just sometimes a tweaked problem set, that satisfies all the requirements. 

So some of the things I was thinking about. At Harvard last year they did labs with each unit, which were really great. They were more practice problems that students worked on before they got to their problem sets. And one of the labs for the Week 1 was called Scrabble. And what Scrabble does-- and I could show it to you just a moment-- what Scrabble does is it tells you the points, basically, for a word, like a Scrabble word. And what I was thinking with a minor change to the existing-- there actually is a CS50 problem spec that was used at Harvard last year-- with a minor change, it could be a problem a Unit 1 problem that satisfies the criteria. 

So I'm going to stop sharing this just for a moment, and I'm going to go to my IDE, and I'll show you a little bit more what I'm talking about. So I'm just refreshing my IDE. It kind of times out. Does anybody have any questions in the meantime, or any comments? 

BERNIE: Margaret, let me-- I'll read one of the questions that came up. So have you found the same issues Douglas did yesterday about students doing better earlier in the year? And that's from-- 

MARGARET: In terms of working on Create really early in the year? 

BERNIE: I believe so. 

MARGARET: You know, that's a really good question. So my AP students have already done some coding in C by the time they take my AP class, and to be honest, most of them don't want to go back to Scratch. I have always a couple of struggling students in that class that prefer to use Scratch, but for the most part, most of them really want to learn how to program more sophisticated stuff. And they enjoy the challenge of writing a more sophisticated program. So I've done it more halfway through the year. I don't wait until the end, because, as I'll talk about in a moment, it always seems to be-- they overcomplicate it. 

And probably like, the more we did, the more complicated the program would be, and the longer it takes to program. So I'm going to show you in a moment, I've come up with some ideas on even putting together some guidelines on how to approach the Create task, to really try to help students realize that they shouldn't be spending-- because the Create performance test, we have to give them 12 hours. In my case, with 45 minute periods, it's almost a month of class time where I'm not really allowed to help them one on one. It's a lot of time. 

And so, I try to give them advice in terms of how to structure their time. Of course, many of them don't listen to it, and they usually take the majority of time to work on the program. So in any case, I don't know if that answers your question. So I usually do it halfway through the year. What I did, in the last few years-- and I'm totally changing my thoughts on it this year-- was that I thought it would be easier for them to write their program in Python, because Python is such a powerful language. But I think now, for next year, I'm going to highly recommend that they write their program in C. 

And part of the reason is that Python is so powerful, it's hard to write a function that has to do all those things without writing a really sophisticated program. Because in C, If you write a linear search function, that function-- and you give it an input of a target value, maybe, and an array-- the linear search function satisfies the function requirements for Create. And do you need to write a linear search function in Python? I mean, you could, but it's kind of writing bad Python, because Python does it for you automatically. 

So for those reasons, next year, I'm going to highly recommend that students do their Create tests in C, earlier in the year. Like, before we get to-- or maybe just introduce a tiny bit of Python, and then have them do their Create task after that. So that students who-- there are some students that come in knowing Python, and love Python, and want to work in Python, but I think that for the majority of students that are not experienced in Python, I'm going to recommend C. I hope I answered your question. So I'll probably do it like in the end of the fall semester in this coming year, sometime around that time. Other questions? 

BERNIE: We have another one from Esther, here. If you have 10 set of 90 minute classes-- and she's speaking about summer class, I guess-- how would you create the curriculum module while teaching them? 

MARGARET: If I have 10 set of 90 minute classes. I'd have to really think about it. In other words, how to break down the curriculum to get everything done in 10 times 90 minutes? 10 90 minute classes? I mean, I would guess that they would have to have a lot more homework than I give my students. I love my students to work on their problems-- most of the class time is them doing their coding problems, so that I'm there, so they're there to support each other. 

But I would think when you have such a limited amount of time to do it, they'd really have to complete a lot of their programs for homework. And maybe even use the lectures, assign them as homework, so that they watch the lectures, and then maybe could come in with questions and review the key points, and really use that class time. Have them, maybe, do a good amount of their programming for homework, and then coming in and doing a lot of problem solving, debugging, clarifying questions, things like that, to really get the most-- because the most valuable part of you, right, is going to be helping them with the problems that they encounter. 

That's my experience also, right? Because they could get a lot of information from videos, but there's also going to be confusion. I don't quite understand this, or I don't know how to fix this bug. And so I think having you be present with them to help them with that is probably going to be one of the more valuable uses of your time. I'm looking at another question, here. I don't know if it's a question, or just a comment that you were making to each other? So, Bernie, any other questions? 

BERNIE: Sure. This is from Miguel. So after the question I just read, this is another one from Miguel. Margaret, considering classes five times a week, you do each course week in how much time? And then, I guess, one week class per course a week? 

MARGARET: Right, so I don't strictly map one week to one CS50 week, because my students also are often taking, like, five AP classes. They have a lot of stresses in their life, and computer science is an elective. They don't need it to graduate, where they do need other things. So my philosophy, I don't want to stress them out that much. And they're very competitive. They're really always looking to get into the high ranking colleges. And so it kind of depends on the group of students. 

So it's not like I have a totally-- it depends. Like, sometimes I go slower than other times. But at the beginning, for my AP class, they've already seen most of the work in Unit 2. So I may have them finish that up in a week, and then move on. Unit 3 is around algorithms. And I'll also introduce other programs. So I don't just do the programs that are done at the Harvard level, which are just a couple of problems with the problem set. I'll have them do search and sorting algorithms-- and I'll show that in a moment-- and other programs as well, to make sure they really understand the algorithmic issues. 

And then for images, I might help them with some of the stuff in Unit 4 around that. So basically, I don't have an exact map from one week of CS50 to one week of what I'm doing. It would probably be-- because basically, we do, for the AP class, it's, what did I say, 2, 3, and 4 is really going to-- or at least 2 and 3, before doing the Create task. And then there's a whole month I have to give them to do the Create task. And then we'll come back and we'll continue to do-- because I don't need them to do the image stuff before Create. 

Anyway, this year I'm going to do it a little differently. So I'm still thinking a little bit about the structure of it, because I'm going to be pushing C a lot more than I did this current year. This current year, we were totally remote, the students didn't even really finish the year before. Many of them dropped off the spectrum. You know, they just barely came to class. So I knew that they didn't really have as much experience coming into the AP class, so I kind of started it pretty early on in Python. But I'm going to do it a little differently this year. 

How do I engage students online? Well, what I've done online, again, is that in the past I never really assigned the CS50 lectures to watch for homework, because I didn't really-- I knew that all students didn't, at that time, have computers available to them at home. Most of them did, but there were always a few that didn't, so I didn't think that was fair. I would make it optional, but it wasn't required this year. 

When we were remote, everybody had a computer at home, so I assigned to watch some of the CS50 lectures. Portions, like two chapters, for homework, and then I would do a couple of warm up questions, just at the beginning of class, to see who was familiar with that material. And then I would pretty quickly have them working together in breakout rooms. So for the most part, most of the class time was the students working together in breakout rooms, solving problems, collaborating, and then I would pop around and visit all the breakout rooms. 

And I'd let them work with their friends, because they didn't really have the chance to socialize like they normally would. So I would pop into the various breakout rooms and have conversations with them and just try to keep the excitement going. So that was basically what I did. 

So I'm seeing children 10, 12 years old, Python first? You know, whatever. I think that C with the training wheels. Just-- I mean, I would never do the more advanced Units, but I think certainly Week 1 in C can be done by middle school kids well. I think it's pretty straightforward, and the syntax is really beneficial, because it's so similar to JavaScript or Java, once you get familiar with the curly braces and the semicolons. 

But I think there's so many different ways to start teaching. You could just do simple stuff in Python, too, and that may be a little less frustrating for kids. They don't have to compile, they don't have to worry about data types and all that. So all right. 

OK, so I was going to go on, if that's OK, I was going to show one of the problems and how I tweaked it to basically satisfy the Create task. All right and then feel free, also, just feel free to ask more questions when they come up. So this is my IDE, and this is a version of Scrabble. So Scrabble, as I said, there's a CS50 problem spec for it. It was done at Harvard last year as a lab. Not in the lab environment, but basically practice problem before getting into the Unit 1 problems. 

And what it is, it assigns points to each letter in a word. And so an array is necessary, or certainly makes the coding much more efficient, right? Because without an array, you would have to have like if-else statements that go, like, a chain of 26 statements, which would be really inefficient coding. And the problem spec itself does have you make a function. So they really give you the function prototype, and you're basically getting two words, two strings, and you're using the compute_score function to calculate the score. 

And then this part would be the part that would be part of the problem that you would complete. You would just say who wins. And then you have the score. So, normally you would just need to do this. You would just need to iterate through every letter in the word, and then add up the points depending on what letter it is, right? You're using the ASCII code, making it uppercase, subtracting capital A, so it's going to correspond from 0 to 25, and then that's going to correspond to an index in the points array. 

And so using that array is not the only way to write the program, but tremendously more efficient than not using an array. But this, I mean, barely doesn't really satisfy that criteria that much, because there's no if statement in there. So what I thought is if we just have one little statement here that says if-- in Scrabble, you're not allowed to have a word with one letter on it, right? There's no points for a one letter word. So I just added this one little thing, if the length of the word is just one character wrong, return zero. Now you have your selection, and if you put in a word of one letter, you have a whole different branch of this function that you're going to complete than if you have a letter of multiple characters. 

So in my thinking, this would really satisfy the criteria for the Create. There's already a problem spec written for it. And the only, only difference-- the check50 would have to be tweaked-- the only difference is adding this one little criteria here. So that was one thing I thought could work pretty well. 

And it's a Unit 1 program. It's not that hard, and it's kind of a fun program. You're thinking about a game. And so what I was thinking is that-- and I haven't done this yet. Again, I'm going to plan on doing this next year, have students do this in unit 1 as one of the problems sets. And then have them write up the response to Create based on this. 

Another thing I thought for Unit 2 could be a modification of Substitution. So Substitution is a CS50 problem where you're using at the command line a jumbled up alphabet that you substitute, depending on the position of the letter of the alphabet, you substitute that letter for the letter in the plaintext. And so there's various uses of arrays. 

I created a tweaked version of this for the problem spec. I made sure there was a function that returned true or false in order to validate the key. And then there is an array, also, that's used, which makes it is it possible to save the letters that are going to be printed out in the ciphertext. 

So what I did, basically, which is a little bit different, is that I created a function. And we have here another array, that just tells if the letter was used or not, which is similar to the way you might do it with the standard problem spec, but it's not required to have a function. So I thought this would be great to have my students do this, requiring the function, tweaking the problem spec. 

And then over here, if they validate the key in a function, if they find one letter in the key that's not an alphabetic character, or if they find one of the letters was already used, a repeated letter, you return false. And if you don't return false, you go through everything, and there's nothing wrong with anything in that key, you return true. 

So again, two different branches and the key-- depending on what the key is, the key with repetitive letters, or key with a letter that's not alphabetic, will return a different branch of the function. So it seems to me this could work as well. And in addition, there is a requirement of having an array, and you do have to have an array in order to have a place to store the letters in a meaningful way, to be able to output them when you're outputting the ciphertext. 

So again, I am not sure if CS50 wants to kind of tweak the problem spec, if Bernie or Carter would want to do that, or I'm happy to do that and share that. But I thought that would be really beneficial to use problems like that in the course of teaching the material. Do you know what I mean? Not just when we're ready for the Create test, now, but as we're going through the material with every unit, we have a function, we have parameter, we have selection, we have iteration. We have all the requirements. 

At the time, after the students get even, just, done with, maybe, the Scratch project and unit 1, they're ready to write the Create test. They don't even need unit 2. But maybe, I was thinking of having them do it like Unit-- maybe the little Scratch thing, for practice, Unit 1, Unit 2. Maybe at that point, we'll see. Or maybe not have them do the Create test. 

Also in Unit 3 there already is a problem called Plurality, a voting problem. And in that problem there's a linear search, and so that also could function as a pre-test. Although it's probably a much more complicated program than most students are going to actually write on their own, in that you're really just writing two functions. 

And the other thing I thought was really important was to have students-- let me find my slides again here-- to have students actually write the function call. I've had a lot of problems with my students. I just find one of the hardest things for them sometimes- not all students, but some students-- is to really understand the idea of a function. 

And since they have to have a screen capture of the function call, and a screen capture the function itself. Also, to be honest, online some students were a little checked out. So it was hard to know if they just weren't listening, or if it just was a very difficult concept to understand. But I think what I'm going to do with these programs this year is not give them the function definition, and just have them complete the to do, but actually make them write the whole function, including the function definition, including the parameters, and including the function call. 

So that it's very clear to them, by the time they get finished with Unit 2, or Week 2, that this is how you write a function, this is what a function call is, this is what a parameter is. This is what we mean also by iteration and selection, which is a little bit of a different terminology there. We might just use a loop and an if statement. But just to make sure that it's totally, totally familiar. 

So that was what I was thinking for C. And I said anything with a linear search function. And then in Python, I wrote a program. I don't a program spec for it. I used it as a demo program again. And so what that was was-- if I can find it here, I think it it's in my IDE under Create, Scramble-- I made a program where they had to-- 

I had this simple little list-- where is it? Word list-- that I imported as a CSV file, or as a text file. And then I scrambled up, using Python, I scrambled the letters in that word. And then I created a function, check_word, where-- I found, in Python, that it was quite difficult to come up with meaningful ways of using iteration and a conditional without sometimes using bad Python. 

So in any case, I put the selection in here by checking if the length of the guess wasn't equal to the length of the word. I put a lot of comments in here to make it clear this is selection, this is iteration, and letter by letter. So why did I have iteration? In Python, I could have just compared one word to the other are very easily, but in order to have the iteration, I ended up creating a loop. 

So again, I felt that in Python, because of the power of Python, it was a little bit more challenging to come up with a good program that wasn't bad Python. Because as I said, I-- let me go back to my slides, hang on-- in Python, I felt that it's just so powerful that it was hard, sometimes, to do it. 

I ended up teaching my students bad Python. Like, don't just say if something in a list, because the AP reader may not recognize that that's iteration, that's a linear search. And then also substitution. You could do it in Python, create a converted program spec, as well, if you wanted to stress Python with students. 

So some of the areas where I found students struggling, also, as an AP reader, and I saw somebody mentioned this, yesterday, also, where students lost points. There were a lot of lost points on the Create test. So when I was grading the Create test, I very rarely gave anybody 6 points. And again, a lot of them were done in block based programs. 

A lot of them used the same-- they must have had, in one of the curriculums, the same exact sample problems, where they got a little bit of sample data, and they got a column from it, and they got another column from it, and then they had indices of the arrays that corresponded to each other. And they did a linear search to find something, found the item in the other list, over and over and over again. 

But the idea where a lot of points were lost was purpose versus function. And quite honestly, I would have lost a point on that, too. I should have seen it, but I just didn't. So purpose really had to be regarding the user. So if I had this grading program, right, if I said the purpose was to calculate a letter grade-- that's what the program does-- I would have to say something like, the purpose was to help a teacher easily find the letter grade for a series of number grades. So the semantics, the little bit of language, was enormously made a big difference in that point. 

And it's kind of a shame, because there were so many things students had to do for that row, right? They had to make a video. They had to talk about what the program did in more detail. And yet many, many, many students lost a point on that, because they simply didn't talk about the purpose as the purpose of the program from the user's point of view. 

Also, a list has to be purposeful, so some students have a list, but didn't really have a list that did very much, or was necessary to the program. Another area points were often lost were how the list manages complexity. Basically, not really giving a good rationale for why there was a list. So that was really necessary. 

Also, noting that the iteration of the list has to be purposeful. So you could have a function that has iteration, but if the iteration doesn't really result in anything, it doesn't really do anything, you lose the point on that. Another big area for students losing points was describing how the function contributed to overall functionality of the program. A lot of people describe what the function did, but didn't really connect it to the whole program. 

And it's a shame. A lot of these programs could have been really great programs, but again, because of the way the response was written up, they lost the point. A lot of confusion about two hypothetical function calls. So the last of the questions, for row 6, they asked about two different function calls. A lot of people interpreted that as two calls in the program. It was really just too hypothetical calls with different arguments passing the function, and then write about how each one of those causes a different branch of your iteration to execute a different branch of the function to execute. There was a lot of confusion about the function call versus the function definition. 

So these were things that I plan on, again, by allowing them to not just do like one practice test that takes almost a month, but really just have them work on problem specs, now, where they're doing these things as part of the curriculum, and then having them write up-- I usually have them write a reflection, as Doug was talking about. And I got that idea from Doug in year one that I was doing this. I usually have them write a reflection about their development process. So instead, I'm going to have them write up the responses to the Create task. 

So other issues that came up was that the written response asks you to describe, of course, what your procedure does, and how it contributes to the program, and that whole response is supposed to be in enough detail for somebody else to recreate it. That whole response is supposed to be done in 200 words. I mean, of course you could go a little over 200 if the whole response was supposed to be for the entire written response, 750 line, but they give you the extra 100 as leeway. 

But after that, they cut off typing words into these little boxes they give you. So it's just another thing to keep in mind, when students are trying to create that's too complicated, it becomes difficult to do that in such a short period of time, such a short number of words. So Yeah, I'm looking at some of the comments here. 

Yeah, so I'm looking at someone commenting that-- Matthew said, I don't think you can do the Performance task too early. [INAUDIBLE] your students not sophisticated enough to answer questions. Yeah, no, I do tend to agree. I don't wait until the end, though, because I have mostly seniors in my AP class, and quite honestly, toward the end of the year, they were already admitted to college, they're half checked out. The first year I did this, I had them do it at the end of the year, and there was already a waning enthusiasm for school. Senioritis and all that. 

So I really need to do it, I think, at the end of the fir-- this year I did it at the beginning of second semester, because I was teaching them Python, and Python was new to them. And we had such short classes. They weren't ready. But this year I'm going to, I think, I'm probably going to do it as part of what we do in semester one. Because I think by that time, they should have enough experience, particularly if we do these problems. With problem specs and written responses, they should have enough time to do it by the end of semester one. 

So, yeah, the function has to be on the simple side, but yet, it has to have these requirements. And again, as I was saying before, I think I just found it much harder to come up with an appropriate function in Python than in C. Because Python you don't even have to have a lot of code to iterate through something, to get a max or a min or an average. I mean, you can but it's not really good Python, right? 

And I remember in one of the programs-- I was converting Plurality to have them-- I thought it's a great program, so I converted it to Python. I rewrote the program spec to do it in Python. And then I got to the vote function, and I was thinking, Oh it should be "if name in names". You're looking to see if the name of the candidate is in the names-- I forget what it is, I guess the names dictionary. So, "if name in names". That's how you would do it Python, right? And what is that? It's a linear search. 

But when I posted that question on the AP discussion board, I was told that the reader may not recognize that as a loop, so you better not teach them very sophisticated way to do it. Do it very simple. So here I am having to teach my students to say, for name in names, if name equals equals whatever. The name that's given, return true. So whereas in C, you're forced to do it that way. 

So that's why I'm more convinced this year than in past years that it may be easier to come up with a simple function in C than trying to do something in Python. So again, these were some of my-- I just already said this, some of their own personal experiences that I give in the short classes. I started the year, pretty much, with Python, and ended up teaching in bad Python for this task, which I really felt terrible about. 

And I also made a bunch of different slides and videos guiding students to Create. I put that in a Google folder. I'll share that with you. And because some of the things I found students doing was just taking way too much time. So I'll just share that with you quickly. I'm running out of time, I realize I'm talking a lot. 

So I did this type of thing, and I shared it in here. Some of this really needs to be tweaked, but just really taking a look at all the different steps you need for the Create Performance Task-- because a lot of students think it's just about writing a program-- that you have to do all this. And coding the program, troubleshooting, testing, refining, repeating, that's just a little part of the process. That's not the whole thing. 

And so I also tried to clarify College Board terms, and that your program has to have all these things in it. Some reasons why students may want a list. Some ideas on why you might want a dictionary if you're doing it in Python, et cetera. So those were some of the things that I did, and everything is in the Google folder that I shared. 

So moving on, I think I pretty much finished this. That just too much time-- and even with all of that, I made videos, I explained it all, I walked through stuff, I walked through my samples, and still students spend most of their time coding their programs. So I'm hoping this year, having more familiarity with what these functions are going to look like, and simple programs that could satisfy the criteria, it'll be a little more straightforward. 

Some of the things I did to augment the curriculum for the multiple choice exam is that I created a series of searching and sorting labs, doing linear search, binary search, bubble sort, selection sort, even merge sort, which I never really assigned because it's a little more difficult. If one or two students are curious, I had them work on that, optionally. And those were labs in the lab environment, with the specs on the left, so that they could work on their own. And I assigned those for homework, and then we went over them together in class. 

I also created some labs-- and I shared all of these in the Google Doc folder-- using a simulation. So when you get far enough into the CS50 curriculum, into SQL, you start to see some ideas of cleaning data, et cetera, to be able to do data analysis. But I also wanted them to see programs that used a random function to do simulation. And so I created a lab that does that, including a simple little program, similar to the AP problems, where you want to simulate 80% of the students getting a particular grade using a random function. And then also the Monty Hall problem. Which again, it's in the lab with the specifications on the left. 

And then I, also, to help students understand more about cleaning data and visualization, I created a program, a COVID analysis program, where students had to actually clean data and in Python. There is a GitHub repo, and The New York Times has that, making it publicly available, of all the state data from the beginning of time. And so, I think it's over here, it's a link. 

So over here I made a Python program that-- using plotly, which is amazing-- a fairly simple program that outputs a graph like this. It creates an HTML file. You could then open it up with HTTP server in the IDE. And students really enjoyed this. And then I also gave them a file with the number of people at each state, which a file that had data that had to be cleaned. And so we worked on it. We cleaned the data together, and then completed this. And I have to say they really found it very relevant, obviously. They really enjoyed doing this. So that was something I did for the COVID analysis. 

And then for the internet part of the test I use some of the code.org videos, AP Classroom videos, et cetera, et cetera. And some ideas I had that I would be happy to share with people, or, again, if any other teachers would like to work on this, is that I thought it would be great to have-- you know how the AP Classroom has all these sample questions in it? I thought it'd be really nice to have a little quiz that goes with each unit, but the quiz also incorporates some of the pseudocode like sample questions that involve loops, or involve conditionals, or if else statements. 

And actually have students do these as quizzes with each week of the curriculum, so that it's not just doing-- up to now, I've mostly just done that type of test prep at the end of the year, before the multiple choice exam, but I think would be really effective to integrate it more as we go along. And I'd love to see that-- I don't know if Bernie and Carter are open to it, maybe even linked those little quizzes that use some of the AP questions, maybe linked-- and maybe it could be teacher resources, I don't know-- but linked to each week in the AP curriculum. 

And also just making sure to introduce some of the terminology that they use on the AP, procedure, selection, iteration, which is a little bit different than functions, if statements, and loops, which I tend to use with CS50. And again, also make sure they get more practice writing functions and function calls from earlier on, because I do find that's an area they always struggle with. And then just also using-- CS50 had some great labs they used last year as preparatory problems for their problem sets, and possibly incorporating more of those as well. Just to have more practice problems. So that's kind of it for my presentation.