JASON HIRSCHHORN: Good morning, everyone, or good afternoon, or thanks for getting up in the middle of the night, depending on where you are. Today's schedule per the email that David had sent out to everyone, first, for an hour this morning we're going to talk a bit about assessments and we're going to spend the, hopefully, the bulk of this time giving you a chance to look through a lot of assessments that have been written already so you can get a feel for the flavor of the assessments, which I think do a nice job of communicating sort of the overall vision of how to engage students. 

And then from 1 to 2 PM Eastern time, we'll have an overview of all the possible courses you can teach before or after CS50 AP. We'll take a break for a bit and then we'll reconvene later today for a panel featuring three CS50 AP teachers, who will each talk a bit about their schools, they'll talk a bit about their history of teaching CS50 AP, some specific things they do to implement the curriculum, and then also about what remote learning has looked like from them. So you can get sort of a sense of all of these teachers and what it's looked like on the ground, and then have plenty of time to ask questions to them or to the larger group. 

So that'll be this afternoon's session, unless you're attending Megan's wedding anniversary, in which case you'll be at that. But otherwise, we have exciting programming for you. 

I'm going to jump in right now by sharing my screen. This morning, as mentioned, we're going to talk a bit about writing assessments. If you weren't here yesterday, this is a bit about my background. If you forgot, here's a bit about my background. And of course, that's not what I'm going to look like. I thought this was funny yesterday so I decided to do it again, put my mask on. But also, it's not funny. You should probably wear a mask when you're outside. 

A note about my school, too, that's not on here is they expect, at the end of every unit, you to have some sort of summative unit assessment. So when I was creating this course, I had to create unit assessments that would take place let's say every month or so. So I had to have something like that, which is-- and you'll see examples of those later today. CS50, which I'll mention in a bit, doesn't do assessments quite like that at the college level. But again, we'll talk about that in a few minutes. 

Here's our agenda. I'd like to start with giving everyone a chance to meet some more of their neighbors. So in a bit after this slide, you'll be put into breakout rooms, but hopefully with a different group. It's pseudorandom. So you might meet some of your same friends, but that's OK, there are some new questions for you to talk about. But hopefully, you'll get to meet some more people as well. 

Then, per the feedback we received from earlier in the workshop, Brian and I are going to demonstrate the features of submit.cs50.io from the teacher and the student perspective. Then we're going to introduce this brand new resource to you all to help us moving forward, leverage everything that your fellow teachers have built and CS50 has built. And then you'll get a chance to dive into and take a look at tons of assessment, so you can get a flavor of the types of questions that fit nicely with the CS50 curriculum. And finally, if you want to come back, there'll be a chance for a Q&A before the next session. 

As before, if you have a question, feel free to raise your hand. Sometimes it's easier to see that and it's easier for people to be able to pay attention to your question if I pause and give you the floor versus if it's scrolling by in the chat. 

But first, of course, let's meet some more of our neighbors. This time, I'd still like you to introduce yourselves because hopefully you'll be with a group of new people, but also a new prompt for you-- I'd like you to share a food you love that you've had recently and why you love that item. Of course, you can ignore these instructions entirely and talk about whatever you like, but here are some things to get you going. And once again, we'll give you about five minutes in those breakout rooms to get to know some of your neighbors. 

While we ponder that, Brian and I are now going to transition to a demonstration of the CS50.IO tools. Or submit.CS50.IO tools rather. Again, a few days ago Brian demonstrated Check 50, Style 50, Submit 50, and Submit.CS50.IO. 

And there were questions on what exactly does this look like from a teacher's or a student's perspective. When I'm setting up my class, when I'm having students submit using these tools, and when I'm checking their work using the automated system. So right now, Brian and I are going to do a live demonstration for you. Brian's going to be the teacher of a class, and I'll be one of his enrolled students. So I'll hand it off to my teacher. 

BRIAN YU: All right. Thanks, Jason. So once you set up a Submit.CS50.IO course, I know that some people were having some technical issues doing so. It looks like there might be an issue on GitHub's end. We've reached out to them to try and sort out that problem. But hopefully we should have that working for everyone shortly. 

But once you set up that course, you'll end up with a page that looks a little something like this. And I'll sort of walk through the key features of it. Up at the top is where you can name the class. So usually people name it after, like, the year or the class period or something like that. 

And then down below, this is the invitation link. And so you can send this link to your students. And when students click on that link, they'll be prompted to join the course. And what that means is that they will be granting you, the teacher, access to all of their submissions that they submit through Submit 50. And Jason will show you what that looks like in a moment. 

Beneath that are all of the slugs, otherwise known as just unique identifiers, for all of the problems that you want to keep track of in the class. And so, I've here just included CS50/problem/2020/x. so all of the 2020 versions of the CS 50 problems I'll be tracking here. But again, if you had written your own problems that you wanted to track, you could just add an additional slug that you wanted to keep track of here as well. So that's where you can do that. 

And then finally, the last setting are other course teachers. So if you have a TA for the class that you want to be able to have access to student submissions as well, you can add invitations to invite other teachers to be staff members for the course as well. And so here, in this case, I've added Kareem, for example, as a course teacher for this class. 

So now what I can do is take this invitation link that I'm supposed to share with students, and I've sent that invitation link to Jason. And I guess now I'll turn things over to Jason to show you what it looks like to accept that and then to submit something. And then I'll show you what it looks like from the teacher end to receive that submission. So, Jason, back to you. 

JASON: I'm going to share my screen, so you can see how this works for me. Here is just a home page. Brian sent me the link in the chat. So I'm going to copy. Let me open that link. I'm going to copy it into my address bar. Hit enter. 

And this is what it looks like for me. It says, you are invited to my CS 50 AP course, taught by Brian Yu and Kareem. This is where I, as a student, can verify that I'm in the right course from the right teacher. It says, I understand this will grant the teachers of this course access to my submissions. Seems reasonable to me, as a student. I will check that, and click on Join Course. 

I don't have anything submitted yet. That's because I'm just joining at the beginning. But maybe now I'll fast forward two months to working in my IDE on the problem Caesar.C. It's taken me a few weeks. I've struggled. I reached out to Brian, he's been a fantastic resource to me. And now I'm ready to submit. 

Here I am in my directory, and I'm going to use the Submit 50 prompt followed by the specific slug for this. We had a question earlier about, how do students-- do students have to memorize these or figure these out? Usually it works just like this. A teacher just gives it to the student, and they can copy or paste it. 

Or I actually have opened the specification. And down here at the bottom of the specification, oh, there's--hi, Brian. There's the slug as well. So students can copy it directly from the problem specification. But I got mine from Brian. I'm going to paste it here. And now I'm ready to hit Enter. 

So a few things you see here I have already logged in, because I've been submitting problems. So I didn't actually have to enter my GitHub username and password again. But if a student hasn't logged in in a while through there IDE, they'd have to re log in. 

Next it's saying, oh, this is the files that we submitted, and these are all the files that I don't want to submit. Those are just a personal project I'm working on. So I'm glad those aren't being submitted to Brian. I'm going to type y here, that I've kept in mind the course's academic honesty policy. 

And now I have this URL that I can go to see my results. So I'm going to click on that URL, open it in a new window. I see my submission with my time stamp. This is, again, from the student's point of view. I'm waiting on my results. I'm a little impatient. 

I see that I passed 10 out of 11 Check 50 checks, and my style was 0.98. So I am not quite happy with this. I wanted to get a perfect score. But I'm going to pause my work. Maybe I'll work on this in a bit. I'm going to hand it back over to my teacher. 

BRIAN YU: All right. Thanks, Jason. So now that Jason has now submitted one of the problems, I'd like to take a look at that submission and see how he did. So I'll go back, and here's the window I was at before. And up at the top, there a couple of buttons that I can click. 

Right now I'm in Settings mode. There's also a students' page. On the students' page, I can see all of the students that are currently enrolled in the course. Right now, it seems that I have three students. And I see Jason down here. He has one submission. 

That was the submission-- the problem that he just submitted, for example. And if I go over to the submissions tab, here I see a list of all of the problems that have been submitted as well as who submitted each one of them. So for Caesar, for example, I see that one student has submitted that problem. I see that Jason has submitted the problem. 

I can see when he submitted it. So he submitted it a minute ago. And then I also get to see the results of the automatic runs of Check 50 and Style 50. So here for Check 50, I see that Jason has passed 10 out of the 11 checks. If I want to see which 10 out of the 11, I can just click on Check 50, and that's going to open up the check 50 results in another page, where I can see, all right, these seem to all pass OK. 

Let's keep going. And OK, so the one that he seems to be failing is handling a non-numeric key where the argument of the program isn't actually a number. That's something that we ask students to check for, but it looks like Jason didn't quite handle that. But everything else looks really good from a correctness standpoint. So well done, Jason. 

And I can also check his style results as well. So here it says Style 50. It looks like 98% of his lines of code were well styled. But if I click on that Style50 link, I can actually see what the suggestion was. And it looks like the suggestion here is that Jason was just missing some indentation on this line right after the beginning of the code. 

And so if I wanted to leave feedback to Jason, for example, comment on that so he can see it as well, I could click on this Zero Comments link next to Jason's submission. And that will take me into a GitHub page, which will let me comment on Jason's work. 

So for example, if I wanted to comment on his indentation, I could just click next to line 9 right here, and just leave a comment to say, looks like this line should be indented. And I can add a comment to that as well. And so now Jason will be able to see that comment when he goes to check on his submission as well. 

So that then is Check 50 results, Style 50 results, and leaving comments on student submissions. And if I wanted to, I could also download his submission as a zip file, if I wanted to download it to my computer in order to run his code on my own computer, for example. That's an option available to me. 

Though most of the time, when I'm using just CS 50s problems, I don't have to do that. Because Check 50 has already run the program for me and told me what the results of all those correctness checks are. And then if I want to, if I want to download all this information, you can export all of this data from any particular problem, as just a CSV spreadsheet that you could open up in a spreadsheet program like Apple numbers or Excel or Google sheets, if you wanted to, to be able to see all of the students who have submitted the problem, when they've submitted that problem, and how many correctness checks they got, and what their style score was. So all that information is there too. 

There are also these toggles here, for latest and all. By default, when you're in Submit.CS50.IO, you're only going to see the latest submission from any student. So if they submitted three or four times, you'll only see the most recent one. 

But if I switch from latest to all, for example, then if a student has submitted multiple times-- so this student, for example, has submitted a couple of times for the same problem, then I'm going to see all of those submissions in this view as well. So if you wanted to go back and look at a student's progression over time, as they might have submitted a problem a couple of times, you can use that view to be able to see all of the times that that student has submitted to. 

So that, then, is a brief look at what that workflow looks like for using Submit 50 and Submit.CS50.IO, where you send the invitation link to your students, then students can submit their work. And then you can then go in and view the results of the automated checks that are run on that code. We'll pause now to see if there are any questions, or if Jason has anything else to add. 

JASON: I'll just share once more my screen, so I can show the student's view before we take any questions. it's worth noting too that I got an email when Brian left the comment. So there's, on my phone, it might be a little hard to see, but I got an email saying looks like this line should be indented. 

Also when I refresh my student view, I can see the one comment note there. I can click on the one comment here to see all of the information my teachers left for me. And just as Brian clicked on Check 50 and Style 50, so too can I, as a student. I'm going to go here and click on my submissions tab. 

Actually, while Brian was sharing information with you all, I was like-- I was bummed by that indentation. I really-- I always make that mistake, and I always forget to do it. So I went back, and I fixed it. And I submitted again, so you can see this time, my style is a perfect 1.00, because I did take my teacher's advice and fix that particular line, and resubmit. 

So I will stop sharing now, and then open it up to anyone who would like to raise their hand and ask any questions about this workflow. We see-- [? Hani, ?] you have a question? 

STUDENT: Yeah. So I'm actually taking the course online right now. And to be honest, I'm not focused on style at all. I'm just focused on getting a program that works, you know. So like my score is typically, like, 11 out of 11 for the correctness, and 0.4, you know, for style. Now can I go back later and try to fix the style and improve my style score? 

BRIAN YU: Absolutely. With the online version of the Class CS50X, you can submit as many times as you want, and we'll only take a look at your most recent score in determining your score for the problem. And if you run Style 50 on the program, it will also tell you where the improvements to the style can be made. And that will hopefully be a useful guide there too. 

JASON: Next we have a question from Matthew. 

STUDENT: Yes, I'm curious about a window of what other folks do, or what you all do, in terms of when you would shut off the ability to continue to submit, either in correctness or in style. I just imagine, like, heading towards the end of the term, and students going back and trying to do this, this, this, this on all these different assignments, and having to change a grade-- Like, do you all set up a time limit or a time frame? 

BRIAN YU: From the teacher's perspective with the tool, you can't actually stop submissions. 

STUDENT: Right. 

BRIAN YU: There's no way for me to cut off submissions. So if students want to submit, they can continue submitting. So ultimately, then-- I know this isn't quite the answer to your question. It comes down to you as a teacher, looking at those time stamps and saying, I told them they had one week from the deadline, and after that I wouldn't submit anything else. 

And I see these time stamps are after that week, so I'm not even going to look at them. But from the tool's perspective, you can't actually shut down students' abilities to submit. 

STUDENT: Oh, and I wouldn't want to. 

BRIAN YU: Yeah. 

STUDENT: Right? I mean, we'd want them to-- right? We always want to get better. I'm just, I just think about-- anyway. 

STUDENT: But if you're, Matthew, recording the scores out of band, and you're on grade book of some sort. Then it really doesn't matter if they continue submitting, because you're not going to make any actionable change. 

BRIAN YU: Another feature that I can mention about the tool is that you can specify a window of times that you want to look at. So, for example-- I'll show you what I'm looking at here. If I go to all submissions, I can see, Jason right now has submitted twice. 

He's submitted once at 12:23 p.m., where he got a style score of 98%. And he submitted a little later at 12:28 p.m. and got a style score of 1 p.m. So, of course, these were only five minutes apart. But you could imagine maybe they were five days apart, for instance, where the deadline might have been in between. 

And so, by default, if I go to the latest submission, I'm looking at his most recent submission that has a perfect score on style. But I also have this option, where I can filter based on a particular range of dates and times. So if I set the range of dates and times to be only until, let's say 12:25 p.m., for example. So before Jason submitted for the second time. 

And I apply that time filter, then the submission that I see for Jason is his first submission, the one that got the 98%, because I'm only restricting the submissions I'm looking at to a particular date and time range. So if you do have a deadline, and you don't want to consider deadlines after a particular time, you can use that feature to be able to limit the view of what you're seeing. 

STUDENT: And Brian, is it fair to assume that when you put on that filter, that applies to all the students for that particular problem set? 

BRIAN YU: For that particular problem, yeah. So if there are multiple problems, then-- multiple problems, you'll have to specify a separate date range for each of the problems. But for any one problem, you can specify what range you want to look at for that. 

STUDENT: You all are so unbelievably detail oriented. I love it. 

JASON: And, Matthew, too, so this afternoon from the three to five panel, we'll have a-- you'll have three teachers who are teaching CS 50. So I think a good question to ask them would be, sort of, how do they handle late submissions? Douglas mentioned, talked a bit about that yesterday with his tardies and lates. But they'll be able to give you three answers on how they handle student submission. 

So if you want to hold on to that question, ask them. I think they'll be able to give you some good answers. And then, oh, Ben. I'll just quickly answer yours, Ben, for what I do. But I think that would be another good question to ask. Ben, in the chat, mentioned if I'd be willing to share how I assigned this in my Schoology platform, which is what the school uses to keep track of assignments. 

Again, I think that would be a good question to ask the panel of teachers later today. But I simply share a link to the problem specification, which details all the background, has links to videos, has example cases, and has the Submit slug at the very end. So I share a link to that online document. So I don't need to reinvent everything. It's already written there and organized really nicely. Does that help, Ben, briefly? I don't know where you are, Ben. Oh, thanks, Ben. 

OK, I'm going to resume sharing my screen. I found this funny comic. I think sometimes when we're in our-- when we're teaching our course, we-- and we don't know what's available out there, we might feel the pressure to have to create things from scratch. And those things we create from scratch might have already been created or done by other people in the world. But also other teachers teaching CS 50 AP. 

So today we're launching this brand-new tool and brand-new email address, share@CS50.Harvard.edu. And our plan is, that you can email us the problems you write, the labs you build, the activities you create, the assessments you design, and whatever else you've got. And then we'll curate it, add it to what CS50 has already built over the years, and post it online, so that you don't have to reinvent the wheel. 

So you'll hear a bit later today, Margaret's created all these fantastic labs. You'll see in a bit some of the assessments I wrote for my own students. Rather than only being able to give them to our own students, or share them with our own students, we're going to create an online portal where you can see all of these wonderful resources that your colleagues have created. 

And this is the email address through which you can share the things you've built with us. Again we'll curate it, get it organized, ensure it's searchable, and then it will be posted on the AP website, so that you can look through labs that have been built, sample problems people have wrote to facilitate a particular skill during an in-class exercise, or assessment questions teachers have written in addition to the ones David and his team have already written. 

And as I saw in the chat, John mentioned that somebody mentioned about somebody who had created a Google form to ask questions about videos as people watched it. Those are the types of things we would love to collect and curate and then share with you all. So again, the email address is live. You don't need to start e-mailing right away. 

But if you have things from the past to share, we would love to have them collected here. And then we will start putting them all together to have you all-- have for you all before the start of the term. Or hopefully well before the start of the term, because I know I start planning at least in early August, if not late July. 

[LAUGHTER] 

OK, so finally we'll move into talking about assessments. And I think, before you jump to click on these two links-- I'll share these links in the chat-- I just wanted to talk a bit about the philosophy behind assessments. And I have not heard this from David Malan directly, but when I took CS50 as a student, I sort of experienced assessments that were unlike anything else I'd experienced in other classes. 

They weren't expecting rote memorization. They weren't asking me to regurgitate what I'd been taught in class. But they were asking me, instead, to apply different things I had learned, both in lectures and in sections, and on my own as I was working through problems. Apply all of that in some new context. 

So I had all the building blocks I needed to be able to be successful. And on top of that, I needed to think creatively. I needed to problem solve. And ultimately, the assessments themselves that I took in CS50 were learning opportunities. And it was this really magical experience for me as a student. 

And when I was a teacher, this year, teaching CS50 AP to my students, I tried to design assessments and questions that would similarly allow them to use what they've learned and apply it in new ways. And actually, I dare say, enjoy the assessments. 

And that's what I heard from some of the students this year, that they looked forward to taking assessments. Because it was asking them to think about new things, and think in slightly different ways, and solve new problems using the information they have learned. So that, I think, is a bit of the philosophy behind assessments in CS50. Before I talk about these two links in particular, David, did you want to add anything else about, sort of your philosophy behind assessment questions? 

DAVID MALAN: Sure. I should disclaim that you probably described it more magically than all students might describe it. I dare say that the flip side of that is that there's often frustration. Students who feel that, hey, we didn't talk about this in class. I have no idea how to solve this. 

But the course really is intended to be a course that gives you enough of what you need to know to then go and teach yourself the other things. So that at the end of the semester, we really have taken off all of those training wheels. And we have-- and this has been a recent thing in recent years, the quizzes that Jason has linked here under CS50's are very different from what they were a few years ago. 

Long story short, I used to take a more traditional approach to asking some multiple choice, true/false, short answers, and the like. But they really were regurgitive of the kind of content we'd discussed in the class. And now, Brian and I and the team really view the quizzes in the class, or the test, as an opportunity to teach something further. 

I mentioned the other day the Hawaii incident, where the missile alert was triggered some time ago. And we used that as an opportunity to get students thinking about real-world issues. And so I often use our assessments now as a vehicle for introducing and, in turn, assessing material that we didn't necessarily have time for or want to spend classroom time on. So as such, they've become an extension of teaching more than they are just quantitative measures. 

JASON: So these two links are not clickable, obviously, through sharing my screen. But this PowerPoint-- or these Google slides are available through the notion link. We will copy these links, and put them in the chat as well, if that hasn't been done already. I don't have the chat open right now. 

But the rest of this session is for you to open these, to look through these, to kind of get a sense of some of these assessment questions. And then, if you're, like, thinking to yourself, I would love to use some of these, you are in luck. Because we're going to take-- these are part of what we're putting into that new tool to share with you all. 

So we will not just have these PDFs in, but we'll have them in as separate questions that you can search through and edit. And so don't feel like you need to download these, save these, copy these right now. This is just to get a flavor of-- sense of the flavor of what's available. And then, oop-- we will ultimately put these together with everything you've sent us, so you have access to that whole wealth of resources. 

[? Ahma ?] asked a good question in the chat. These only have the questions. They don't have the answers. But we are going to make available questions and answers. All of the assessments I made have the answer and answer keys. 

So all of that will be available to teachers. That's just not what is being shared with you right now. Feel free to go look through these. The next 20 minutes we'll be on here for you to ask questions as you're diving in. But that's the end of the content we have for you before the next session starts in about 20 minutes.