DAVID MALAN: All right, hello, everyone. This is the CS50x educator workshop. Welcome back. So nice to see so many familiar faces and names now. Before we dive into today's session, which is on writing assessments, quizzes, and other assignments, led by CS50's own, Margaret Tanzosh, allow me to see if anyone here in the group would like to say a hello and a little bit about yourself, if you haven't yet shared. Anyone willing to say hello with a blue hand? 

Raymond, start with you. 

AUDIENCE: Hi, everyone. This is Raymond, from Brazil. I have watched the afternoon session from Margaret. So pay close attention to what she has to say because she has really good advice for us. So let's go. That's it. 

DAVID MALAN: Wonderful. thank you for teeing us up. Any other hellos or greetings from folks? No? All right, I think all the more time for Margaret's introduction, then. 

So Margaret Tanzosh has been teaching in New York City at high school. She's also served as a teaching fellow for CS50's extension school version. She's done an extraordinary job adapting materials and bringing her own into her classroom. She's wonderfully invited us a couple of times down for hackathons, of sorts, with her students. And we're joined as well by CS50's own, Brian Hume. But first, Margaret, take it away. 

MARGARET TANZOSH: Goodness, thank you so much for such a wonderful introduction. I'm going to share a couple of slides. And then we'll take it from there. 

All right, so some of you I've already met. But if I haven't met, you just to give you a little idea where I'm coming from. This is the school that I work at in New York City. It's a public K through 12 school. It's a gifted and talented school. So we have a lot of students that are really very much ready to learn. 

We also have IEP students and struggling students. So I have a wide variety. I teach primarily 11th and 12th graders. And this was from a hackathon we had last year that David was kind enough to come and visit and spend the day with us. We invited middle school and high school. And they had an amazing time that day, delving into some great programming problem. 

So I've been teaching CS50 at Nest+m. It stands for new explorations into science technology and math, for five years. I have a lot of students, typically. I generally have about four sections of an intro level of computer science, one section of AP Computer Science principles. And because I've been able to so customize and really fit the CS50 curriculum to what I'm looking to do, I've been able to make it work for all of my classes. 

So one of the things we're going to do when we start out is that we're going to work with Brian. We are going to give you a little bit of an overview of what it looks like to actually have that whole workflow from submitting at the command line, a program, from my ID. And you can see on the teacher's side what it looks like from the teacher's Submit50.IO, as well. 

So let's go back. Sorry. So I'm going to come into my IDE. And you could see here I have a version of Cash. And it looks pretty good. I'm going to try to submit it and see what happens. So I've CDed it to the correct directory. And I'm going to use the correct slug. Each program has its own slug. And oh, because I typed it in wrong. 

But before I do that, actually, let me actually turn it over to Brian to show how this is actually going to get to him and then come back to me. I'll fix my syntax. And I'll set submit it up on time. 

BRIAN YU: All right, thanks very much, Margaret. So if you were here for the session two days ago, I showed you submit.cs50.io, which is the web application that you can use in order to collect student submissions and also to view the results of Check50 and Style50 that have been run on your students' work. So Margaret and I today are going to show you is just a sample of what that workflow actually looks like in practice. 

So I have here a sample course that I've created on submit.cs50.io. And what you'll notice here is that in my configuration page, I have an invitation. This invitation link is something that you can send to your students. And when your students click on that invitation link, they'll be prompted to join your course on submit.cs50.io. And what that means is that you, the teacher, will get access to your students GitHub repositories. You have the ability to see their work, see what they've submitted, see the results of automated correctness tracking, and also have the ability to provide feedback on your students code, all via GitHub. 

Down beneath that, there's a section for specifying which slugs or unique identifiers you would like to use as part of the course. So for example, if you're teaching a course, it's based on the CS50x 2020 materials, I could include something like cs50/problem/2020/x inside of the slug section of this configuration page. And that means that any time that any of my students submit a problem that begins with this particular prefix, then I will be able to see that problem. And I'll be able to take a look at the results there, too. 

So I'll now turn it back over to Margaret. And she's going to show you what it looks like to actually submit a problem. And then I'll come back and show you what it looks like from the teacher's perspective, once a student submits their work. Margaret, back to you. 

MARGARET TANZOSH: Thank you, Brian. So I'm going to come back to my IDE. And what I did before is that I forgot to type it in [INAUDIBLE] So the code is going to be submit50, and then followed by the slug. And you notice how the slug has that prefix that Brian was talking about. So this is going to be viewed by him when I submit it. And I'm going to be typing in my GitHub username, my GitHub password. 

And at this point, I think the code will begin to start to upload. And it's telling me that the file that's going to be submitted is my source code, cash.c. And the file that's not going to be submitted is my compiled code, my object code, because once I submit it, the Check50 portion of the Submit50 is actually going to compile my code and make sure that it's compiled properly and run a series of tests. So I'm going to say, yes, to the academic policy, academic honesty policy. And this is now going to be uploaded. 

And if I wanted to see my results, I could click on this. But I'm a student. A lot of my students don't really always check. I'm just going to leave it at that. I think it looks good. So I'll just send that back to Brian. 

BRIAN YU: All right, so Margaret has now submitted her work. And now it's time for me, the teacher, to take a look at it and review it. So if I go back into submit.cs50.io, you'll notice along the top, there are a couple of different pages associated with each course. I have so far been showing you the settings page, which just allows me to configure the invitation link and configure the submissions logs for the course, for example. 

But over on the left hand side, you'll see a link that takes me to the course's submissions. And that's the page where I can see all of the work that my students have submitted to this course on submit.cs50.io. So if I click on submissions, I'll be taken to a page that shows me all of the various different problems to which students have submitted. But if I'm only interested in Cash, for example, in that problem, I can just click on that problem. And now I'm just looking at submissions for the Cash problem. 

And I can see here that one student has submitted work to this particular submission slogan. And I can now see, all right, here's Margaret's submission. She submitted one minute ago. And importantly, immediately beneath that, I get to see the results of Check50 and Style. 

So immediately after Margaret submitted her work, submit.cs50.io downloaded her code, ran Check50 and Style50 on it. And now it's automatically given to me the correctness results and the style results for this particular submission. 

So I can see, next to Check50, for example, that Margaret has passed 10 out of the 11 checks that were run on this problem. And if I want to see those Check50 results in a little bit more detail, I can go ahead and just open this up. I'll open it in a new tab and visit it. And here now I can see all of the checks that she's passed. And I can also see the check that she didn't pass. It looks like she's not quite handling rejecting a negative input, which is one of the requirements in the problem specification. 

But I can see here all the checks that were passed. And all of the checks that were not. So that can help me to provide feedback for her. And usually, by looking at the checks, I can get a pretty good sense for what she did right and where there might be room for improvement. 

So going back to the submit.cs50.io page, I also see her Style50 results. So those are scored on a scale from 0 to 1, where 1 or 100% means that all of the lines are correctly styled and 0 meaning none of the lines. So the 0.94 is pretty good. But if I open up the Style50 link, much as I can open up the Check50 link to see details about check50, I can also open up a style50 link to see the style details. And it looks like, just based on this green highlight here, there are some areas within Margaret's code where maybe she should add a space just for consistency because we tend to like there to be a space around operators, like multiplication, for example. So that might be an example of some style feedback that style50 is providing. 

And I can now see both of those results here. To the right of both the check50 and the style50 results is a place where I can now provide comments on Margaret's submission. So if I click on 0 comments and open that up, recall that when a student submits via submit50, they're ultimately submitting their work to GitHub, which we're using as the host for all of student submitted work. And because of that, we get to take advantage of GitHub's great commenting features that allows us to provide inline feedback on students. 

So if, for example, I wanted to provide feedback to Margaret about what style50 had said about adding a space around that multiplication operator, I could find that line of code, which, in this case, is line 9, right here. Hovering over the line, I get a little plus symbol. And clicking on that line opens up a text field, where I could leave some feedback, like, "remember to include spaces on both sides of the multiplication operator." 

And you can use markdown syntax. It's familiar with Markdown syntax to make things bold or make things mono space, if you're trying to include code into your comment, as well. And then I can add that comment, if I would like to. So Margaret then we'll be able to see that comment if she signs into submit.cs50.io or signs into GitHub, for example. And if she has GitHub's email notifications, which are turned on by default, she'll get an email notifying her that I've left this comment on her code. And she can then reply to it, allowing us to facilitate a conversation about the student's work in order to try to provide that sort of feedback. 

So that, then, is what submit.cs50.io looks like. It's right now showing me Margaret's latest submission. But submit.cs50.io allows for students to submit multiple times if they want to. I happen to know that Margaret submitted this problem a little earlier today, too. So up here in the upper right, I'm currently in the latest mode, which is only going to show me each student's most recent submission. But if I toggle that switch, switch to all mode, instead of just seeing one submission from Margaret, I'm now seeing both of her submissions. And I can see that 10 hours ago she submitted as well and only got eight out of the 11 checks correct. Now five minutes ago, she got 10 out of the 11 checks correct. So I can see that she's improving, that she submitted again. She's made some changes. And now it looks like a submission is better than it was last time. 

So this tool, hopefully, will allow you to more quickly be able to get access to the results of your student submissions, to be able to comment on code, to get the results of correctness checking on students' code and more. And if you want to export that data, so that you can use that data in your own grade book, or you want to process that information in some way. You can always export all of this submission data here. It'll be downloaded as a CSB spreadsheet file that you could then open in a text editor or in a more traditional spreadsheet program, something like Microsoft Excel or Apple Numbers or Google sheets. All of those are possible options for viewing that data and then using that data in whatever way you want. 

So that, then, is a look at submit.cs50.io. And before we move on, any questions for me or Margaret about anything there? 

AUDIENCE: OK, so my question is how do you? And this is for Margaret. How do you manage the grading of students of submissions in order to optimize your time? And what kind of tricks or workflow rules have you developed over Time 

MARGARET TANZOSH: So what I do for my 100 or so introductory students, I assign a grade based on correctness and style. I give feedback on design, based on what I see many students submitting something that's maybe not efficiently done. But I'll assign the grade based on correctness of style. 

So what I do is I download the CSV file of all of the submissions of all of my students. And then I have a Python script that goes through, selects the most, the highest score of all the submissions, and then kind of calculates it based on 75% correctness, 25% of style, adds together. They get everything. They get 100%. And then I map that in my CSV file to their student ID numbers, so I could upload it to my school grade book. 

So it saves a lot of time being able to handle that type of grading, without having to individually kind of type things in over and over. So I find that's been really helpful. And there are many different ways to do it. 

I also find it really terrific that, as Brian was showing, he was able to put kind of the root of the slugs for like the CS50x course in his settings panel in that particular class. And so I'll use many of the CS50 problems. But I'll also create some of my own. So I'm able to create my own roots for the slugs that I'm using. And then any of the problems either the CS50 problems or the problems I've created, all show up in the submit50 grade book. 

So it's just been a great way. And also, the students are getting immediate feedback because when they go and they look at that link that's created, once they do their submission, they can immediately see what their score is. And they have the opportunity to resubmit. I let them submit basically as many times as they want until the end of the particular unit that we're on, so that they can continue to make their code better and better and to improve their grade. 

So do we have a good question? Do I see [INAUDIBLE]? Did I pronounce that correctly? 

AUDIENCE: Hi, I had a question about just what the students can see. Can they see the detail, like the style report? So they couldn't just notice that the multiplication [INAUDIBLE] space. And they just kind of fix it right afterwards. Or does the teacher have to go and combine them. [INAUDIBLE] 

MARGARET TANZOSH: Sure. let me go back. Let me go back, and I'll share my screen. And I'll show you exactly what it looks like on the student side. 

So on the student side, there were a couple different tools that students can use to help them take a look at how their work is going to be ultimately graded. So if you can hopefully see on the bottom of my terminal, instead of just typing it. And most of the program specs we'll have this. But like I typed in the Submit50 before, I could also type in, for many problems, for all CS50 problems, the check50's not turned on in some of the more advanced products. 

I think CS50 team really wants students to be able to look to test their own code, without having to depend on a series of tests. But through some of the introductory problems, that check50 is turned on. And in the same way as a student was able to get their results graded with the submit50, they can see all the checks here. And you could see which ones ran correctly and which ones did not run correctly. 

And so my students love when they see those green happy faces, makes them very happy. And so it gives them a clue as to how their score is going to work out. And then in addition, they can go to the link here, which is very much the same as the link that we saw before. And if they click on this, they can actually see the same report that Brian saw. So this is going to give them a little bit more detail to see, oh, exactly what kind of the test was run when that input was rejected. 

So this gives them a little bit of a clue as well as to what their score is going to look like. And then there's also style50. So style50 works with just the source code. So it's always going to be a style50 and then the source code, which is going to be cash.c. And here again, it's very much the same report that Brian saw, but from a student perspective. 

So many my students will check their code. And before they submit it, they'll run the style50. They'll fix their indentations. And so they can pretty much assure that they're going to get a pretty good style grade before they submit it. But before running check50, sometimes I find that students become a little bit too dependent on running check50 and attempted to develop their code. And before they even run it and test it themselves, start running check50. And I really try to stress that it's very important to run your own code, so that you really get a sense of how your program interacts with you. Because otherwise, you're not actually seeing those interactions and that little bit of a user interface, right? Asking for the prompt. What does the prompt look like? What does the output actually look like? 

So anyway, yes. So the student can certainly see all these things, as well, as the teacher. All right do we have any other questions here before I move on? 

BRIAN YU: All right, I think [INAUDIBLE] still has his hand raised, if you'd like to ask a question. 

AUDIENCE: Can you hear me? 

BRIAN YU: We can hear you. 

AUDIENCE: Yeah, just I would like to introduce myself. My name is [INAUDIBLE]. I am a lecturer in the King Fahd University from Saudi Arabia. And I've been following this workshop in a very detailed manner. And it has been very, very helpful, especially your tips on reflections yesterday. 

My question is, I was just wondering about the overall structure. So the instructor account, is it a separate account on GitHub? How do you differentiate? Would I need to log in as a regular user? Also, I was wondering about the slug. So does it correspond to an actual folder structure, or it's just a name. 

So I'm just interested in more details about all of this setup that you have. Thank you very much. 

[INTERPOSING VOICES] 

BRIAN YU: Sure, I can take that one. Yeah, so in short, there is no there's no notion of a separate teacher account on submit.cs50.io. Every account is just a GitHub account. But what we do have on submit.cs50.io courses, and every course can be associated with one or more teachers of that course and then 0 or more students, for that course. So when you create a course on submit.cs50.io, you automatically become a teacher for that course, which means you get to view all of your students' work for that course. 

And you have the ability to add other teachers if, for example, you have a teaching assistant in your class, for example, that you also want to grant access to all of that work. And then when students click on the invitation link, they become students of that course, as well. But the account itself doesn't have any notion of whether it is a student or a teacher. That's all of course specific, such that Margaret, for example, was just a student in my course. But she very well might have been the teacher of other courses that she teaches on her own. So she's sort of simultaneously a student in my course, but also a teacher of other courses that she teaches, for instance. 

As for the structure of what the submission slug means, I mentioned this. Let me go ahead and show you what that looks like. Give me one second just to pull it up. But in short, the submission slug is a combination of a couple of things. The submission slug is a combination of a GitHub repository, a branch on that GitHub repository, and then a folder that is on that branch. 

So for example, in the case of the submission slug for Cash, you'll see that so for CS50 problems 2020x Cash, CS50/problems, that is the name of the GitHub repository, where we have stored all of the checks for all of our problems. So if you go to GitHub/CS50/problems, you'll find all of our checks. 

Every GitHub repository is divided into branches and we generally. Use those branches to separate out different versions of the course, like the 2019 offering of the course as compared to the 2020 version. So the second part of the submission slug, 2020/x, that is the name of a branch of that GitHub repository. So that is the branch where you'll find the checks. And then on that branch are a whole bunch of folders, one folder for each problem. And so the last part of the submissions slug Cash here represents a folder on that branch. 

So ultimately, the unique identifier submissions like taken in the whole first has the repository, then the branch, then the path that will lead you to the correctness checks. And for more details about that, you can take a look at the session from two days ago that I gave on submitting and grading, which went over this and how to actually write checks of your own. And a draft recording of that session is already available online, if you'd like to take a look at. 

MARGARET TANZOSH: OK, thank you, Brian. That was great. Any other questions before I move on? 

OK, all right so I am going to go back and talk a little bit about CS50 labs. CS50 labs I find to be enormously helpful for helping to differentiate in my classroom. So as I mentioned, I have a wide range of students. I have 100 and something introductory students. I might have 30 AP students. And my introductory students kind of range from students who might be sitting and taking Calculus BC right next to a student who might have failed algebra and was added to my class because they really need the math credit and has never programmed before. 

In fact, most of the students that come in my class have done minimal, if any, programming before. Some take to it immediately and just love it and spend half their night working ahead on all the programs. So I like to differentiate and make sure that everybody is working at the level that they are able to do, able to be engaged. 

So the labs are really helpful for the new students, starting out. I find it really helps reduce the cognitive demand because it takes away it kind of the need to manage certain things of the terminal. So the terminals are still there. And I know you've seen examples. And I'll show you examples of some of the labs that I've created, as well. The terminal is there. So they still have to have that real world kind of sense of compiling code and executing code. But they don't need to, for instance, manage the directories in their terminal, which I find for new students can be a little cumbersome. They sometimes forget to change with the right directory. 

And if I have, say, 30 students sitting around a classroom and I'm trying to help someone who's struggling, students on the other side are, oh, how come I can't run my code? My code won't compile, blah, blah, blah. And it's often just because they're in the wrong directory. So this kind of takes away those issues while they're just getting used to the whole environment, right? The whole programming environment. 

And the labs can also start with starter code. So as you'll see, the labs, and some of the CS50 problems that have been created as labs also will come maybe with the C libraries already included or the main function is there, so that they could just kind of really focus on the logic at the very beginning and not get caught up with too much of the stuff that's not quite as intellectually interesting, but stat with really kind of thinking about how to get started on their problems. 

And it also allows me the ability to do a lot of scaffolding. So the way the labs are created gives us the ability to have these little hidden hints, where they can click on a button. And I might actually create pseudocode and give students who don't know how to start a problem a couple of steps. This is what you do need to do first. I'll embed instructional video, sometimes, kind of hide them under a button, so that students that are kind of moving at a good pace and don't want to get-- they don't want to get too much information. 

I actually have some students that don't want anybody to tell them too much how to do. They want to struggle and figure it out on their own. They don't want too many hints. So for them, they don't have to see all these other bits of pieces that are there, the pseudocode, the videos. They just delve right into it and figure it out on their own. 

So it really gives me the idea, the ability to really customize it that way. Also, the way that the labs are presented, there's not this very long program spec that, for certain students, could be a little bit overwhelming, when there's so much to read. A little next button in the labs is extremely helpful because it allows me just to present a little chunk of information at a time. 

And of course, it's totally customizable. And I use the labs not only for the CS50 problems, as you'll see. I've converted many of the CS50 problems into the labs for my own students, using my own students, to introduce them. So these three students, I used my phone, made a little YouTube video. They introduce the problem. And while they may not be as eloquent as Brian in talking about how the problem is actually going to be created. It's very engaging the students in my class to see their classmates doing it, right? It's like, oh, I know him. If he can take this class and he can succeeded at this class, I probably can do it, too. 

And so I find that, again, it just helps the students get a little more connected. And it just makes it kind of a cool thing to be able to do. And you could see here, on the lower right. So once we kind of click on the Next button, the lab scrolls up a page at a time and. This little spoiler here give some ideas on what type of start you might have for degrees Celsius. 

Another thing that you could do in the labs is give a text file, so that students could write some notes or some pseudocode. And what I'll do is, I'm, sitting around that table with the struggling students and helping them get started is that will open up a lab and open up the pseudocode tab and start to write down some steps, step by step. What exactly you do to get started on this problem. So there could be a lot of fun. 

The labs I pretty much run, for my first semester, at least for my introductory students, I pretty much convert everything into labs. There are some existing CS50 labs that have been used. And I use those. Sometimes, I tweak them a little bit to add a little more scaffolding and then also create any of the other problems as labs. And then also, sometimes, I'll add my own. 

I even created a few labs beginning of semester two, we'll do some HTML and CSS before they get started on the home page project. And I've even made a few of those. So students can again have a little bit of a place where they could sit and really kind of begin to absorb the material, independently, when I'm not available to come over and help them. So that tends to be something that I rely on quite a lot in my class. 

And then I've also created quite a number of labs that are instructional in nature, that teach students about specific programming constructs. So here, again, I might have a student, for instance, that's working, they started one of the unit two programs and they forget how to use command line arguments. And so I've created a lab on command line arguments, why you need to test with the number of command line arguments, what happens when you don't. And so I will introduce these again on the-- I think I mentioned yesterday, I create a website, where I have links to all the problems. And I also have links to all the labs. 

And in these labs, I will kind of point students in those directions, when they're stuck and be, OK, take a look at this lab. See if that can kind of help you with that. And then in addition to the description of the left, maybe do embedded videos. Sometimes, I'll embed animated gif files, so that they can see like an animation, how does Bubble Sort work. I'll have a little program for them to solve on the right, where a lot of the code is supplied. And they're really just completing a couple of lines of code. So it's not a whole day long exercise. It's 15 minutes they can come. They can get some. Practice and they could kind of master a particular concept. 

So I've put the link in the slides. And I can add this link, as well, to the chat window, if you're interested. But this is a list of all the labs that I have here, so far. And could take a look at a couple of these, if you like. So for instance, I have information about what are different data types. So again, for some students that are just getting started out, what do we mean by a data type? Why do we need to include the CS50 library on top? Here we have the little Next button and some of the differences between native data types and some of the eventually, some of the CS50, examples of them, how do you get inputs for them, et cetera, et cetera and then some practice. 

So then I have a little bit of code here, on the right side, where most of it is already completed,. And they're just during a few of these little bits of code under each of the comments, declaring an end. What does that look like? How do we get user input using Get End and so on. And then, as well, it will lead them to a reference sheet. 

We'll take a look at what else I have here. I'll have, for instance, so different kinds of loops. What is the point of a for loop. And again, examples, how, for maybe students it might have been introduced initially to scratch, how does that compare to a scratch programming construct. And then here's the idea of the little hint. Here's something taking a moment to load. But it might be a little bit of an idea. Keep in mind, you could use, I, in calculating something inside of your loop. So again, a number of different things to look at. I have, as well, somebody created little animations for to take a look at how does linear research work. 

And I've created these. I've used the research and sorting labs from my AP students, so that they could get a sense, also, of computational complexity, which are discussed in here. And just like a little bit of an animation to kind of visually show what the actual algorithm does, how it works. And then on the right, I created this little program of a bingo card with numbers. And they're looking to see if the number's on the bingo card by completing the function here to do linear search. 

And so, again, they're not having to worry too much yet about even how to pass an array to a function. But they get to actually see the code that's done, which is also instructive in nature, and get to kind of test out their own skills in seeing if they could create that particular code. 

And then finally, I'll have various different-- let's see. What do I have for insertion sort, as well, where I'm trying to kind of show the difference between how some of these various sorting algorithms work, and how we're searching for the 6 was in the right position and et cetera. We're kind of inserting into the array in the right place. And then again, they need to develop the insertion sort algorithm here. 

And for my students, my AP students, I created my own submit50s. I didn't add them into the lab, but I put them on the website. And they actually had to submit these and that went toward their class were grade, as well. Because this was part of the practice, but I required that they do as an AP course. And then also in here, talking about computational complexity. And what does that mean, et cetera, et cetera? And then their little hint over here, where I give them quite a bit of pseudocode to kind of figure out how to get started. 

So I even made one with a file IO. And I think I went as far as something like unit 4 with images, where it was very inspired by one of Brian's new problem sets, where he has some image program, where students are making filters. So this is just, again, all the code is given on the right. And they're basically going to be changing the little Smiley face here to a different color. And just waiting for this to come up. Just a moment. 

And again, the nice thing is that they don't have to worry about downloading code and unzipping it. They can just get started right away. Everything is right here. They're just completing a function in the helpers file. And all the other bits and pieces are already existing. 

So again, it's just a way for students to get extra practice. And I've also included a link here to the repository, which if anybody wants to fork it, I'll be eventually-- there's that share link that I'll be sharing, as well. Eventually, sending it to CS50, they're going to be curating various teacher resources. But if anybody wanted to take a look at that. And just to take a look at me creating the labs and stuff. I know there is going to be changing. But I'm sure much of the structure will be fairly similar. And it's really just a read me file in a GitHub markdown. And then some special codes that allow you to make the Next buttons. 

Really easy to embed videos, to embed images. And then I just include the files that I'm going to have come up on the right side in the lab, that they're going to use as starter code. So here for instance, this would be the code that is going to be coming up once I open up the lab. 

So I really find that by using these, it just allows me to really kind of customize, for my students, what I need to do to help them be self-sufficient, so that they can all succeed. Even though I may need to spend more time with certain students, the other students have the resources there that help, hopefully, explain things, the kind of things that I've seen that come up over and over again. So that's, basically, what I've been doing for the labs. 

So let me just stop for just a moment and see if anybody has any questions. 

AUDIENCE: So the question is, regarding working with the students, my question is regarding working issues [INAUDIBLE] the labs, my question is, is there anything you have learned from producing the labs for students that [INAUDIBLE] apply inside of the class? 

MARGARET TANZOSH: I'm not sure I heard you fully. You're asking me, is there anything that I've learned from the creating the labs that help the students learn in class? 

AUDIENCE: No, creating the labs with these students. 

MARGARET TANZOSH: With the students? 

Well, I haven't created, but like the students are not seeing behind the scenes, like the GitHub code that I've created, if that's what you're asking. I created the labs. And then I've presented them, as you saw them, not through the markdown code. But basically, the way the lab is presented, right? With the instructions on the left and the coding environment on the right. 

So again, I apologize. I'm not sure I fully understand your question. I've tried to come up with topics that I've seen students struggle with that they might sometimes feel unclear about how to move forward with and then have additional explanation in the labs. So I don't require that they buy books. And sometimes online references, particularly in C, can be a little confusing and overwhelming for a new student to look at. So I try to kind of take it to the basics and introduce the concepts, without students having to get overwhelmed with certain things to see when they look up online on their own. They'll maybe get caught up with pointers and even input functions, scanf, et cetera, which could be a little bit much a student that's new to coding to have to deal with. 

So I don't know if I answered your question. Do you want to add onto that? 

AUDIENCE: What I mean is like-- 

MARGARET TANZOSH: Oh, the students that appear in the videos? 

AUDIENCE: Correctly. The students-- yeah, I mean, working the students to record the videos, guiding them through [INAUDIBLE] Is there anything you have learned from that? 

MARGARET TANZOSH: Oh, well, basically, I'll work with them, and they'll create a little script. And then I'll take a few takes, just take a little video on my phone. And then when we're both happy with it, I'll uploaded to YouTube. And at that point, it's very easy to embed it in the lab. 

So sometimes, you're shy. So it takes a few times. And I think it's worth it. And I think that the other students enjoy the opportunity to see their classmates present the problem, as well. 

So all right. Any other any other questions about what we've seen so far, the labs, the types of customization you could do with that. 

Well, I'll post a link to the GitHub repository. Sure. I'll put that actually right now in the-- I'll post that right now in the chat window for anybody I'd like to see that. 

All right. 

Yes, question? 

AUDIENCE: Yes, I just was looking at some of your lab examples. I have two questions. First, we saw a very nice animation in the insertion sort example. So it does not require any special software, or could you share your experience? And secondly, yesterday, you were talking about some coronavirus related projects or programs. So could you share your experience on this? 

MARGARET TANZOSH: Sure. So with the animations, I used After Effects to create those. I started making them in PowerPoint. And it was not quite giving me what I wanted. So I used After Effects, so that I could have a lot of control, in terms of how the numbers are moving around. 

So that was how I did those. And then I exported them as video. And I turned it into a gif. And then when I imported it, using the same exact syntax, just to import a standard image in the readme file, it automatically animates. So just as if you put a gif up, put a gif up on the HTML, right? Up on a web page. 

So I did that. And then I'm sorry, your other question was-- just remind me the second half of your question again. 

AUDIENCE: Yeah, you were talking about? 

MARGARET TANZOSH: The Simulation. Yes, sure. 

Well, I can show you what I did. So I created a-- I'll go back to share my screen. I was very inspired, actually, with a virus. So I was inspired by another teacher at my school, who teaches middle school computer science. And he had created an animation of, based on a Washington Post article, about how people can infect each other, when they come in contact with each other. 

And so I redid it, using-- this was done in JavaScript. And so I did two different things. One of the projects was to have students recreate some of this. And then another part of the project, for students that maybe didn't feel as able, toward the end of the year, remote learning, or maybe didn't feel that they were able to do quite as much programming, I had them actually just use this as a reference to talk a little about how a computer simulation could happen. 

So what I did was that I put in various different things that could be controlled. And then I used chart.js to create a chart. And again, the idea being some ideas on how you could change, use a model, to get ideas on how to stop a virus from spreading. 

So that was what that was about. And then I just had them do a little essay on doing additional research on how computer modeling could be used, computer modeling and simulation. What is it? And how could it be used, which, for the AP class, is actually a topic of the AP class that the AP curriculum explores. But I think a good one because it also just brings kind of to the real world application of one of the ways that we could use these programs that we're creating. So that was that was, again, trying to keep it relevant. And something that people are going to be feeling that they can connect with. 

So all right, do we have any other questions? [INAUDIBLE] Yeah, it was fun. Again, it was so not totally originally. It originally came from a similar simulation done by the Washington Post. But again, I think it shows an example that people are concerned about it and that the students could relate to. So I'll usually try to come up with a topic. I'll create writing problems. I've actually even used the labs for students to do writing problems there, where rather than have to have another portal or another submission system for them to submit their writing assignments, I have used labs, at times, to have the description of what they're supposed to write on the left. And then I'll just have a text file already up. They just fill in their name, their class period, their date. And then they could just write in their text file and then use a Submit50 to submit that, as well. 

So I've used them for a lot of different purposes. Again, just trying to reduce the confusion of students having to submit one thing here and one thing there, to kind of just keep it consistent throughout the semester. So those were some of the things that I did. 

Any other questions or comments? 

All right, so coming back here. So again, that was about the labs. And one of the things to keep in mind is that-- and you may have heard this earlier in the week-- that Harvard is giving us this link, this email, to send off any teacher created problems or labs or activities or assessments, how they're going to be curated by the 650 team. And then they'll be posted online, so that they can be shared, which I'm certainly looking forward to seeing to get ideas on new activities and new problems, of different kinds of assessments to use. Particularly if we start out the year during remote learning, I'm going to be rethinking myself what types of assessments I'm going to do, regarding quizzes. 

So this is something to keep in mind. I'll be submitting the labs to this email. It's been a busy week, but I'll be doing that shortly. And yeah, so hopefully, that will be another resource for everybody to be able to have access to. And then finally, the last topic that I was planning on talking about is a little bit about assessments. 

So CS50's kind of changed the way they do assessments through the years. This first link is for their quizzes. And you can see here on their website, they have these quizzes now that, for every week after the student watches the lecture, they have a limited amount of time to complete the quiz. The quizzes in Google Docs, and they're the kinds of questions that it's not so much about memorizing syntax or even having something that you're easily going to immediately look up, but a little bit more conceptual in nature. 

Here, about taking a look at maybe something that might involve integer overflow or integer underflow, taking a look at various different ways of creating code and asking why one is more efficient than the other. So I think these are really interesting ideas. And they've given us the link here, so that we can incorporate these quizzes, if we want to, or these ideas, into our own classes. 

Back in the past years, the types of problems were a little bit more standard, where it might be complete a function to do something or something along those lines. And initially, when I started out my class, as well, I used more traditional kind of exams like that. And I have been experimenting with different formats. So some of the link here basically goes to a folder on my Google Drive, where I could show you the initial quizzes. And this idea with the light bulbs, this was actually stolen from an old CS50 quiz. I steal as much as I can from them. I always have because they're so cool. 

But I thought that was a fun question. Students are trying to figure out how to create a binary number by turning light bulbs on and off. And I would give these a couple of times a semester or basically, at the end of each unit, in the classroom, as a four period quiz. 

Now the thing about my class is that my periods got 45 minutes. And in past years, I've had more students with disabilities that really require extra time, sometimes even double time when there's any kind of a full period exam. So since my class is elective, as this began to be an issue, I became concerned about having to pull a student out of maybe their math class or out of their social studies class. And they need that class to graduate from high school. And they have to pull that student out of the class and have them go to a different location. 

So instead, what I tried to do in more recent years was to create a very short multiple choice quiz that I offer a half a period for. It's a 20 minute quiz. And I did that by having still 10 questions. But it takes a lot less time for a student to bubble in something on a multiple choice form than it does to actually write in longhand. So this quiz, this was actually for like unit two. A unit one quiz was actually called quiz 0, unit 2 practice. But I didn't quiz on scratch. 

So this was quiz 1, corresponding to things like strings, command line arguments. And I would very carefully Proctor around the classroom, so they were spaced apart from each other. I'd have multiple versions of the quizzes that I'd give out. This little zip grade thing is really cool. It allows you to literally use your phone to scan in the answer sheet. And it automatically saves it and let you download it as a CSV file for everybody's grades. It allows me to give the students an instant grade, as well. 

So the advantage was, of the multiple choice, which I'm not 100% sure I'm going to stick with it. But the advantage was I would make it a 20 minute quiz, and then students that need a double time could just finish it during the period of a class. In other words, I didn't have to pull them out of the second class. 

For my AP students, again, I initially started out with kind of long form questions, things combined with both some coding problems that they had to write. And these generally took, along with some multiple choice questions that pretty much came up with some of the practice AP exams. But again, it was sometimes hard for students to finish in 45 minutes. So I also went to multiple choice last year. These little diagrams are the kind of format that they might see on their AP exam. 

So I combined CS50 type questions with AP style questions. But I'm going to certainly be rethinking what I'm going to do if we do indeed start out remote. Because it's just too easy for students to just share with each other the multiple choice response. I mean, I could jumble up the questions, I guess. But I'm going to be rethinking that. So if anybody would like to share assessments they've done, I'm really looking to get other ideas, as well. 

And then CS50 is going to be putting together a number of their old quizzes and creating a quiz bank, so that we have one repository to go to get some old quiz questions that we might incorporate or our classes, as well. So I've always kind of been snooping around through some old quizzes available, at CS50.tv, going back before 2017, where I've stolen some of those old quiz questions. But in any case, that should make it much easier to find some things and to hopefully be able to have some new ideas for the future. 

So I see Ramon has a question. 

AUDIENCE: Yeah, so I would like to ask you if you could share with us some any success stories of past students, not in terms of pursuing a career in computer science, but in terms of like a shy and not very confident student that ended up becoming very confident in the class at the end of the course. And I would like you to tell us which of the customizations of the course that you have done do you think has played a key role in those students development? 

MARGARET TANZOSH: Yeah, well I've had a number of students that have started with feeling intimidated about getting started programming. As I think I mentioned yesterday, I often have them do a big reflection at the end of the semester one, where they write about their development as a computer scientist. And it's very interesting to me to kind of see how they see their own growth and development from where they started on day 0 to where they've come at that point in time. 

But I have had students that it's been very exciting to see their growth. Last year, I had a student, an IEP student, who was a bit of a struggling student in my introductory class. He was very determined and went on with my AP classes here. Actually, more than one. I had at least two or three students like that that kind of struggled at the beginning and were able to get help from peers and did fine in the class. 

But it took a lot of support to get them there. But by the AP class, they were really quite competent programmers at working on the CS50 problems. And they're planning on possibly-- they're seniors. They graduated this year. And are possibly looking at even majoring in computer science. So we'll see if they stay with it. But I've had a number of students. And I'm proud to say, also, a number of girls, who came into the class and weren't sure if this was something they'd like and actually ended up moving on and deciding to computer science in college. 

So there's quite a number of interesting stories that way. And one of the things that help, I think, well, I guess my own philosophy is that I try to just really be there to support the students. So I try to give them every chance they can to succeed. I really stress that it's really about just sticking with it, persistence, that everybody gets the stuff at a different time in their life. You could take a few months and then you catch it and you got it and now all makes sense. 

So I'm just trying to stay really positive, being there for the students, being there to support them if they need it. If they want to come in for lunch for extra help, I'll make that available to them, whatever I can do to help them succeed. So I don't know. I mean, you know, I think certainly the scaffolding has helped in the past few years that I've had more special Ed students. It allows me to spend more time working with them and allowing students to be more independent, so that I don't have to be kind of running around the classroom quite so much with the little questions. I could focus on some of the bigger issues, kind of thinking through algorithms, instead of just, why can't I compile my code because I'm in the wrong directory, things like that. 

So I don't know. I mean, I'm still learning myself. I'm always trying to come up with new ideas and always trying to improve. But it's been fun. It's just been a lot of fun to see all the students developing and see how many of them actually want to go on afterwards with the subject. So anyway. So I don't know if I've answered your question, exactly. But that's been kind of my experience with the students. 

All right, so I think we're going to wrap it up, if nobody has any other questions. It was really fun meeting you. You guys had some great questions. Thank you so much for hearing what I had to say. It was a lot of fun. So good luck to you all. And let me know if you have any other questions. Stay in touch.