1 00:00:00,000 --> 00:00:01,436 2 00:00:01,436 --> 00:00:03,936 JASON HIRSCHHORN: Good morning, everyone, or good afternoon, 3 00:00:03,936 --> 00:00:07,370 or thanks for getting up in the middle of the night, 4 00:00:07,370 --> 00:00:08,760 depending on where you are. 5 00:00:08,760 --> 00:00:14,420 Today's schedule per the email that David had sent out to everyone, first, 6 00:00:14,420 --> 00:00:17,720 for an hour this morning we're going to talk a bit about assessments 7 00:00:17,720 --> 00:00:21,020 and we're going to spend the, hopefully, the bulk 8 00:00:21,020 --> 00:00:24,200 of this time giving you a chance to look through a lot of assessments 9 00:00:24,200 --> 00:00:26,480 that have been written already so you can get a feel 10 00:00:26,480 --> 00:00:29,930 for the flavor of the assessments, which I think 11 00:00:29,930 --> 00:00:34,520 do a nice job of communicating sort of the overall vision of how 12 00:00:34,520 --> 00:00:36,710 to engage students. 13 00:00:36,710 --> 00:00:40,940 And then from 1 to 2 PM Eastern time, we'll 14 00:00:40,940 --> 00:00:44,690 have an overview of all the possible courses 15 00:00:44,690 --> 00:00:48,238 you can teach before or after CS50 AP. 16 00:00:48,238 --> 00:00:50,030 We'll take a break for a bit and then we'll 17 00:00:50,030 --> 00:00:56,500 reconvene later today for a panel featuring three CS50 AP teachers, who 18 00:00:56,500 --> 00:00:59,120 will each talk a bit about their schools, 19 00:00:59,120 --> 00:01:01,840 they'll talk a bit about their history of teaching 20 00:01:01,840 --> 00:01:06,890 CS50 AP, some specific things they do to implement the curriculum, 21 00:01:06,890 --> 00:01:10,580 and then also about what remote learning has looked like from them. 22 00:01:10,580 --> 00:01:13,195 So you can get sort of a sense of all of these teachers 23 00:01:13,195 --> 00:01:16,070 and what it's looked like on the ground, and then have plenty of time 24 00:01:16,070 --> 00:01:18,930 to ask questions to them or to the larger group. 25 00:01:18,930 --> 00:01:21,230 So that'll be this afternoon's session, unless you're 26 00:01:21,230 --> 00:01:24,630 attending Megan's wedding anniversary, in which case you'll be at that. 27 00:01:24,630 --> 00:01:28,650 But otherwise, we have exciting programming for you. 28 00:01:28,650 --> 00:01:32,352 I'm going to jump in right now by sharing my screen. 29 00:01:32,352 --> 00:01:34,310 This morning, as mentioned, we're going to talk 30 00:01:34,310 --> 00:01:37,520 a bit about writing assessments. 31 00:01:37,520 --> 00:01:41,900 If you weren't here yesterday, this is a bit about my background. 32 00:01:41,900 --> 00:01:44,690 If you forgot, here's a bit about my background. 33 00:01:44,690 --> 00:01:49,050 And of course, that's not what I'm going to look like. 34 00:01:49,050 --> 00:01:52,550 I thought this was funny yesterday so I decided to do it again, put my mask on. 35 00:01:52,550 --> 00:01:53,630 But also, it's not funny. 36 00:01:53,630 --> 00:01:56,600 You should probably wear a mask when you're outside. 37 00:01:56,600 --> 00:01:59,810 A note about my school, too, that's not on here 38 00:01:59,810 --> 00:02:02,870 is they expect, at the end of every unit, 39 00:02:02,870 --> 00:02:06,410 you to have some sort of summative unit assessment. 40 00:02:06,410 --> 00:02:08,449 So when I was creating this course, I had 41 00:02:08,449 --> 00:02:13,640 to create unit assessments that would take place let's say every month or so. 42 00:02:13,640 --> 00:02:16,580 So I had to have something like that, which is-- 43 00:02:16,580 --> 00:02:19,210 and you'll see examples of those later today. 44 00:02:19,210 --> 00:02:22,430 CS50, which I'll mention in a bit, doesn't do assessments 45 00:02:22,430 --> 00:02:24,230 quite like that at the college level. 46 00:02:24,230 --> 00:02:27,240 But again, we'll talk about that in a few minutes. 47 00:02:27,240 --> 00:02:28,302 Here's our agenda. 48 00:02:28,302 --> 00:02:30,260 I'd like to start with giving everyone a chance 49 00:02:30,260 --> 00:02:32,300 to meet some more of their neighbors. 50 00:02:32,300 --> 00:02:35,458 So in a bit after this slide, you'll be put into breakout rooms, 51 00:02:35,458 --> 00:02:37,000 but hopefully with a different group. 52 00:02:37,000 --> 00:02:37,830 It's pseudorandom. 53 00:02:37,830 --> 00:02:40,535 So you might meet some of your same friends, but that's OK, 54 00:02:40,535 --> 00:02:42,660 there are some new questions for you to talk about. 55 00:02:42,660 --> 00:02:45,118 But hopefully, you'll get to meet some more people as well. 56 00:02:45,118 --> 00:02:49,800 Then, per the feedback we received from earlier in the workshop, 57 00:02:49,800 --> 00:02:54,380 Brian and I are going to demonstrate the features of submit.cs50.io 58 00:02:54,380 --> 00:02:57,473 from the teacher and the student perspective. 59 00:02:57,473 --> 00:03:00,140 Then we're going to introduce this brand new resource to you all 60 00:03:00,140 --> 00:03:05,960 to help us moving forward, leverage everything that your fellow teachers 61 00:03:05,960 --> 00:03:08,690 have built and CS50 has built. And then you'll 62 00:03:08,690 --> 00:03:12,420 get a chance to dive into and take a look at tons of assessment, 63 00:03:12,420 --> 00:03:17,180 so you can get a flavor of the types of questions that fit nicely with the CS50 64 00:03:17,180 --> 00:03:17,990 curriculum. 65 00:03:17,990 --> 00:03:22,100 And finally, if you want to come back, there'll be a chance for a Q&A 66 00:03:22,100 --> 00:03:23,600 before the next session. 67 00:03:23,600 --> 00:03:28,760 As before, if you have a question, feel free to raise your hand. 68 00:03:28,760 --> 00:03:32,390 Sometimes it's easier to see that and it's easier for people 69 00:03:32,390 --> 00:03:35,030 to be able to pay attention to your question if I pause 70 00:03:35,030 --> 00:03:39,170 and give you the floor versus if it's scrolling by in the chat. 71 00:03:39,170 --> 00:03:43,140 But first, of course, let's meet some more of our neighbors. 72 00:03:43,140 --> 00:03:45,368 This time, I'd still like you to introduce yourselves 73 00:03:45,368 --> 00:03:47,660 because hopefully you'll be with a group of new people, 74 00:03:47,660 --> 00:03:49,790 but also a new prompt for you-- 75 00:03:49,790 --> 00:03:53,150 I'd like you to share a food you love that you've had recently 76 00:03:53,150 --> 00:03:54,692 and why you love that item. 77 00:03:54,692 --> 00:03:56,900 Of course, you can ignore these instructions entirely 78 00:03:56,900 --> 00:04:00,570 and talk about whatever you like, but here are some things to get you going. 79 00:04:00,570 --> 00:04:04,100 And once again, we'll give you about five minutes in those breakout rooms 80 00:04:04,100 --> 00:04:05,765 to get to know some of your neighbors. 81 00:04:05,765 --> 00:04:06,052 82 00:04:06,052 --> 00:04:07,760 While we ponder that, Brian and I are now 83 00:04:07,760 --> 00:04:12,870 going to transition to a demonstration of the CS50.IO tools. 84 00:04:12,870 --> 00:04:17,130 Or submit.CS50.IO tools rather. 85 00:04:17,130 --> 00:04:22,760 Again, a few days ago Brian demonstrated Check 50, Style 50, Submit 50, 86 00:04:22,760 --> 00:04:25,130 and Submit.CS50.IO. 87 00:04:25,130 --> 00:04:29,180 And there were questions on what exactly does this look like from a teacher's 88 00:04:29,180 --> 00:04:31,280 or a student's perspective. 89 00:04:31,280 --> 00:04:34,032 When I'm setting up my class, when I'm having students 90 00:04:34,032 --> 00:04:35,990 submit using these tools, and when I'm checking 91 00:04:35,990 --> 00:04:38,440 their work using the automated system. 92 00:04:38,440 --> 00:04:42,110 So right now, Brian and I are going to do a live demonstration for you. 93 00:04:42,110 --> 00:04:44,270 Brian's going to be the teacher of a class, 94 00:04:44,270 --> 00:04:46,550 and I'll be one of his enrolled students. 95 00:04:46,550 --> 00:04:48,410 So I'll hand it off to my teacher. 96 00:04:48,410 --> 00:04:49,243 BRIAN YU: All right. 97 00:04:49,243 --> 00:04:50,270 Thanks, Jason. 98 00:04:50,270 --> 00:04:53,420 So once you set up a Submit.CS50.IO course, 99 00:04:53,420 --> 00:04:56,240 I know that some people were having some technical issues doing so. 100 00:04:56,240 --> 00:04:58,490 It looks like there might be an issue on GitHub's end. 101 00:04:58,490 --> 00:05:00,980 We've reached out to them to try and sort out that problem. 102 00:05:00,980 --> 00:05:03,770 But hopefully we should have that working for everyone shortly. 103 00:05:03,770 --> 00:05:05,803 But once you set up that course, you'll end up 104 00:05:05,803 --> 00:05:07,970 with a page that looks a little something like this. 105 00:05:07,970 --> 00:05:11,180 And I'll sort of walk through the key features of it. 106 00:05:11,180 --> 00:05:13,580 Up at the top is where you can name the class. 107 00:05:13,580 --> 00:05:16,460 So usually people name it after, like, the year or the class period 108 00:05:16,460 --> 00:05:18,030 or something like that. 109 00:05:18,030 --> 00:05:20,360 And then down below, this is the invitation link. 110 00:05:20,360 --> 00:05:22,670 And so you can send this link to your students. 111 00:05:22,670 --> 00:05:26,120 And when students click on that link, they'll be prompted to join the course. 112 00:05:26,120 --> 00:05:29,420 And what that means is that they will be granting you, 113 00:05:29,420 --> 00:05:32,840 the teacher, access to all of their submissions 114 00:05:32,840 --> 00:05:34,520 that they submit through Submit 50. 115 00:05:34,520 --> 00:05:37,430 And Jason will show you what that looks like in a moment. 116 00:05:37,430 --> 00:05:40,280 Beneath that are all of the slugs, otherwise 117 00:05:40,280 --> 00:05:42,290 known as just unique identifiers, for all 118 00:05:42,290 --> 00:05:46,820 of the problems that you want to keep track of in the class. 119 00:05:46,820 --> 00:05:52,340 And so, I've here just included CS50/problem/2020/x. 120 00:05:52,340 --> 00:05:57,000 so all of the 2020 versions of the CS 50 problems I'll be tracking here. 121 00:05:57,000 --> 00:06:00,140 But again, if you had written your own problems that you wanted to track, 122 00:06:00,140 --> 00:06:03,590 you could just add an additional slug that you wanted to keep track of here 123 00:06:03,590 --> 00:06:04,890 as well. 124 00:06:04,890 --> 00:06:06,360 So that's where you can do that. 125 00:06:06,360 --> 00:06:09,410 And then finally, the last setting are other course teachers. 126 00:06:09,410 --> 00:06:11,180 So if you have a TA for the class that you 127 00:06:11,180 --> 00:06:14,580 want to be able to have access to student submissions as well, 128 00:06:14,580 --> 00:06:18,740 you can add invitations to invite other teachers to be 129 00:06:18,740 --> 00:06:20,670 staff members for the course as well. 130 00:06:20,670 --> 00:06:23,270 And so here, in this case, I've added Kareem, for example, 131 00:06:23,270 --> 00:06:26,400 as a course teacher for this class. 132 00:06:26,400 --> 00:06:29,480 So now what I can do is take this invitation link 133 00:06:29,480 --> 00:06:31,370 that I'm supposed to share with students, 134 00:06:31,370 --> 00:06:34,283 and I've sent that invitation link to Jason. 135 00:06:34,283 --> 00:06:36,200 And I guess now I'll turn things over to Jason 136 00:06:36,200 --> 00:06:39,500 to show you what it looks like to accept that and then to submit something. 137 00:06:39,500 --> 00:06:42,083 And then I'll show you what it looks like from the teacher end 138 00:06:42,083 --> 00:06:43,450 to receive that submission. 139 00:06:43,450 --> 00:06:45,650 So, Jason, back to you. 140 00:06:45,650 --> 00:06:50,210 JASON: I'm going to share my screen, so you can see how this works for me. 141 00:06:50,210 --> 00:06:51,410 Here is just a home page. 142 00:06:51,410 --> 00:06:55,790 Brian sent me the link in the chat. 143 00:06:55,790 --> 00:06:56,705 So I'm going to copy. 144 00:06:56,705 --> 00:06:58,700 Let me open that link. 145 00:06:58,700 --> 00:07:02,650 I'm going to copy it into my address bar. 146 00:07:02,650 --> 00:07:04,070 Hit enter. 147 00:07:04,070 --> 00:07:06,050 And this is what it looks like for me. 148 00:07:06,050 --> 00:07:12,380 It says, you are invited to my CS 50 AP course, taught by Brian Yu and Kareem. 149 00:07:12,380 --> 00:07:16,270 This is where I, as a student, can verify 150 00:07:16,270 --> 00:07:20,220 that I'm in the right course from the right teacher. 151 00:07:20,220 --> 00:07:23,270 It says, I understand this will grant the teachers of this course access 152 00:07:23,270 --> 00:07:24,320 to my submissions. 153 00:07:24,320 --> 00:07:26,490 Seems reasonable to me, as a student. 154 00:07:26,490 --> 00:07:28,870 I will check that, and click on Join Course. 155 00:07:28,870 --> 00:07:32,500 156 00:07:32,500 --> 00:07:34,478 I don't have anything submitted yet. 157 00:07:34,478 --> 00:07:36,520 That's because I'm just joining at the beginning. 158 00:07:36,520 --> 00:07:39,070 But maybe now I'll fast forward two months 159 00:07:39,070 --> 00:07:47,080 to working in my IDE on the problem Caesar.C. It's taken me a few weeks. 160 00:07:47,080 --> 00:07:47,770 I've struggled. 161 00:07:47,770 --> 00:07:50,740 I reached out to Brian, he's been a fantastic resource to me. 162 00:07:50,740 --> 00:07:53,800 And now I'm ready to submit. 163 00:07:53,800 --> 00:08:00,190 Here I am in my directory, and I'm going to use the Submit 50 prompt followed 164 00:08:00,190 --> 00:08:02,710 by the specific slug for this. 165 00:08:02,710 --> 00:08:05,837 We had a question earlier about, how do students-- do students have 166 00:08:05,837 --> 00:08:07,420 to memorize these or figure these out? 167 00:08:07,420 --> 00:08:08,800 Usually it works just like this. 168 00:08:08,800 --> 00:08:12,190 A teacher just gives it to the student, and they can copy or paste it. 169 00:08:12,190 --> 00:08:14,620 Or I actually have opened the specification. 170 00:08:14,620 --> 00:08:21,610 And down here at the bottom of the specification, oh, there's--hi, Brian. 171 00:08:21,610 --> 00:08:22,693 There's the slug as well. 172 00:08:22,693 --> 00:08:25,360 So students can copy it directly from the problem specification. 173 00:08:25,360 --> 00:08:26,640 But I got mine from Brian. 174 00:08:26,640 --> 00:08:28,180 I'm going to paste it here. 175 00:08:28,180 --> 00:08:30,130 And now I'm ready to hit Enter. 176 00:08:30,130 --> 00:08:42,429 177 00:08:42,429 --> 00:08:46,977 So a few things you see here I have already logged in, 178 00:08:46,977 --> 00:08:48,560 because I've been submitting problems. 179 00:08:48,560 --> 00:08:52,010 So I didn't actually have to enter my GitHub username and password again. 180 00:08:52,010 --> 00:08:55,100 But if a student hasn't logged in in a while through there IDE, 181 00:08:55,100 --> 00:08:57,590 they'd have to re log in. 182 00:08:57,590 --> 00:09:01,070 Next it's saying, oh, this is the files that we submitted, 183 00:09:01,070 --> 00:09:03,850 and these are all the files that I don't want to submit. 184 00:09:03,850 --> 00:09:06,380 Those are just a personal project I'm working on. 185 00:09:06,380 --> 00:09:09,890 So I'm glad those aren't being submitted to Brian. 186 00:09:09,890 --> 00:09:13,745 I'm going to type y here, that I've kept in mind the course's academic honesty 187 00:09:13,745 --> 00:09:14,245 policy. 188 00:09:14,245 --> 00:09:18,160 189 00:09:18,160 --> 00:09:21,740 And now I have this URL that I can go to see my results. 190 00:09:21,740 --> 00:09:24,895 So I'm going to click on that URL, open it in a new window. 191 00:09:24,895 --> 00:09:28,070 192 00:09:28,070 --> 00:09:30,138 I see my submission with my time stamp. 193 00:09:30,138 --> 00:09:32,180 This is, again, from the student's point of view. 194 00:09:32,180 --> 00:09:34,130 I'm waiting on my results. 195 00:09:34,130 --> 00:09:37,070 I'm a little impatient. 196 00:09:37,070 --> 00:09:44,190 I see that I passed 10 out of 11 Check 50 checks, and my style was 0.98. 197 00:09:44,190 --> 00:09:46,900 So I am not quite happy with this. 198 00:09:46,900 --> 00:09:48,570 I wanted to get a perfect score. 199 00:09:48,570 --> 00:09:51,540 But I'm going to pause my work. 200 00:09:51,540 --> 00:09:53,280 Maybe I'll work on this in a bit. 201 00:09:53,280 --> 00:09:55,207 I'm going to hand it back over to my teacher. 202 00:09:55,207 --> 00:09:56,040 BRIAN YU: All right. 203 00:09:56,040 --> 00:09:57,270 Thanks, Jason. 204 00:09:57,270 --> 00:09:59,880 So now that Jason has now submitted one of the problems, 205 00:09:59,880 --> 00:10:02,818 I'd like to take a look at that submission and see how he did. 206 00:10:02,818 --> 00:10:05,110 So I'll go back, and here's the window I was at before. 207 00:10:05,110 --> 00:10:07,860 And up at the top, there a couple of buttons that I can click. 208 00:10:07,860 --> 00:10:09,780 Right now I'm in Settings mode. 209 00:10:09,780 --> 00:10:11,425 There's also a students' page. 210 00:10:11,425 --> 00:10:13,800 On the students' page, I can see all of the students that 211 00:10:13,800 --> 00:10:15,450 are currently enrolled in the course. 212 00:10:15,450 --> 00:10:17,920 Right now, it seems that I have three students. 213 00:10:17,920 --> 00:10:19,050 And I see Jason down here. 214 00:10:19,050 --> 00:10:20,190 He has one submission. 215 00:10:20,190 --> 00:10:24,140 That was the submission-- the problem that he just submitted, for example. 216 00:10:24,140 --> 00:10:28,230 And if I go over to the submissions tab, here I 217 00:10:28,230 --> 00:10:31,350 see a list of all of the problems that have been submitted 218 00:10:31,350 --> 00:10:33,100 as well as who submitted each one of them. 219 00:10:33,100 --> 00:10:37,560 So for Caesar, for example, I see that one student has submitted that problem. 220 00:10:37,560 --> 00:10:40,330 I see that Jason has submitted the problem. 221 00:10:40,330 --> 00:10:41,670 I can see when he submitted it. 222 00:10:41,670 --> 00:10:44,040 So he submitted it a minute ago. 223 00:10:44,040 --> 00:10:46,590 And then I also get to see the results of the automatic runs 224 00:10:46,590 --> 00:10:48,240 of Check 50 and Style 50. 225 00:10:48,240 --> 00:10:53,070 So here for Check 50, I see that Jason has passed 10 out of the 11 checks. 226 00:10:53,070 --> 00:10:56,910 If I want to see which 10 out of the 11, I can just click on Check 50, 227 00:10:56,910 --> 00:11:00,900 and that's going to open up the check 50 results in another page, where 228 00:11:00,900 --> 00:11:04,350 I can see, all right, these seem to all pass OK. 229 00:11:04,350 --> 00:11:06,000 Let's keep going. 230 00:11:06,000 --> 00:11:08,550 And OK, so the one that he seems to be failing 231 00:11:08,550 --> 00:11:12,840 is handling a non-numeric key where the argument of the program 232 00:11:12,840 --> 00:11:14,160 isn't actually a number. 233 00:11:14,160 --> 00:11:16,285 That's something that we ask students to check for, 234 00:11:16,285 --> 00:11:18,530 but it looks like Jason didn't quite handle that. 235 00:11:18,530 --> 00:11:21,660 But everything else looks really good from a correctness standpoint. 236 00:11:21,660 --> 00:11:23,390 So well done, Jason. 237 00:11:23,390 --> 00:11:25,840 And I can also check his style results as well. 238 00:11:25,840 --> 00:11:26,940 So here it says Style 50. 239 00:11:26,940 --> 00:11:30,780 It looks like 98% of his lines of code were well styled. 240 00:11:30,780 --> 00:11:33,300 But if I click on that Style50 link, I can actually 241 00:11:33,300 --> 00:11:34,710 see what the suggestion was. 242 00:11:34,710 --> 00:11:37,590 And it looks like the suggestion here is that Jason was just 243 00:11:37,590 --> 00:11:43,463 missing some indentation on this line right after the beginning of the code. 244 00:11:43,463 --> 00:11:46,380 And so if I wanted to leave feedback to Jason, for example, comment on 245 00:11:46,380 --> 00:11:50,970 that so he can see it as well, I could click on this Zero Comments link 246 00:11:50,970 --> 00:11:52,970 next to Jason's submission. 247 00:11:52,970 --> 00:11:56,670 And that will take me into a GitHub page, which 248 00:11:56,670 --> 00:11:58,950 will let me comment on Jason's work. 249 00:11:58,950 --> 00:12:02,520 So for example, if I wanted to comment on his indentation, 250 00:12:02,520 --> 00:12:09,220 I could just click next to line 9 right here, and just leave a comment to say, 251 00:12:09,220 --> 00:12:13,180 looks like this line should be indented. 252 00:12:13,180 --> 00:12:16,010 And I can add a comment to that as well. 253 00:12:16,010 --> 00:12:18,670 And so now Jason will be able to see that comment when he goes 254 00:12:18,670 --> 00:12:22,510 to check on his submission as well. 255 00:12:22,510 --> 00:12:25,830 So that then is Check 50 results, Style 50 results, 256 00:12:25,830 --> 00:12:27,670 and leaving comments on student submissions. 257 00:12:27,670 --> 00:12:31,110 And if I wanted to, I could also download his submission as a zip file, 258 00:12:31,110 --> 00:12:33,060 if I wanted to download it to my computer 259 00:12:33,060 --> 00:12:36,540 in order to run his code on my own computer, for example. 260 00:12:36,540 --> 00:12:38,030 That's an option available to me. 261 00:12:38,030 --> 00:12:41,510 Though most of the time, when I'm using just CS 50s problems, 262 00:12:41,510 --> 00:12:42,510 I don't have to do that. 263 00:12:42,510 --> 00:12:45,240 Because Check 50 has already run the program for me 264 00:12:45,240 --> 00:12:49,920 and told me what the results of all those correctness checks are. 265 00:12:49,920 --> 00:12:52,800 And then if I want to, if I want to download all this information, 266 00:12:52,800 --> 00:12:57,210 you can export all of this data from any particular problem, 267 00:12:57,210 --> 00:12:59,910 as just a CSV spreadsheet that you could open up 268 00:12:59,910 --> 00:13:02,010 in a spreadsheet program like Apple numbers 269 00:13:02,010 --> 00:13:04,255 or Excel or Google sheets, if you wanted to, 270 00:13:04,255 --> 00:13:07,650 to be able to see all of the students who have submitted the problem, 271 00:13:07,650 --> 00:13:11,220 when they've submitted that problem, and how many correctness checks they got, 272 00:13:11,220 --> 00:13:13,140 and what their style score was. 273 00:13:13,140 --> 00:13:15,660 So all that information is there too. 274 00:13:15,660 --> 00:13:20,070 There are also these toggles here, for latest and all. 275 00:13:20,070 --> 00:13:22,140 By default, when you're in Submit.CS50.IO, 276 00:13:22,140 --> 00:13:25,530 you're only going to see the latest submission from any student. 277 00:13:25,530 --> 00:13:27,480 So if they submitted three or four times, 278 00:13:27,480 --> 00:13:29,850 you'll only see the most recent one. 279 00:13:29,850 --> 00:13:33,060 But if I switch from latest to all, for example, 280 00:13:33,060 --> 00:13:37,180 then if a student has submitted multiple times-- so this student, for example, 281 00:13:37,180 --> 00:13:39,537 has submitted a couple of times for the same problem, 282 00:13:39,537 --> 00:13:42,370 then I'm going to see all of those submissions in this view as well. 283 00:13:42,370 --> 00:13:45,360 So if you wanted to go back and look at a student's progression 284 00:13:45,360 --> 00:13:48,300 over time, as they might have submitted a problem a couple of times, 285 00:13:48,300 --> 00:13:51,050 you can use that view to be able to see all of the times 286 00:13:51,050 --> 00:13:54,230 that that student has submitted to. 287 00:13:54,230 --> 00:13:57,980 So that, then, is a brief look at what that workflow looks 288 00:13:57,980 --> 00:14:02,630 like for using Submit 50 and Submit.CS50.IO, 289 00:14:02,630 --> 00:14:05,570 where you send the invitation link to your students, 290 00:14:05,570 --> 00:14:07,460 then students can submit their work. 291 00:14:07,460 --> 00:14:11,810 And then you can then go in and view the results of the automated checks that 292 00:14:11,810 --> 00:14:14,520 are run on that code. 293 00:14:14,520 --> 00:14:17,280 We'll pause now to see if there are any questions, 294 00:14:17,280 --> 00:14:19,110 or if Jason has anything else to add. 295 00:14:19,110 --> 00:14:22,170 JASON: I'll just share once more my screen, 296 00:14:22,170 --> 00:14:25,800 so I can show the student's view before we take any questions. 297 00:14:25,800 --> 00:14:29,460 it's worth noting too that I got an email when Brian left the comment. 298 00:14:29,460 --> 00:14:33,490 So there's, on my phone, it might be a little hard to see, 299 00:14:33,490 --> 00:14:38,010 but I got an email saying looks like this line should be indented. 300 00:14:38,010 --> 00:14:43,200 Also when I refresh my student view, I can see the one comment note there. 301 00:14:43,200 --> 00:14:48,760 I can click on the one comment here to see 302 00:14:48,760 --> 00:14:52,180 all of the information my teachers left for me. 303 00:14:52,180 --> 00:14:57,540 And just as Brian clicked on Check 50 and Style 50, so too 304 00:14:57,540 --> 00:14:59,630 can I, as a student. 305 00:14:59,630 --> 00:15:02,100 I'm going to go here and click on my submissions tab. 306 00:15:02,100 --> 00:15:08,410 Actually, while Brian was sharing information with you all, I was like-- 307 00:15:08,410 --> 00:15:09,900 I was bummed by that indentation. 308 00:15:09,900 --> 00:15:12,360 I really-- I always make that mistake, and I always forget to do it. 309 00:15:12,360 --> 00:15:13,740 So I went back, and I fixed it. 310 00:15:13,740 --> 00:15:18,090 And I submitted again, so you can see this time, my style is a perfect 1.00, 311 00:15:18,090 --> 00:15:21,810 because I did take my teacher's advice and fix that particular line, 312 00:15:21,810 --> 00:15:22,620 and resubmit. 313 00:15:22,620 --> 00:15:25,437 So I will stop sharing now, and then open it up 314 00:15:25,437 --> 00:15:27,270 to anyone who would like to raise their hand 315 00:15:27,270 --> 00:15:30,180 and ask any questions about this workflow. 316 00:15:30,180 --> 00:15:33,277 We see-- [? Hani, ?] you have a question? 317 00:15:33,277 --> 00:15:33,860 STUDENT: Yeah. 318 00:15:33,860 --> 00:15:37,730 So I'm actually taking the course online right now. 319 00:15:37,730 --> 00:15:40,910 And to be honest, I'm not focused on style at all. 320 00:15:40,910 --> 00:15:45,070 I'm just focused on getting a program that works, you know. 321 00:15:45,070 --> 00:15:51,680 So like my score is typically, like, 11 out of 11 for the correctness, and 0.4, 322 00:15:51,680 --> 00:15:53,150 you know, for style. 323 00:15:53,150 --> 00:16:00,825 Now can I go back later and try to fix the style and improve my style score? 324 00:16:00,825 --> 00:16:01,700 BRIAN YU: Absolutely. 325 00:16:01,700 --> 00:16:05,840 With the online version of the Class CS50X, you can submit as many times 326 00:16:05,840 --> 00:16:09,530 as you want, and we'll only take a look at your most recent score 327 00:16:09,530 --> 00:16:11,400 in determining your score for the problem. 328 00:16:11,400 --> 00:16:13,400 And if you run Style 50 on the program, it 329 00:16:13,400 --> 00:16:18,110 will also tell you where the improvements to the style can be made. 330 00:16:18,110 --> 00:16:20,960 And that will hopefully be a useful guide there too. 331 00:16:20,960 --> 00:16:23,530 JASON: Next we have a question from Matthew. 332 00:16:23,530 --> 00:16:27,400 STUDENT: Yes, I'm curious about a window of what other folks do, 333 00:16:27,400 --> 00:16:32,320 or what you all do, in terms of when you would shut off 334 00:16:32,320 --> 00:16:38,180 the ability to continue to submit, either in correctness or in style. 335 00:16:38,180 --> 00:16:41,380 I just imagine, like, heading towards the end of the term, 336 00:16:41,380 --> 00:16:44,600 and students going back and trying to do this, this, this, 337 00:16:44,600 --> 00:16:48,080 this on all these different assignments, and having to change a grade-- 338 00:16:48,080 --> 00:16:51,448 Like, do you all set up a time limit or a time frame? 339 00:16:51,448 --> 00:16:53,740 BRIAN YU: From the teacher's perspective with the tool, 340 00:16:53,740 --> 00:16:55,975 you can't actually stop submissions. 341 00:16:55,975 --> 00:16:56,600 STUDENT: Right. 342 00:16:56,600 --> 00:17:00,040 BRIAN YU: There's no way for me to cut off submissions. 343 00:17:00,040 --> 00:17:04,246 So if students want to submit, they can continue submitting. 344 00:17:04,246 --> 00:17:07,329 So ultimately, then-- I know this isn't quite the answer to your question. 345 00:17:07,329 --> 00:17:09,954 It comes down to you as a teacher, looking at those time stamps 346 00:17:09,954 --> 00:17:13,450 and saying, I told them they had one week from the deadline, 347 00:17:13,450 --> 00:17:15,500 and after that I wouldn't submit anything else. 348 00:17:15,500 --> 00:17:17,740 And I see these time stamps are after that week, 349 00:17:17,740 --> 00:17:20,290 so I'm not even going to look at them. 350 00:17:20,290 --> 00:17:23,440 But from the tool's perspective, you can't actually shut down 351 00:17:23,440 --> 00:17:25,089 students' abilities to submit. 352 00:17:25,089 --> 00:17:26,589 STUDENT: Oh, and I wouldn't want to. 353 00:17:26,589 --> 00:17:26,880 BRIAN YU: Yeah. 354 00:17:26,880 --> 00:17:27,505 STUDENT: Right? 355 00:17:27,505 --> 00:17:29,170 I mean, we'd want them to-- right? 356 00:17:29,170 --> 00:17:30,580 We always want to get better. 357 00:17:30,580 --> 00:17:32,470 I'm just, I just think about-- 358 00:17:32,470 --> 00:17:33,550 anyway. 359 00:17:33,550 --> 00:17:36,317 STUDENT: But if you're, Matthew, recording the scores out of band, 360 00:17:36,317 --> 00:17:37,900 and you're on grade book of some sort. 361 00:17:37,900 --> 00:17:41,108 Then it really doesn't matter if they continue submitting, because you're not 362 00:17:41,108 --> 00:17:43,610 going to make any actionable change. 363 00:17:43,610 --> 00:17:46,550 BRIAN YU: Another feature that I can mention about the tool 364 00:17:46,550 --> 00:17:50,430 is that you can specify a window of times that you want to look at. 365 00:17:50,430 --> 00:17:52,670 So, for example-- 366 00:17:52,670 --> 00:17:54,740 I'll show you what I'm looking at here. 367 00:17:54,740 --> 00:17:58,660 If I go to all submissions, I can see, Jason right now has submitted twice. 368 00:17:58,660 --> 00:18:04,040 He's submitted once at 12:23 p.m., where he got a style score of 98%. 369 00:18:04,040 --> 00:18:06,920 And he submitted a little later at 12:28 p.m. 370 00:18:06,920 --> 00:18:08,390 and got a style score of 1 p.m. 371 00:18:08,390 --> 00:18:10,070 So, of course, these were only five minutes apart. 372 00:18:10,070 --> 00:18:12,903 But you could imagine maybe they were five days apart, for instance, 373 00:18:12,903 --> 00:18:15,480 where the deadline might have been in between. 374 00:18:15,480 --> 00:18:18,500 And so, by default, if I go to the latest submission, 375 00:18:18,500 --> 00:18:20,300 I'm looking at his most recent submission 376 00:18:20,300 --> 00:18:23,180 that has a perfect score on style. 377 00:18:23,180 --> 00:18:25,670 But I also have this option, where I can filter 378 00:18:25,670 --> 00:18:28,280 based on a particular range of dates and times. 379 00:18:28,280 --> 00:18:34,730 So if I set the range of dates and times to be only until, let's say 12:25 p.m., 380 00:18:34,730 --> 00:18:35,610 for example. 381 00:18:35,610 --> 00:18:38,900 So before Jason submitted for the second time. 382 00:18:38,900 --> 00:18:43,310 And I apply that time filter, then the submission that I see for Jason 383 00:18:43,310 --> 00:18:47,138 is his first submission, the one that got the 98%, because I'm only 384 00:18:47,138 --> 00:18:50,430 restricting the submissions I'm looking at to a particular date and time range. 385 00:18:50,430 --> 00:18:53,347 So if you do have a deadline, and you don't want to consider deadlines 386 00:18:53,347 --> 00:18:55,520 after a particular time, you can use that feature 387 00:18:55,520 --> 00:18:58,460 to be able to limit the view of what you're seeing. 388 00:18:58,460 --> 00:19:02,000 STUDENT: And Brian, is it fair to assume that when you put on that filter, 389 00:19:02,000 --> 00:19:06,020 that applies to all the students for that particular problem set? 390 00:19:06,020 --> 00:19:07,900 BRIAN YU: For that particular problem, yeah. 391 00:19:07,900 --> 00:19:10,190 So if there are multiple problems, then-- 392 00:19:10,190 --> 00:19:12,770 multiple problems, you'll have to specify a separate date 393 00:19:12,770 --> 00:19:14,700 range for each of the problems. 394 00:19:14,700 --> 00:19:16,580 But for any one problem, you can specify what 395 00:19:16,580 --> 00:19:18,170 range you want to look at for that. 396 00:19:18,170 --> 00:19:20,510 STUDENT: You all are so unbelievably detail oriented. 397 00:19:20,510 --> 00:19:21,650 I love it. 398 00:19:21,650 --> 00:19:26,210 JASON: And, Matthew, too, so this afternoon from the three to five panel, 399 00:19:26,210 --> 00:19:29,140 we'll have a-- you'll have three teachers who are teaching CS 50. 400 00:19:29,140 --> 00:19:31,820 So I think a good question to ask them would be, sort of, how 401 00:19:31,820 --> 00:19:33,355 do they handle late submissions? 402 00:19:33,355 --> 00:19:36,230 Douglas mentioned, talked a bit about that yesterday with his tardies 403 00:19:36,230 --> 00:19:36,730 and lates. 404 00:19:36,730 --> 00:19:39,230 But they'll be able to give you three answers on how 405 00:19:39,230 --> 00:19:41,312 they handle student submission. 406 00:19:41,312 --> 00:19:43,520 So if you want to hold on to that question, ask them. 407 00:19:43,520 --> 00:19:46,790 I think they'll be able to give you some good answers. 408 00:19:46,790 --> 00:19:50,455 And then, oh, Ben. 409 00:19:50,455 --> 00:19:52,580 I'll just quickly answer yours, Ben, for what I do. 410 00:19:52,580 --> 00:19:54,872 But I think that would be another good question to ask. 411 00:19:54,872 --> 00:19:57,470 Ben, in the chat, mentioned if I'd be willing to share 412 00:19:57,470 --> 00:20:02,270 how I assigned this in my Schoology platform, which is what the school uses 413 00:20:02,270 --> 00:20:03,590 to keep track of assignments. 414 00:20:03,590 --> 00:20:05,810 Again, I think that would be a good question to ask the panel of teachers 415 00:20:05,810 --> 00:20:06,310 later today. 416 00:20:06,310 --> 00:20:10,160 But I simply share a link to the problem specification, 417 00:20:10,160 --> 00:20:15,890 which details all the background, has links to videos, has example cases, 418 00:20:15,890 --> 00:20:18,650 and has the Submit slug at the very end. 419 00:20:18,650 --> 00:20:22,775 So I share a link to that online document. 420 00:20:22,775 --> 00:20:24,400 So I don't need to reinvent everything. 421 00:20:24,400 --> 00:20:28,060 It's already written there and organized really nicely. 422 00:20:28,060 --> 00:20:29,927 Does that help, Ben, briefly? 423 00:20:29,927 --> 00:20:31,260 I don't know where you are, Ben. 424 00:20:31,260 --> 00:20:33,420 Oh, thanks, Ben. 425 00:20:33,420 --> 00:20:38,960 OK, I'm going to resume sharing my screen. 426 00:20:38,960 --> 00:20:42,010 I found this funny comic. 427 00:20:42,010 --> 00:20:45,230 I think sometimes when we're in our-- 428 00:20:45,230 --> 00:20:49,300 when we're teaching our course, we-- 429 00:20:49,300 --> 00:20:51,550 and we don't know what's available out there, 430 00:20:51,550 --> 00:20:56,140 we might feel the pressure to have to create things from scratch. 431 00:20:56,140 --> 00:20:59,470 And those things we create from scratch might have already been created or done 432 00:20:59,470 --> 00:21:02,380 by other people in the world. 433 00:21:02,380 --> 00:21:04,420 But also other teachers teaching CS 50 AP. 434 00:21:04,420 --> 00:21:10,780 So today we're launching this brand-new tool and brand-new email address, 435 00:21:10,780 --> 00:21:14,380 share@CS50.Harvard.edu. 436 00:21:14,380 --> 00:21:19,090 And our plan is, that you can email us the problems 437 00:21:19,090 --> 00:21:21,850 you write, the labs you build, the activities you 438 00:21:21,850 --> 00:21:25,240 create, the assessments you design, and whatever else you've got. 439 00:21:25,240 --> 00:21:29,590 And then we'll curate it, add it to what CS50 has already built over the years, 440 00:21:29,590 --> 00:21:32,990 and post it online, so that you don't have to reinvent the wheel. 441 00:21:32,990 --> 00:21:35,170 So you'll hear a bit later today, Margaret's created 442 00:21:35,170 --> 00:21:36,498 all these fantastic labs. 443 00:21:36,498 --> 00:21:38,290 You'll see in a bit some of the assessments 444 00:21:38,290 --> 00:21:40,160 I wrote for my own students. 445 00:21:40,160 --> 00:21:43,440 Rather than only being able to give them to our own students, 446 00:21:43,440 --> 00:21:45,190 or share them with our own students, we're 447 00:21:45,190 --> 00:21:47,140 going to create an online portal where you 448 00:21:47,140 --> 00:21:52,310 can see all of these wonderful resources that your colleagues have created. 449 00:21:52,310 --> 00:21:55,160 And this is the email address through which you can 450 00:21:55,160 --> 00:21:57,500 share the things you've built with us. 451 00:21:57,500 --> 00:22:02,060 Again we'll curate it, get it organized, ensure it's searchable, 452 00:22:02,060 --> 00:22:04,940 and then it will be posted on the AP website, 453 00:22:04,940 --> 00:22:07,790 so that you can look through labs that have been built, 454 00:22:07,790 --> 00:22:11,750 sample problems people have wrote to facilitate a particular skill 455 00:22:11,750 --> 00:22:15,410 during an in-class exercise, or assessment questions teachers 456 00:22:15,410 --> 00:22:17,735 have written in addition to the ones David and his team 457 00:22:17,735 --> 00:22:18,610 have already written. 458 00:22:18,610 --> 00:22:24,690 459 00:22:24,690 --> 00:22:30,360 And as I saw in the chat, John mentioned that somebody mentioned about somebody 460 00:22:30,360 --> 00:22:33,570 who had created a Google form to ask questions about videos as people 461 00:22:33,570 --> 00:22:34,297 watched it. 462 00:22:34,297 --> 00:22:37,380 Those are the types of things we would love to collect and curate and then 463 00:22:37,380 --> 00:22:39,360 share with you all. 464 00:22:39,360 --> 00:22:42,660 So again, the email address is live. 465 00:22:42,660 --> 00:22:44,560 You don't need to start e-mailing right away. 466 00:22:44,560 --> 00:22:46,680 But if you have things from the past to share, 467 00:22:46,680 --> 00:22:49,440 we would love to have them collected here. 468 00:22:49,440 --> 00:22:53,550 And then we will start putting them all together to have you all-- 469 00:22:53,550 --> 00:22:56,092 have for you all before the start of the term. 470 00:22:56,092 --> 00:22:58,050 Or hopefully well before the start of the term, 471 00:22:58,050 --> 00:23:02,811 because I know I start planning at least in early August, if not late July. 472 00:23:02,811 --> 00:23:05,020 [LAUGHTER] 473 00:23:05,020 --> 00:23:10,540 OK, so finally we'll move into talking about assessments. 474 00:23:10,540 --> 00:23:15,370 And I think, before you jump to click on these two links-- 475 00:23:15,370 --> 00:23:17,050 I'll share these links in the chat-- 476 00:23:17,050 --> 00:23:21,310 I just wanted to talk a bit about the philosophy behind assessments. 477 00:23:21,310 --> 00:23:24,910 And I have not heard this from David Malan directly, 478 00:23:24,910 --> 00:23:28,210 but when I took CS50 as a student, I sort of experienced 479 00:23:28,210 --> 00:23:32,230 assessments that were unlike anything else I'd experienced in other classes. 480 00:23:32,230 --> 00:23:35,380 They weren't expecting rote memorization. 481 00:23:35,380 --> 00:23:38,680 They weren't asking me to regurgitate what I'd been taught in class. 482 00:23:38,680 --> 00:23:42,280 But they were asking me, instead, to apply 483 00:23:42,280 --> 00:23:47,230 different things I had learned, both in lectures and in sections, and on my own 484 00:23:47,230 --> 00:23:48,970 as I was working through problems. 485 00:23:48,970 --> 00:23:52,120 Apply all of that in some new context. 486 00:23:52,120 --> 00:23:55,780 So I had all the building blocks I needed to be able to be successful. 487 00:23:55,780 --> 00:23:59,500 And on top of that, I needed to think creatively. 488 00:23:59,500 --> 00:24:01,630 I needed to problem solve. 489 00:24:01,630 --> 00:24:05,020 And ultimately, the assessments themselves that I took in CS50 490 00:24:05,020 --> 00:24:07,240 were learning opportunities. 491 00:24:07,240 --> 00:24:12,055 And it was this really magical experience for me as a student. 492 00:24:12,055 --> 00:24:16,510 And when I was a teacher, this year, teaching CS50 AP to my students, 493 00:24:16,510 --> 00:24:20,380 I tried to design assessments and questions that 494 00:24:20,380 --> 00:24:24,610 would similarly allow them to use what they've learned 495 00:24:24,610 --> 00:24:26,170 and apply it in new ways. 496 00:24:26,170 --> 00:24:28,748 And actually, I dare say, enjoy the assessments. 497 00:24:28,748 --> 00:24:30,790 And that's what I heard from some of the students 498 00:24:30,790 --> 00:24:33,520 this year, that they looked forward to taking assessments. 499 00:24:33,520 --> 00:24:36,190 Because it was asking them to think about new things, 500 00:24:36,190 --> 00:24:40,990 and think in slightly different ways, and solve new problems using 501 00:24:40,990 --> 00:24:42,910 the information they have learned. 502 00:24:42,910 --> 00:24:49,120 So that, I think, is a bit of the philosophy behind assessments in CS50. 503 00:24:49,120 --> 00:24:51,730 Before I talk about these two links in particular, David, 504 00:24:51,730 --> 00:24:54,730 did you want to add anything else about, sort of your philosophy 505 00:24:54,730 --> 00:24:57,240 behind assessment questions? 506 00:24:57,240 --> 00:24:57,990 DAVID MALAN: Sure. 507 00:24:57,990 --> 00:24:59,865 I should disclaim that you probably described 508 00:24:59,865 --> 00:25:02,670 it more magically than all students might describe it. 509 00:25:02,670 --> 00:25:06,630 I dare say that the flip side of that is that there's often frustration. 510 00:25:06,630 --> 00:25:10,230 Students who feel that, hey, we didn't talk about this in class. 511 00:25:10,230 --> 00:25:12,660 I have no idea how to solve this. 512 00:25:12,660 --> 00:25:16,200 But the course really is intended to be a course that gives you 513 00:25:16,200 --> 00:25:19,710 enough of what you need to know to then go and teach yourself the other things. 514 00:25:19,710 --> 00:25:21,630 So that at the end of the semester, we really 515 00:25:21,630 --> 00:25:24,015 have taken off all of those training wheels. 516 00:25:24,015 --> 00:25:26,640 And we have-- and this has been a recent thing in recent years, 517 00:25:26,640 --> 00:25:29,955 the quizzes that Jason has linked here under CS50's are 518 00:25:29,955 --> 00:25:32,080 very different from what they were a few years ago. 519 00:25:32,080 --> 00:25:34,580 Long story short, I used to take a more traditional approach 520 00:25:34,580 --> 00:25:36,543 to asking some multiple choice, true/false, 521 00:25:36,543 --> 00:25:37,710 short answers, and the like. 522 00:25:37,710 --> 00:25:40,590 But they really were regurgitive of the kind of content 523 00:25:40,590 --> 00:25:42,150 we'd discussed in the class. 524 00:25:42,150 --> 00:25:45,060 And now, Brian and I and the team really view 525 00:25:45,060 --> 00:25:48,180 the quizzes in the class, or the test, as an opportunity 526 00:25:48,180 --> 00:25:49,650 to teach something further. 527 00:25:49,650 --> 00:25:53,580 I mentioned the other day the Hawaii incident, where the missile 528 00:25:53,580 --> 00:25:55,877 alert was triggered some time ago. 529 00:25:55,877 --> 00:25:57,960 And we used that as an opportunity to get students 530 00:25:57,960 --> 00:25:59,770 thinking about real-world issues. 531 00:25:59,770 --> 00:26:04,050 And so I often use our assessments now as a vehicle for introducing 532 00:26:04,050 --> 00:26:07,860 and, in turn, assessing material that we didn't necessarily have time 533 00:26:07,860 --> 00:26:10,320 for or want to spend classroom time on. 534 00:26:10,320 --> 00:26:14,640 So as such, they've become an extension of teaching more than they are just 535 00:26:14,640 --> 00:26:16,260 quantitative measures. 536 00:26:16,260 --> 00:26:20,907 JASON: So these two links are not clickable, obviously, 537 00:26:20,907 --> 00:26:21,990 through sharing my screen. 538 00:26:21,990 --> 00:26:25,560 But this PowerPoint-- or these Google slides 539 00:26:25,560 --> 00:26:27,190 are available through the notion link. 540 00:26:27,190 --> 00:26:29,273 We will copy these links, and put them in the chat 541 00:26:29,273 --> 00:26:31,560 as well, if that hasn't been done already. 542 00:26:31,560 --> 00:26:33,780 I don't have the chat open right now. 543 00:26:33,780 --> 00:26:38,918 But the rest of this session is for you to open these, to look through these, 544 00:26:38,918 --> 00:26:41,460 to kind of get a sense of some of these assessment questions. 545 00:26:41,460 --> 00:26:44,250 And then, if you're, like, thinking to yourself, 546 00:26:44,250 --> 00:26:46,740 I would love to use some of these, you are in luck. 547 00:26:46,740 --> 00:26:48,690 Because we're going to take-- these are part 548 00:26:48,690 --> 00:26:51,960 of what we're putting into that new tool to share with you all. 549 00:26:51,960 --> 00:26:54,840 So we will not just have these PDFs in, but we'll 550 00:26:54,840 --> 00:26:58,840 have them in as separate questions that you can search through and edit. 551 00:26:58,840 --> 00:27:01,890 And so don't feel like you need to download these, save these, copy 552 00:27:01,890 --> 00:27:02,970 these right now. 553 00:27:02,970 --> 00:27:04,440 This is just to get a flavor of-- 554 00:27:04,440 --> 00:27:06,210 sense of the flavor of what's available. 555 00:27:06,210 --> 00:27:12,240 And then, oop-- we will ultimately put these together 556 00:27:12,240 --> 00:27:14,880 with everything you've sent us, so you have access 557 00:27:14,880 --> 00:27:17,670 to that whole wealth of resources. 558 00:27:17,670 --> 00:27:20,040 [? Ahma ?] asked a good question in the chat. 559 00:27:20,040 --> 00:27:22,953 These only have the questions. 560 00:27:22,953 --> 00:27:24,120 They don't have the answers. 561 00:27:24,120 --> 00:27:27,507 But we are going to make available questions and answers. 562 00:27:27,507 --> 00:27:30,090 All of the assessments I made have the answer and answer keys. 563 00:27:30,090 --> 00:27:32,580 So all of that will be available to teachers. 564 00:27:32,580 --> 00:27:35,010 That's just not what is being shared with you right now. 565 00:27:35,010 --> 00:27:37,540 Feel free to go look through these. 566 00:27:37,540 --> 00:27:40,680 The next 20 minutes we'll be on here for you to ask questions as you're 567 00:27:40,680 --> 00:27:41,520 diving in. 568 00:27:41,520 --> 00:27:45,210 But that's the end of the content we have for you before the next session 569 00:27:45,210 --> 00:27:48,590 starts in about 20 minutes.