1 00:00:00,000 --> 00:00:01,380 2 00:00:01,380 --> 00:00:03,550 SHAANAN COHNEY: OK, hello, everyone. 3 00:00:03,550 --> 00:00:05,280 So because I've always wanted to do this, 4 00:00:05,280 --> 00:00:12,151 and I do it in my own classroom, lights, camera, and action. 5 00:00:12,151 --> 00:00:14,139 [MUSIC PLAYING] 6 00:00:14,139 --> 00:00:24,079 7 00:00:24,079 --> 00:00:25,570 [BEGIN VIDEO PLAYBACK] 8 00:00:25,570 --> 00:00:27,558 9 00:00:27,558 --> 00:00:29,049 [PHONE RINGING] 10 00:00:29,049 --> 00:00:31,550 11 00:00:31,550 --> 00:00:32,980 - Hello? 12 00:00:32,980 --> 00:00:34,360 Oh my god. 13 00:00:34,360 --> 00:00:36,020 FOA is starting? 14 00:00:36,020 --> 00:00:37,470 OK, I need my puffer. 15 00:00:37,470 --> 00:00:39,235 Where's my puffer? 16 00:00:39,235 --> 00:00:40,985 - I think ChatGPT can be used effectively. 17 00:00:40,985 --> 00:00:41,420 [PHONE RINGING] 18 00:00:41,420 --> 00:00:42,140 - Oh my gosh. 19 00:00:42,140 --> 00:00:42,920 Sorry. 20 00:00:42,920 --> 00:00:44,700 FOA is starting now? 21 00:00:44,700 --> 00:00:45,210 Bye bye. 22 00:00:45,210 --> 00:00:46,146 Bye bye. 23 00:00:46,146 --> 00:00:47,604 [PHONE RINGING] 24 00:00:47,604 --> 00:00:50,040 - It's time for Foundations of Algorithms? 25 00:00:50,040 --> 00:00:52,280 I need to get my sonic screwdriver! 26 00:00:52,280 --> 00:00:56,250 - I'm really looking forward to my military coup of the university. 27 00:00:56,250 --> 00:00:56,765 - Me too. 28 00:00:56,765 --> 00:00:59,065 I can't wait to get rid of these hot desks. 29 00:00:59,065 --> 00:01:00,220 [PHONE RINGING] 30 00:01:00,220 --> 00:01:01,040 31 00:01:01,040 --> 00:01:02,060 - Oh my god! 32 00:01:02,060 --> 00:01:03,115 FOA is starting! 33 00:01:03,115 --> 00:01:03,615 - What? 34 00:01:03,615 --> 00:01:05,140 FOA is starting? 35 00:01:05,140 --> 00:01:06,520 I should go get my parachute! 36 00:01:06,520 --> 00:01:08,716 - I've got to set my cryptowatch! 37 00:01:08,716 --> 00:01:09,490 [PHONE RINGING] 38 00:01:09,490 --> 00:01:11,360 - Wait, it's time FOA already? 39 00:01:11,360 --> 00:01:13,796 Let me get my cat. 40 00:01:13,796 --> 00:01:15,242 [PHONE RINGING] 41 00:01:15,242 --> 00:01:16,760 - FOA is starting? 42 00:01:16,760 --> 00:01:20,111 I've got to go meet the teaching fellows. 43 00:01:20,111 --> 00:01:21,860 [END PLAYBACK] 44 00:01:21,860 --> 00:01:24,950 SHAANAN COHNEY: So welcome, all of you, to today's class, 45 00:01:24,950 --> 00:01:28,970 which is Foundations of Algorithms with me, Dr. Shaanan Cohney, here 46 00:01:28,970 --> 00:01:30,360 at the University of Melbourne. 47 00:01:30,360 --> 00:01:33,080 So for the next few minutes, all of you are now my students. 48 00:01:33,080 --> 00:01:38,120 And if I look out over there into the classroom, I can see about 450 of you 49 00:01:38,120 --> 00:01:39,090 this semester. 50 00:01:39,090 --> 00:01:43,220 And were we in my other semester, there would be somewhere around 1,000 of you 51 00:01:43,220 --> 00:01:45,580 out there in the auditorium. 52 00:01:45,580 --> 00:01:48,500 All of you come from a variety of degree programs. 53 00:01:48,500 --> 00:01:52,030 In my university, in Australia, there are six degree programs 54 00:01:52,030 --> 00:01:53,920 that we're anticipating that all of you come 55 00:01:53,920 --> 00:01:59,110 from, from places as wide as music, commerce, and the business school 56 00:01:59,110 --> 00:02:03,250 to bio sciences and, of course, the faculty of science that many of you 57 00:02:03,250 --> 00:02:05,330 are coming to us from. 58 00:02:05,330 --> 00:02:09,050 We have 10 teaching fellows, also known as teaching assistants, 59 00:02:09,050 --> 00:02:13,400 or TAs in other countries, who are going to be helping out with our course. 60 00:02:13,400 --> 00:02:17,210 And then four first-year fellows, that I'll talk a little bit more about 61 00:02:17,210 --> 00:02:19,730 later, who help answer some of the questions 62 00:02:19,730 --> 00:02:23,330 that you'll give us throughout the semester. 63 00:02:23,330 --> 00:02:28,040 We have one production associate who is one of our very own students 64 00:02:28,040 --> 00:02:30,020 and is going to be helping out with the camera. 65 00:02:30,020 --> 00:02:31,493 Hello, Hrithik over there. 66 00:02:31,493 --> 00:02:33,410 I mean, he's back in Australia, so he probably 67 00:02:33,410 --> 00:02:37,610 can't see me all the way from Boston, but I'll give him a wave anyway. 68 00:02:37,610 --> 00:02:40,790 And then, of course, we have our two professors 69 00:02:40,790 --> 00:02:44,160 who are going to be delivering at least the lecture components of the class. 70 00:02:44,160 --> 00:02:52,640 There is Dr. Jianzhong Qi on my left at least, and of course, myself. 71 00:02:52,640 --> 00:02:58,310 So today, I'm going to tell you about the 80% outcome with 5% resources 72 00:02:58,310 --> 00:03:02,750 method, or what I call the 80 in 5 that I, Shaanan Cohney, 73 00:03:02,750 --> 00:03:06,410 have applied to my class, Foundations of Algorithms, 74 00:03:06,410 --> 00:03:09,410 that is quite similar to Harvard CS50. 75 00:03:09,410 --> 00:03:13,400 And how I tried to reverse engineer some of the components of CS50 76 00:03:13,400 --> 00:03:17,300 and adapt them to my own university context. 77 00:03:17,300 --> 00:03:19,890 So a tiny bit about me. 78 00:03:19,890 --> 00:03:24,080 Now, we're back in the Boston, Cambridge, Harvard 79 00:03:24,080 --> 00:03:28,790 room, where I can now speak to you as stepping outside of our classroom 80 00:03:28,790 --> 00:03:30,810 and back into the educators' workshop. 81 00:03:30,810 --> 00:03:34,490 So I am responsible for the first-year computer science curriculum 82 00:03:34,490 --> 00:03:36,000 at the University of Melbourne. 83 00:03:36,000 --> 00:03:42,800 We have about 1,000 to 1,500 students who take the introductory course with us 84 00:03:42,800 --> 00:03:43,710 every year. 85 00:03:43,710 --> 00:03:49,670 And we have about 5,000 overall students in a university with about 40,000 86 00:03:49,670 --> 00:03:50,730 students total. 87 00:03:50,730 --> 00:03:54,290 So a very significant fraction of the overall student body 88 00:03:54,290 --> 00:04:00,080 takes our class, just like the students who take Harvard CS50 in the college 89 00:04:00,080 --> 00:04:01,350 here. 90 00:04:01,350 --> 00:04:04,830 So the main theme of today's talk is going 91 00:04:04,830 --> 00:04:08,850 to be thinking about where the biggest room for improvement is in teaching. 92 00:04:08,850 --> 00:04:11,100 And while I've spent a lot of time reflecting on this, 93 00:04:11,100 --> 00:04:13,590 I want you as well to keep in mind today, 94 00:04:13,590 --> 00:04:17,790 where do you think you could make a small improvement that 95 00:04:17,790 --> 00:04:21,760 would be a large change in terms of the actual outcome? 96 00:04:21,760 --> 00:04:24,900 So is there a 5% change that you could make that would 97 00:04:24,900 --> 00:04:27,340 result in an 80% different outcome? 98 00:04:27,340 --> 00:04:28,990 And that's really today's talk. 99 00:04:28,990 --> 00:04:33,090 So as much as we're going to draw very heavily from CS50's curriculum 100 00:04:33,090 --> 00:04:35,790 structure, pedagogy, and design, at the end 101 00:04:35,790 --> 00:04:37,920 of the day, whatever context you're teaching in 102 00:04:37,920 --> 00:04:39,670 is going to be a little bit different. 103 00:04:39,670 --> 00:04:43,020 And so you're going to have to identify your own set of changes 104 00:04:43,020 --> 00:04:43,880 that you can make. 105 00:04:43,880 --> 00:04:45,630 Some of them will be a little bit smaller. 106 00:04:45,630 --> 00:04:47,200 Some of them will be a bit larger. 107 00:04:47,200 --> 00:04:49,020 But whatever your resources are, you've got 108 00:04:49,020 --> 00:04:53,820 to figure out the right things to do that will have the largest changes. 109 00:04:53,820 --> 00:04:58,710 So we already know a little bit about CS50 from all the discussion we've had 110 00:04:58,710 --> 00:04:59,560 so far. 111 00:04:59,560 --> 00:05:05,030 But what I teach is really a bit more like CS50 plus 1/2. 112 00:05:05,030 --> 00:05:07,340 So it's not the very, very intro class. 113 00:05:07,340 --> 00:05:10,090 They've taken a little bit of programming before in Python. 114 00:05:10,090 --> 00:05:13,400 And now, they're moving on to something slightly more difficult. 115 00:05:13,400 --> 00:05:16,840 And, of course, because I'm operating in a different university framework 116 00:05:16,840 --> 00:05:19,240 and particularly in higher education, I have 117 00:05:19,240 --> 00:05:22,060 to fit within the structure of the larger university 118 00:05:22,060 --> 00:05:27,160 and can't just teach CS50 directly without some longer term changes. 119 00:05:27,160 --> 00:05:31,150 So CS50, as we know, is a lot of resources, plenty of staff, 120 00:05:31,150 --> 00:05:34,273 administrative support, certain financial support. 121 00:05:34,273 --> 00:05:36,190 And also from a practical perspective, there's 122 00:05:36,190 --> 00:05:40,820 a lot that actually makes it work the way it's taught here at Harvard College. 123 00:05:40,820 --> 00:05:45,380 They also have a lot of TAs and just like me, a very large enrollment. 124 00:05:45,380 --> 00:05:48,910 However, unlike me, they also have to cater to all these very 125 00:05:48,910 --> 00:05:51,260 substantial external actors. 126 00:05:51,260 --> 00:05:54,760 And so taking that aside for a minute, catering just to my classroom 127 00:05:54,760 --> 00:05:57,910 or just to my university, it gives me a little bit more flexibility 128 00:05:57,910 --> 00:05:58,700 around what I do. 129 00:05:58,700 --> 00:06:03,010 I don't have to worry about my 1.4 million YouTube subscribers. 130 00:06:03,010 --> 00:06:07,460 So what does my class look like as opposed to what CS50 is like? 131 00:06:07,460 --> 00:06:09,050 What changes do I have to make? 132 00:06:09,050 --> 00:06:12,490 Well, firstly, we've only got two professors, 133 00:06:12,490 --> 00:06:14,330 and we're very, very part-time. 134 00:06:14,330 --> 00:06:19,030 So I spend half of one semester I can dedicate to this class. 135 00:06:19,030 --> 00:06:23,210 And it's only about 0.2 of a full-time position for each of us. 136 00:06:23,210 --> 00:06:27,460 This equates to 400 hours of work per semester, 137 00:06:27,460 --> 00:06:31,030 give or take, which is 36 hours of lectures, where we're actually 138 00:06:31,030 --> 00:06:33,220 standing in front of the class. 139 00:06:33,220 --> 00:06:37,640 And then we also do have a little bit of TA time, but not very much. 140 00:06:37,640 --> 00:06:42,640 So beyond the sections that we run every week, we are given two hours of TA 141 00:06:42,640 --> 00:06:45,330 time per student that's enrolled to make sure 142 00:06:45,330 --> 00:06:47,830 that we get all the grading done and all the additional help 143 00:06:47,830 --> 00:06:49,430 that that student might need. 144 00:06:49,430 --> 00:06:52,910 There are no other resources that the university provides. 145 00:06:52,910 --> 00:06:56,570 So everything that we have to do, we have to do on this smaller budget. 146 00:06:56,570 --> 00:07:01,970 And this is really important because almost none of us are here at CS50. 147 00:07:01,970 --> 00:07:05,950 I'm very lucky to be here today, but the rest of us are in our own scenarios 148 00:07:05,950 --> 00:07:08,090 where our resources are going to look different. 149 00:07:08,090 --> 00:07:10,060 And so whatever your resources are, you're 150 00:07:10,060 --> 00:07:15,190 going to have to think, OK, how do I spend my resources to get the best 151 00:07:15,190 --> 00:07:15,980 bang for buck? 152 00:07:15,980 --> 00:07:19,990 What are the things that CS50 does that are really effective, that you don't 153 00:07:19,990 --> 00:07:22,690 need a large budget for, that you don't need a giant staff for, 154 00:07:22,690 --> 00:07:26,570 that you can do in your own classroom? 155 00:07:26,570 --> 00:07:31,380 So here's where we take seriously one half of CS50's invitation. 156 00:07:31,380 --> 00:07:36,270 So CS50's invitation is to adopt, but it's also to adapt. 157 00:07:36,270 --> 00:07:40,200 This means that you don't have to necessarily take the whole course as is. 158 00:07:40,200 --> 00:07:44,010 You can take the pieces that work very well for your setting. 159 00:07:44,010 --> 00:07:46,490 These pieces might be the lectures, but they 160 00:07:46,490 --> 00:07:49,290 might be some of the pedagogy, some of the class policies. 161 00:07:49,290 --> 00:07:51,300 Maybe it's the way they handle extensions. 162 00:07:51,300 --> 00:07:53,030 Maybe it's some of the tooling. 163 00:07:53,030 --> 00:07:55,160 All of these different things are components 164 00:07:55,160 --> 00:07:57,770 that you could consider incorporating to your class, 165 00:07:57,770 --> 00:08:01,790 whether you're teaching CS50 itself or something a little different, 166 00:08:01,790 --> 00:08:04,270 like the class that I teach. 167 00:08:04,270 --> 00:08:07,690 So our theme here is, can we get 80% of what 168 00:08:07,690 --> 00:08:11,860 CS50 is doing in each of these different realms with only, let's say, 169 00:08:11,860 --> 00:08:13,210 5% of the inputs-- 170 00:08:13,210 --> 00:08:18,890 5% of the staff time, 5% of the finances, 5% of the practical support? 171 00:08:18,890 --> 00:08:21,270 So what makes CS50 work? 172 00:08:21,270 --> 00:08:24,030 What are the different pieces that we could potentially take? 173 00:08:24,030 --> 00:08:27,090 You've heard a lot, particularly in Guy's talk yesterday, 174 00:08:27,090 --> 00:08:29,400 he spoke a lot about what makes CS50 tick. 175 00:08:29,400 --> 00:08:32,120 And this is my version of the different pieces 176 00:08:32,120 --> 00:08:34,470 that I think you could potentially select from. 177 00:08:34,470 --> 00:08:37,580 And we'll go through these one by one and talk about the pieces 178 00:08:37,580 --> 00:08:41,150 that I was able to take for my class and how they were effective, 179 00:08:41,150 --> 00:08:43,710 even in a lower resourced environment. 180 00:08:43,710 --> 00:08:45,890 So we'll start off with curriculum, and we'll 181 00:08:45,890 --> 00:08:48,860 talk first about where my curriculum began 182 00:08:48,860 --> 00:08:52,790 when I joined the university adopting a class that already existed. 183 00:08:52,790 --> 00:08:55,490 And then where we've ended up today. 184 00:08:55,490 --> 00:08:58,700 So the CS50 curriculum looks something like this-- 185 00:08:58,700 --> 00:09:02,540 a week of very basic introduction to programming in Scratch, 186 00:09:02,540 --> 00:09:07,760 followed by some substantial work in C, talking a bit about algorithms 187 00:09:07,760 --> 00:09:08,610 in that component. 188 00:09:08,610 --> 00:09:10,310 We all remember the Tiedeman assignment. 189 00:09:10,310 --> 00:09:12,590 Then a week to get us ready to go to Python 190 00:09:12,590 --> 00:09:15,920 and then three weeks that really take the skills that students 191 00:09:15,920 --> 00:09:19,290 have learned earlier and putting them in a setting where 192 00:09:19,290 --> 00:09:21,310 students can build a real web app. 193 00:09:21,310 --> 00:09:24,870 So there are three weeks dedicated to learning SQL, Flask, and then 194 00:09:24,870 --> 00:09:27,510 the project that allows students to really take 195 00:09:27,510 --> 00:09:29,700 that foundational knowledge they've used before 196 00:09:29,700 --> 00:09:33,470 and demonstrate it in a very broad and flexible way. 197 00:09:33,470 --> 00:09:37,220 The Melbourne curriculum before I joined-- so I joined in 2021-- 198 00:09:37,220 --> 00:09:40,200 looked a little similar, but also a little different. 199 00:09:40,200 --> 00:09:44,420 So there was six weeks of Introduction to programming in C 200 00:09:44,420 --> 00:09:47,270 and there was a lot of discussion of algorithms. 201 00:09:47,270 --> 00:09:51,740 And then the final six weeks of the class in our 12-week semester was spent 202 00:09:51,740 --> 00:09:55,700 dedicated to various different types of data structures and algorithms, 203 00:09:55,700 --> 00:09:58,740 which is appropriate given that this wasn't their very first class. 204 00:09:58,740 --> 00:10:01,830 They'd already had some exposure to programming before. 205 00:10:01,830 --> 00:10:05,390 So the next six weeks spent a little time going through some basic data 206 00:10:05,390 --> 00:10:08,830 structures and basic algorithms. 207 00:10:08,830 --> 00:10:14,290 But CS50 seems to cover more in less time during the semester 208 00:10:14,290 --> 00:10:16,730 than we were covering in a full 12 weeks. 209 00:10:16,730 --> 00:10:21,340 So in just 10 weeks, CS50 is cramming in a lot of content. 210 00:10:21,340 --> 00:10:23,650 It covers a lot of the algorithm's content 211 00:10:23,650 --> 00:10:26,110 that we did, perhaps not quite as far, but then 212 00:10:26,110 --> 00:10:29,050 it also managed to get all the way through teaching Python, 213 00:10:29,050 --> 00:10:32,440 substantial amounts of web development, and all that in really 214 00:10:32,440 --> 00:10:34,310 just 10 weeks of lectures. 215 00:10:34,310 --> 00:10:35,930 So how is that done? 216 00:10:35,930 --> 00:10:38,770 Well, you could say all we need to do is we 217 00:10:38,770 --> 00:10:43,120 should take exactly the things that CS50 teaches and then put them 218 00:10:43,120 --> 00:10:47,090 in our classroom, and everything will instantly be fixed. 219 00:10:47,090 --> 00:10:51,470 Unfortunately, the reason that CS50 works is not just because of the things 220 00:10:51,470 --> 00:10:52,640 they've chosen to teach. 221 00:10:52,640 --> 00:10:56,120 And so it's a mistake to adopt someone else's content whole scale 222 00:10:56,120 --> 00:11:00,810 without understanding how it's delivered and why it works in their context. 223 00:11:00,810 --> 00:11:03,860 So you also have to think about the other components of CS50 224 00:11:03,860 --> 00:11:07,980 that make teaching this class in 10 weeks a viable proposition. 225 00:11:07,980 --> 00:11:12,020 So this includes the pedagogy, the way they think about how they're teaching 226 00:11:12,020 --> 00:11:15,410 and how things are structured, the quality of the delivery, 227 00:11:15,410 --> 00:11:18,410 and the excitement of the delivery that engages students and gets 228 00:11:18,410 --> 00:11:22,730 them to put in that necessary effort, as well as the assessment components, which 229 00:11:22,730 --> 00:11:26,210 is really where students do a lot of their self-driven learning. 230 00:11:26,210 --> 00:11:30,020 And ultimately, where they're going to pick up the skills that they see 231 00:11:30,020 --> 00:11:31,440 demonstrated in lecture. 232 00:11:31,440 --> 00:11:34,880 So if we don't have students putting in the same amount of effort and time, 233 00:11:34,880 --> 00:11:38,630 we're probably not going to be able to cover the same content in 10 weeks 234 00:11:38,630 --> 00:11:41,470 because students will fall behind. 235 00:11:41,470 --> 00:11:44,950 So now that we've taken at least a first look at curriculum, 236 00:11:44,950 --> 00:11:49,240 and we'll come back a bit later to where we landed, we should talk about some 237 00:11:49,240 --> 00:11:55,030 of these other ingredients that will enable us to make this curricular shift. 238 00:11:55,030 --> 00:11:58,510 So one thing that students often say about CS50, 239 00:11:58,510 --> 00:12:01,810 and something that I think I've managed to accomplish in my classroom, 240 00:12:01,810 --> 00:12:05,180 is to make the class feel a bit like a movie. 241 00:12:05,180 --> 00:12:07,760 So we've all watched the CS50 lectures. 242 00:12:07,760 --> 00:12:13,300 We see how it's presented in super high production quality that's filmed in 4K, 243 00:12:13,300 --> 00:12:16,570 ultra high definition, high dynamic range, 244 00:12:16,570 --> 00:12:20,710 all these little buzzwords that end up giving you a good picture and good sound 245 00:12:20,710 --> 00:12:21,310 quality. 246 00:12:21,310 --> 00:12:23,650 And particularly with the sound, students 247 00:12:23,650 --> 00:12:27,840 notice when what they're seeing in front of them, if it looks good or looks bad. 248 00:12:27,840 --> 00:12:31,420 Now, many of my students end up watching my lecture videos 249 00:12:31,420 --> 00:12:33,580 instead of actually coming to class, which 250 00:12:33,580 --> 00:12:36,130 is a common phenomenon in many parts of the world, 251 00:12:36,130 --> 00:12:40,210 as students are increasingly using the flexibility of being able to pause 252 00:12:40,210 --> 00:12:42,020 lectures, slow down, speed them up. 253 00:12:42,020 --> 00:12:45,280 So this is what CS50 looks like. 254 00:12:45,280 --> 00:12:48,280 And this is what Foundations of Algorithms 255 00:12:48,280 --> 00:12:50,680 looked like my very first year when I joined 256 00:12:50,680 --> 00:12:56,880 and was using the built-in projector system and lecture capture in the room. 257 00:12:56,880 --> 00:12:59,380 And depending on what kind of institution you're looking at, 258 00:12:59,380 --> 00:13:02,150 you may have this, or you may have nothing whatsoever. 259 00:13:02,150 --> 00:13:05,180 So you can't really see what's going on there. 260 00:13:05,180 --> 00:13:08,560 So I didn't think my students would rather learn from that. 261 00:13:08,560 --> 00:13:10,960 But within about a year, I was able to get something that 262 00:13:10,960 --> 00:13:13,570 looked much clearer and much better. 263 00:13:13,570 --> 00:13:17,890 And depending on how much you like playing with cameras and toys, 264 00:13:17,890 --> 00:13:21,670 you can go on a wide spectrum from I'm going to have a nice webcam 265 00:13:21,670 --> 00:13:24,280 and I'm going to shine a light appropriately on my face 266 00:13:24,280 --> 00:13:28,220 so that students can see me all the way to a multi-camera production. 267 00:13:28,220 --> 00:13:30,490 And so because I'm one of those more techie people, 268 00:13:30,490 --> 00:13:33,100 I thought, you know, let's figure out how 269 00:13:33,100 --> 00:13:36,340 we can do this slightly more sophisticated recording 270 00:13:36,340 --> 00:13:38,690 setup on a much smaller budget. 271 00:13:38,690 --> 00:13:44,140 Now, again, what I'm about to talk about I did with not a tiny budget. 272 00:13:44,140 --> 00:13:46,850 I spent a little bit of money and built a medium budget. 273 00:13:46,850 --> 00:13:50,790 But you can do this on your own sliding scale for what's appropriate for you. 274 00:13:50,790 --> 00:13:53,920 You don't need to go all the way to CS50 the way Harvard does it. 275 00:13:53,920 --> 00:13:56,170 You don't even need to go all the way to what I do it. 276 00:13:56,170 --> 00:13:58,960 Again, the message is, what are the small changes you can 277 00:13:58,960 --> 00:14:02,580 make that will have the big outcome? 278 00:14:02,580 --> 00:14:05,770 So our goal here in this component was to make 279 00:14:05,770 --> 00:14:09,940 the lectures memorable, accessible, and high quality 280 00:14:09,940 --> 00:14:14,320 by using video and audio production techniques to take another step up 281 00:14:14,320 --> 00:14:16,340 from what they were getting before. 282 00:14:16,340 --> 00:14:20,510 And this is something that CS50 has thought very heavily and very carefully 283 00:14:20,510 --> 00:14:24,890 about, including in this one paper that goes beyond just the fancy cameras 284 00:14:24,890 --> 00:14:28,730 to talk about elements of theatricality that get brought into the classroom. 285 00:14:28,730 --> 00:14:32,240 How to take techniques from drama, production design, staging, 286 00:14:32,240 --> 00:14:34,520 that sort of thing, to make students feel 287 00:14:34,520 --> 00:14:37,820 like they're more immersed in what's going on and not just sitting 288 00:14:37,820 --> 00:14:41,750 watching someone stand at a lectern, as you might be doing for the next half 289 00:14:41,750 --> 00:14:43,500 hour, giving a bit of a talk. 290 00:14:43,500 --> 00:14:46,730 So to actually bring them into the world of the lecture. 291 00:14:46,730 --> 00:14:51,170 So one way that CS50 does this, and that we do this fairly affordably, 292 00:14:51,170 --> 00:14:52,530 is to use props. 293 00:14:52,530 --> 00:14:56,120 And these props don't have to be super fancy stuff made 294 00:14:56,120 --> 00:14:57,720 by a special prop designer. 295 00:14:57,720 --> 00:14:59,070 They can be small. 296 00:14:59,070 --> 00:15:02,060 You can see in this photo, potentially, one of the students 297 00:15:02,060 --> 00:15:03,680 is holding up a number two. 298 00:15:03,680 --> 00:15:06,590 And I'm teaching sorting using house numbers 299 00:15:06,590 --> 00:15:10,080 that I bought from a local hardware store for about $5 each. 300 00:15:10,080 --> 00:15:13,790 Now, while, of course, you could go bigger and better, even 301 00:15:13,790 --> 00:15:17,040 this small change, students already felt like they were more 302 00:15:17,040 --> 00:15:20,350 immersed than just drawing the numbers on the whiteboard. 303 00:15:20,350 --> 00:15:24,340 On the right here, you can see the big blue brain. 304 00:15:24,340 --> 00:15:27,150 And this was part of an art exhibit that was being thrown away 305 00:15:27,150 --> 00:15:28,760 after the art exhibit was being done. 306 00:15:28,760 --> 00:15:32,200 And I thought, you know, I could bring this costume into my classroom. 307 00:15:32,200 --> 00:15:35,250 And our students get excited whenever we bring out the big blue brain 308 00:15:35,250 --> 00:15:37,620 because it means that a student volunteer is 309 00:15:37,620 --> 00:15:40,380 going to be selected to come up to the front of the classroom 310 00:15:40,380 --> 00:15:42,700 and do some exciting demonstration. 311 00:15:42,700 --> 00:15:47,410 So the big blue brain itself doesn't have any inherent educational value. 312 00:15:47,410 --> 00:15:50,610 But by incorporating some elements of design and creativity 313 00:15:50,610 --> 00:15:54,760 into the classroom, you can get students excited about the content itself. 314 00:15:54,760 --> 00:15:57,660 And eventually, these theatrical elements 315 00:15:57,660 --> 00:16:01,000 become representative of substantive course elements. 316 00:16:01,000 --> 00:16:06,510 So by creating that link between the fun stuff and the substantive stuff, 317 00:16:06,510 --> 00:16:10,320 you can begin to create links in the students' minds 318 00:16:10,320 --> 00:16:13,470 that actually enable them to learn the material better 319 00:16:13,470 --> 00:16:16,520 and to get excited about the material. 320 00:16:16,520 --> 00:16:20,630 So this is what it actually looks like up close from my little control panel. 321 00:16:20,630 --> 00:16:23,630 Now, I don't have as good a way of hiding what's going on, 322 00:16:23,630 --> 00:16:27,470 but this actually makes it handy to explain how I'm running my class. 323 00:16:27,470 --> 00:16:30,320 So you can see the same kind of thing that's 324 00:16:30,320 --> 00:16:34,460 happening in CS50, just with the cables hidden away a little better. 325 00:16:34,460 --> 00:16:39,290 So essentially, what's going on-- this is a very schematic stick figure 326 00:16:39,290 --> 00:16:43,910 diagram of it-- is I have two laptops, one of which is controlling the code 327 00:16:43,910 --> 00:16:45,900 and one of which is controlling the slides. 328 00:16:45,900 --> 00:16:47,810 And then sometimes I'll plug an iPad in. 329 00:16:47,810 --> 00:16:51,500 And then I have a cheap little device-- you can get these for as little as a few 330 00:16:51,500 --> 00:16:52,440 hundred dollars-- 331 00:16:52,440 --> 00:16:56,160 that enables me to switch what's on the screen at any given time. 332 00:16:56,160 --> 00:16:58,400 And just by pressing a button, then I can quickly 333 00:16:58,400 --> 00:17:02,070 switch between the code and the presentation material. 334 00:17:02,070 --> 00:17:05,250 And this ends up actually saving a lot of class time. 335 00:17:05,250 --> 00:17:08,119 I calculated it saves me probably six minutes 336 00:17:08,119 --> 00:17:11,640 per lecture of futzing with things and switching between things. 337 00:17:11,640 --> 00:17:16,810 And this is an extraordinary amount in a lecture that's only 55 minutes long. 338 00:17:16,810 --> 00:17:19,800 So by investing a little bit in technology, not only does it make 339 00:17:19,800 --> 00:17:22,380 the presentation look cleaner because students 340 00:17:22,380 --> 00:17:25,800 see these fancy transitions between the slides and the code, 341 00:17:25,800 --> 00:17:29,490 but it actually saves time and allows me to focus on the teaching 342 00:17:29,490 --> 00:17:33,300 and a little less on the technology, even while requiring a little more 343 00:17:33,300 --> 00:17:34,540 thought to begin with. 344 00:17:34,540 --> 00:17:36,820 But you don't necessarily have to do that. 345 00:17:36,820 --> 00:17:40,690 But if you do want to, it does cost a little bit at the end of the day. 346 00:17:40,690 --> 00:17:43,710 So this was my personal investment into the class 347 00:17:43,710 --> 00:17:45,940 with a mix of camera equipment. 348 00:17:45,940 --> 00:17:49,950 Now, again, what I really, really want to emphasize here is you don't have 349 00:17:49,950 --> 00:17:51,700 to spend the $14,000. 350 00:17:51,700 --> 00:17:53,830 You don't have to spend $140,000. 351 00:17:53,830 --> 00:17:56,260 You could do this with $140. 352 00:17:56,260 --> 00:17:59,670 And just the types of things that you buy, depending on your situation, 353 00:17:59,670 --> 00:18:01,120 will be a little different. 354 00:18:01,120 --> 00:18:03,720 And so just like me, you might soon, too, 355 00:18:03,720 --> 00:18:07,410 be lugging a suitcase full of items from your office 356 00:18:07,410 --> 00:18:10,140 to where you actually end up teaching, whether that's props, 357 00:18:10,140 --> 00:18:11,190 whether that's cameras. 358 00:18:11,190 --> 00:18:15,940 Bringing extra stuff into the classroom can help enhance the experience. 359 00:18:15,940 --> 00:18:18,340 As well after the lecture, one thing that I 360 00:18:18,340 --> 00:18:20,980 do that I find very effective as an educator 361 00:18:20,980 --> 00:18:24,320 is actually going back and watching a recording of what I've done. 362 00:18:24,320 --> 00:18:27,010 Now, I do spend a lot of time editing what I've done 363 00:18:27,010 --> 00:18:28,610 and finding the bits that worked. 364 00:18:28,610 --> 00:18:30,520 And one of the surprising things is I find 365 00:18:30,520 --> 00:18:35,570 sometimes saying less actually results in students learning more. 366 00:18:35,570 --> 00:18:38,540 So sometimes I'll have an explanation that missed the mark. 367 00:18:38,540 --> 00:18:41,950 And if I cut that out and make a note to myself the next time-- next time, 368 00:18:41,950 --> 00:18:44,920 don't give this explanation-- sometimes that's just 369 00:18:44,920 --> 00:18:47,750 as good as adding an explanation. 370 00:18:47,750 --> 00:18:50,410 So while the editing process is useful for producing 371 00:18:50,410 --> 00:18:54,130 this very polished finished product, just like happens at CS50, 372 00:18:54,130 --> 00:18:58,030 the more important part of that process is not actually the video editing. 373 00:18:58,030 --> 00:19:02,780 It's watching back myself and then making changes based on what I see. 374 00:19:02,780 --> 00:19:06,950 And this is, again, as someone says in the comments, less is more. 375 00:19:06,950 --> 00:19:09,460 So figuring out what are the important things that 376 00:19:09,460 --> 00:19:10,973 are actually getting things across. 377 00:19:10,973 --> 00:19:11,890 I also see a question. 378 00:19:11,890 --> 00:19:13,940 What software do you use for video editing? 379 00:19:13,940 --> 00:19:18,600 I use DaVinci Resolve, and I believe that's also what the team here uses. 380 00:19:18,600 --> 00:19:22,780 So there also are time costs that I pay to make this happen. 381 00:19:22,780 --> 00:19:25,290 And this is some of that extra 5% investment, 382 00:19:25,290 --> 00:19:26,910 might be a little more than 5%. 383 00:19:26,910 --> 00:19:28,770 Oops, there we go. 384 00:19:28,770 --> 00:19:32,820 So I do invest substantial extra time in my classroom 385 00:19:32,820 --> 00:19:34,680 getting set up for it every time. 386 00:19:34,680 --> 00:19:39,150 Right now, this looks like an hour of additional set up and tear-down time. 387 00:19:39,150 --> 00:19:44,310 There's me setting up all the cables myself and my student production 388 00:19:44,310 --> 00:19:45,150 associate. 389 00:19:45,150 --> 00:19:49,110 And this results in about an extra $3,500 dollars of staffing costs, 390 00:19:49,110 --> 00:19:52,680 which might be relevant if you're at a university budget where you have 391 00:19:52,680 --> 00:19:55,210 to think very carefully about your TA time. 392 00:19:55,210 --> 00:19:57,810 And on top of that, about two hours of my time 393 00:19:57,810 --> 00:20:01,570 for every hour of class time in doing post-production, 394 00:20:01,570 --> 00:20:03,490 so doing this video editing. 395 00:20:03,490 --> 00:20:07,930 Now, one thing that I should note is while this is my current version, when 396 00:20:07,930 --> 00:20:12,620 I started doing this, I did not have an extra hour of setup and tear-down time. 397 00:20:12,620 --> 00:20:16,030 I had the traditional five minutes in between classes. 398 00:20:16,030 --> 00:20:19,930 The hour that I have now is just so I can do it in a relaxed and less frazzled 399 00:20:19,930 --> 00:20:20,480 way. 400 00:20:20,480 --> 00:20:24,080 And I've already lost more hair than I would like doing it the five-minute way. 401 00:20:24,080 --> 00:20:30,080 So giving myself an hour is now my preferred method. 402 00:20:30,080 --> 00:20:34,340 And then, of course, there's the fun aspect of creating catchy-- 403 00:20:34,340 --> 00:20:37,500 these are YouTube thumbnails that I've used for some of my slides. 404 00:20:37,500 --> 00:20:40,850 But more of the point here is creating a fun theme 405 00:20:40,850 --> 00:20:43,230 for every class that kind of hooks students in. 406 00:20:43,230 --> 00:20:45,110 And there are different ways of doing this, 407 00:20:45,110 --> 00:20:48,830 whether this is a demo at the start of the class or a fun thumbnail 408 00:20:48,830 --> 00:20:52,320 that you post to students to get them hyped before the class begins. 409 00:20:52,320 --> 00:20:55,408 This is another method of drawing students in. 410 00:20:55,408 --> 00:20:56,325 [BEGIN VIDEO PLAYBACK] 411 00:20:56,325 --> 00:20:57,750 [MUSIC PLAYING] 412 00:20:57,750 --> 00:21:02,510 413 00:21:02,510 --> 00:21:04,775 - This is the magical star of dereferencing. 414 00:21:04,775 --> 00:21:08,660 The power of the magical star of dereferencing 415 00:21:08,660 --> 00:21:11,150 is to open Bruce's mailbox. 416 00:21:11,150 --> 00:21:13,495 So open Bruce's mailbox, take out what's inside. 417 00:21:13,495 --> 00:21:14,840 [MUSIC PLAYING] 418 00:21:14,840 --> 00:21:16,220 419 00:21:16,220 --> 00:21:17,930 OK, what do you see there? 420 00:21:17,930 --> 00:21:19,310 - 0x50. 421 00:21:19,310 --> 00:21:22,340 - The other thing that happens when you use the star of dereferencing 422 00:21:22,340 --> 00:21:25,075 is you now go to the location on the paper. 423 00:21:25,075 --> 00:21:26,570 [MUSIC PLAYING] 424 00:21:26,570 --> 00:21:27,070 425 00:21:27,070 --> 00:21:29,990 So 0x50 matches 0x50. 426 00:21:29,990 --> 00:21:32,077 And we read out what you find there. 427 00:21:32,077 --> 00:21:32,910 So what do you find? 428 00:21:32,910 --> 00:21:34,371 [MUSIC PLAYING] 429 00:21:34,371 --> 00:21:38,177 430 00:21:38,177 --> 00:21:38,760 [END PLAYBACK] 431 00:21:38,760 --> 00:21:42,300 SHAANAN COHNEY: So again, this is just another example of the type of content 432 00:21:42,300 --> 00:21:44,100 that you can actually put together pretty 433 00:21:44,100 --> 00:21:46,140 quickly if you are recording your lectures 434 00:21:46,140 --> 00:21:47,950 and then get students very excited. 435 00:21:47,950 --> 00:21:52,210 So I'd say that was about another 20 minutes of effort after the fact. 436 00:21:52,210 --> 00:21:54,930 And students then started sharing that around with friends 437 00:21:54,930 --> 00:21:58,050 and encouraging more people to come to the class, which is, again, 438 00:21:58,050 --> 00:22:00,440 beneficial to the environment overall. 439 00:22:00,440 --> 00:22:03,590 440 00:22:03,590 --> 00:22:07,550 So while I've spoken a fair bit about production and technology, I've said, 441 00:22:07,550 --> 00:22:10,740 you know what if you buy this other camera, what if you film your lectures. 442 00:22:10,740 --> 00:22:13,650 These are all cool, and they're definitely added bonuses. 443 00:22:13,650 --> 00:22:16,650 But I would say that they're definitely not enough on their own. 444 00:22:16,650 --> 00:22:19,370 You can have the world's most beautiful looking lectures, 445 00:22:19,370 --> 00:22:21,530 but if you don't have the other ingredients, 446 00:22:21,530 --> 00:22:23,720 you haven't actually done anything other than create 447 00:22:23,720 --> 00:22:25,830 a very slick looking YouTube video. 448 00:22:25,830 --> 00:22:29,300 So here is an example of actually taking some of these elements 449 00:22:29,300 --> 00:22:31,340 and combining them for pedagogical value. 450 00:22:31,340 --> 00:22:35,300 Now, CS50 has done a number of similar exercises in the past. 451 00:22:35,300 --> 00:22:38,480 I'm going to let it run for one minute so you can see what kinds of things 452 00:22:38,480 --> 00:22:41,780 you can do in the classroom with relatively simple props that actually 453 00:22:41,780 --> 00:22:46,170 use the props as a way to hook students into the material you're teaching, 454 00:22:46,170 --> 00:22:47,950 and really deliver pedagogical value. 455 00:22:47,950 --> 00:22:48,867 [BEGIN VIDEO PLAYBACK] 456 00:22:48,867 --> 00:22:53,220 - So first card and then the second card. 457 00:22:53,220 --> 00:22:57,570 The first card will close, then the third card. 458 00:22:57,570 --> 00:22:59,010 And then the-- 459 00:22:59,010 --> 00:23:00,480 - Second card closes. 460 00:23:00,480 --> 00:23:02,070 - Fourth card. 461 00:23:02,070 --> 00:23:03,575 And then they will switch. 462 00:23:03,575 --> 00:23:08,560 463 00:23:08,560 --> 00:23:10,120 And everything-- 464 00:23:10,120 --> 00:23:10,750 - Over here. 465 00:23:10,750 --> 00:23:13,150 - OK, then the first card. 466 00:23:13,150 --> 00:23:14,900 - OK, you guys are getting the hang of it. 467 00:23:14,900 --> 00:23:16,600 Let's speed it up slightly. 468 00:23:16,600 --> 00:23:18,010 We're almost through. 469 00:23:18,010 --> 00:23:23,740 - Third card, fourth card, fifth card, sixth card. 470 00:23:23,740 --> 00:23:24,770 That's right. 471 00:23:24,770 --> 00:23:26,630 - Yep, looks good to me so far. 472 00:23:26,630 --> 00:23:30,060 - Card, and eighth card. 473 00:23:30,060 --> 00:23:32,020 [LAUGHTER] 474 00:23:32,020 --> 00:23:34,300 All right. 475 00:23:34,300 --> 00:23:36,250 OK. 476 00:23:36,250 --> 00:23:38,210 First, second, third. 477 00:23:38,210 --> 00:23:40,090 They switch second and third. 478 00:23:40,090 --> 00:23:42,670 And then first and second switch. 479 00:23:42,670 --> 00:23:46,150 Yeah, that's sorted. 480 00:23:46,150 --> 00:23:47,890 I think we need to go through and verify. 481 00:23:47,890 --> 00:23:51,140 So one, two, three, four, five, six, seven. 482 00:23:51,140 --> 00:23:52,345 - OK, everyone turn around. 483 00:23:52,345 --> 00:23:53,890 - And that's right. 484 00:23:53,890 --> 00:23:55,890 - Can we have a big round of applause for Disan? 485 00:23:55,890 --> 00:23:57,176 [APPLAUSE] 486 00:23:57,176 --> 00:23:58,587 487 00:23:58,587 --> 00:23:59,170 [END PLAYBACK] 488 00:23:59,170 --> 00:24:02,320 SHAANAN COHNEY: So while that is a simple sorting exercise, 489 00:24:02,320 --> 00:24:05,230 bringing in these giant human-sized playing cards 490 00:24:05,230 --> 00:24:08,740 suddenly made the experience a lot more tangible, a lot more exciting, 491 00:24:08,740 --> 00:24:12,700 and a lot more memorable through this technique of theatricality 492 00:24:12,700 --> 00:24:16,270 that CS50 uses that we could bring into our own classroom, 493 00:24:16,270 --> 00:24:18,950 despite not actually being CS50 itself. 494 00:24:18,950 --> 00:24:21,910 And you can do this kind of thing, whether it's with sheets of paper, 495 00:24:21,910 --> 00:24:24,820 like David talked about yesterday, with giant playing 496 00:24:24,820 --> 00:24:27,460 cards, house numbers that you buy from the hardware store, 497 00:24:27,460 --> 00:24:29,270 or whatever it is you use. 498 00:24:29,270 --> 00:24:33,550 The techniques of theatricality can be brought into your own classroom 499 00:24:33,550 --> 00:24:35,830 with only a little bit of extra creativity 500 00:24:35,830 --> 00:24:38,260 and a little bit of extra effort. 501 00:24:38,260 --> 00:24:41,350 So one thing that I do that's different from CS50 502 00:24:41,350 --> 00:24:44,890 is I do use a lot of cold calling in my classroom, 503 00:24:44,890 --> 00:24:46,860 trying to remember students' names. 504 00:24:46,860 --> 00:24:50,120 And I find that for my classroom, and at least with the number of students 505 00:24:50,120 --> 00:24:53,000 that actually show up, this is a very helpful approach 506 00:24:53,000 --> 00:24:56,060 for actually getting student involvement and bringing students 507 00:24:56,060 --> 00:24:57,240 to the front of the class. 508 00:24:57,240 --> 00:25:00,440 So this is one of my ingredients that's very highly debated 509 00:25:00,440 --> 00:25:03,050 and educators have very different approaches 510 00:25:03,050 --> 00:25:06,180 to this because students will sometimes have a negative reaction. 511 00:25:06,180 --> 00:25:09,110 But I've found it's valuable in my classroom at least. 512 00:25:09,110 --> 00:25:10,945 So how do you actually do all of this? 513 00:25:10,945 --> 00:25:12,570 How do you learn how to do all of this? 514 00:25:12,570 --> 00:25:14,370 How did I learn how to do all of this? 515 00:25:14,370 --> 00:25:17,820 Well, there's one surprisingly simple answer to it. 516 00:25:17,820 --> 00:25:21,650 517 00:25:21,650 --> 00:25:24,860 I spent a lot of time on YouTube University. 518 00:25:24,860 --> 00:25:28,100 So this is whether it's looking at CS50's material 519 00:25:28,100 --> 00:25:31,400 or looking at other educators or looking at content creators 520 00:25:31,400 --> 00:25:34,400 on the internet who are talking about how to set up a camera so you 521 00:25:34,400 --> 00:25:36,150 don't get glare in your glasses. 522 00:25:36,150 --> 00:25:39,500 There are so many free resources out there for teachers now. 523 00:25:39,500 --> 00:25:42,320 And I would encourage you to actually spend some of your time 524 00:25:42,320 --> 00:25:45,410 as professional development going out there and actually looking 525 00:25:45,410 --> 00:25:49,220 at what other people do and then taking the best elements that are going to fit 526 00:25:49,220 --> 00:25:51,980 your classroom and bringing them in. 527 00:25:51,980 --> 00:25:56,960 I've also found that the listserv for teachers, the email listserv, 528 00:25:56,960 --> 00:25:59,450 has been an incredible source of support. 529 00:25:59,450 --> 00:26:02,060 And CS50 staff have been very helpful if you 530 00:26:02,060 --> 00:26:05,090 do start running against challenges with adopting 531 00:26:05,090 --> 00:26:07,250 some of their tools or their pedagogy and have 532 00:26:07,250 --> 00:26:09,920 been very helpful and very responsive. 533 00:26:09,920 --> 00:26:14,060 So that covers the pedagogy and the engaging delivery and active learning 534 00:26:14,060 --> 00:26:15,050 components. 535 00:26:15,050 --> 00:26:18,530 We spoke a little bit about active learning yesterday in Guy's section. 536 00:26:18,530 --> 00:26:21,230 But there's also some of the tooling that CS50 uses. 537 00:26:21,230 --> 00:26:23,980 And this can be adopted and adapted as well. 538 00:26:23,980 --> 00:26:26,450 So CS50 has their website. 539 00:26:26,450 --> 00:26:31,430 And they have, again, much like many of the other things for their class, 540 00:26:31,430 --> 00:26:33,120 made the source code available. 541 00:26:33,120 --> 00:26:36,770 And so I was able to adapt that and produce a similar looking website 542 00:26:36,770 --> 00:26:40,430 for my own class building off their code, which now enables 543 00:26:40,430 --> 00:26:44,150 my students to have the same kind of positive experience accessing 544 00:26:44,150 --> 00:26:47,030 the material beyond the learning management 545 00:26:47,030 --> 00:26:48,720 system built into the university. 546 00:26:48,720 --> 00:26:52,500 Now, again, your mileage may vary depending on your institution's rules, 547 00:26:52,500 --> 00:26:54,780 but I found that adopting some of the code 548 00:26:54,780 --> 00:26:59,560 that CS50 uses to run their infrastructure very, very helpful. 549 00:26:59,560 --> 00:27:02,140 One other thing that's different from what 550 00:27:02,140 --> 00:27:06,730 CS50 has done that my group has done is tried to do some more automated style 551 00:27:06,730 --> 00:27:11,230 checking, where CS50 is traditionally relied on humans, even though the AI 552 00:27:11,230 --> 00:27:15,520 grading is coming more and more into a part of their infrastructure. 553 00:27:15,520 --> 00:27:17,680 So this was a paper that my team presented 554 00:27:17,680 --> 00:27:21,890 at SIGCSE, which is a computer science education conference in the last year. 555 00:27:21,890 --> 00:27:25,180 And we built a tool that aims to provide students helpful error 556 00:27:25,180 --> 00:27:30,040 messages to understand not just how to fix their bad style in code, 557 00:27:30,040 --> 00:27:33,550 but also what might be a pedagogical learning from it. 558 00:27:33,550 --> 00:27:35,950 And so while the comments that I've shown here 559 00:27:35,950 --> 00:27:41,020 are reduced in size, if you were to click on any of these, 560 00:27:41,020 --> 00:27:44,470 you would see an expanded explanation explaining to the student what 561 00:27:44,470 --> 00:27:46,070 this element of style actually means. 562 00:27:46,070 --> 00:27:48,940 So again, this is an indication of where tooling can actually 563 00:27:48,940 --> 00:27:52,000 make you more efficient as a teacher, giving you more time 564 00:27:52,000 --> 00:27:53,240 for the things that matter. 565 00:27:53,240 --> 00:27:56,320 So a 5% investment in technology in this case 566 00:27:56,320 --> 00:28:00,110 allows us to spend more of our TA time on the one-on-one student 567 00:28:00,110 --> 00:28:03,710 interaction and the things that are harder for computers to deliver, 568 00:28:03,710 --> 00:28:06,890 even while AI tools are advancing and actually 569 00:28:06,890 --> 00:28:10,190 allowing us to do more and more of the substantive hands-on stuff 570 00:28:10,190 --> 00:28:14,260 and replacing more and more elements of what TAs would have used to do. 571 00:28:14,260 --> 00:28:19,570 So, again, with this, the feedback that we got from TAs was universally saying, 572 00:28:19,570 --> 00:28:23,980 I now don't have to spend as much time on the menial tasks and the things that 573 00:28:23,980 --> 00:28:27,160 don't need me, and I can spend more time on the things that 574 00:28:27,160 --> 00:28:29,710 are going to make the real changes to the students. 575 00:28:29,710 --> 00:28:32,410 Helping them through whatever their substantial struggles are. 576 00:28:32,410 --> 00:28:36,010 Helping them cover gaps in understanding and sometimes even 577 00:28:36,010 --> 00:28:38,080 being a helpful ear to them when there are 578 00:28:38,080 --> 00:28:43,700 things going on outside the classroom that are impacting their learning. 579 00:28:43,700 --> 00:28:46,190 So another thing that you have just heard 580 00:28:46,190 --> 00:28:48,440 about that I've managed to integrate into my own class 581 00:28:48,440 --> 00:28:50,390 and I want to give a positive report of that 582 00:28:50,390 --> 00:28:57,690 is using CS50's AI tools to help answer questions on our class discussion board. 583 00:28:57,690 --> 00:29:01,280 Now, you might be surprised, you went through a session before, 584 00:29:01,280 --> 00:29:03,947 and you spent a little-- it took about an hour of building code. 585 00:29:03,947 --> 00:29:06,488 And you might not feel super confident that this is something 586 00:29:06,488 --> 00:29:07,880 that you could roll out tomorrow. 587 00:29:07,880 --> 00:29:14,120 But one of my experiences was CS50 gave this talk, a workshop at SIGCSE 588 00:29:14,120 --> 00:29:16,430 earlier in this year as well, and I was very excited. 589 00:29:16,430 --> 00:29:19,010 And within eight hours of hearing the talk, 590 00:29:19,010 --> 00:29:21,660 I was able to roll it out into my own class. 591 00:29:21,660 --> 00:29:26,120 And so this was a very powerful-- again, adopting what CS50 did, 592 00:29:26,120 --> 00:29:29,210 but using slightly different configuration on the back-end 593 00:29:29,210 --> 00:29:32,540 so that I was able to roll it out with a little less effort and a little 594 00:29:32,540 --> 00:29:34,680 less time than they necessarily did. 595 00:29:34,680 --> 00:29:39,530 So this was again, one of these very effective 5% effort, 596 00:29:39,530 --> 00:29:42,440 but very substantial outcome very quickly. 597 00:29:42,440 --> 00:29:48,080 And we found our outcomes were quite similar to CS50 adopting this tool. 598 00:29:48,080 --> 00:29:51,260 This is also true of the programming environment, 599 00:29:51,260 --> 00:29:55,970 kind of CS50 Codespaces, that has been incredibly effective. 600 00:29:55,970 --> 00:29:58,700 So we've experimented with Codespaces. 601 00:29:58,700 --> 00:30:00,830 Ultimately, in our environment, we decided 602 00:30:00,830 --> 00:30:05,720 to go with edstem, which is a tool that CS50 has used in certain contexts. 603 00:30:05,720 --> 00:30:08,060 One thing that I'm actually a little worried about 604 00:30:08,060 --> 00:30:12,980 as an educator being direct is I don't like having too many of my tools 605 00:30:12,980 --> 00:30:16,350 reliant on infrastructure that I don't control. 606 00:30:16,350 --> 00:30:18,432 The reason is, no matter how good a company is, 607 00:30:18,432 --> 00:30:21,140 I'm always a little bit worried that one day that something might 608 00:30:21,140 --> 00:30:25,170 happen to that company, and they won't be able to support me anymore. 609 00:30:25,170 --> 00:30:29,180 So that's one thing that I very much appreciate about CS50's tooling is 610 00:30:29,180 --> 00:30:32,360 that even when they use something, like Github, which is incredibly helpful 611 00:30:32,360 --> 00:30:34,730 and has been a great partner, they also retain 612 00:30:34,730 --> 00:30:37,490 a lot of control over the infrastructure and are able to deploy it 613 00:30:37,490 --> 00:30:39,890 somewhere else should they need to. 614 00:30:39,890 --> 00:30:42,470 So edstem has been very useful for me, but I 615 00:30:42,470 --> 00:30:47,670 am looking towards adopting some of CS50's cloud tools. 616 00:30:47,670 --> 00:30:51,450 So beyond the technology and infrastructure, 617 00:30:51,450 --> 00:30:55,050 another thing that makes CS50 very, very powerful for students 618 00:30:55,050 --> 00:30:58,110 is this sense of community that's generated 619 00:30:58,110 --> 00:30:59,570 through many of their activities. 620 00:30:59,570 --> 00:31:00,487 [BEGIN VIDEO PLAYBACK] 621 00:31:00,487 --> 00:31:01,908 [MUSIC PLAYING] 622 00:31:01,908 --> 00:31:05,770 623 00:31:05,770 --> 00:31:06,770 - Hi, everyone. 624 00:31:06,770 --> 00:31:09,610 Welcome to the University of Melbourne, and welcome to the School 625 00:31:09,610 --> 00:31:11,630 of Computing and Information Systems. 626 00:31:11,630 --> 00:31:12,820 I'm Jud from CISA. 627 00:31:12,820 --> 00:31:16,210 And today, I'll be taking you on a tour to the First-year Learning 628 00:31:16,210 --> 00:31:17,950 Center located in Melbourne Connect. 629 00:31:17,950 --> 00:31:21,700 Every day, there'll be tutoring for Foundations of Computing and Foundations 630 00:31:21,700 --> 00:31:24,020 of Algorithms, and snacks will be provided. 631 00:31:24,020 --> 00:31:24,585 Come along. 632 00:31:24,585 --> 00:31:25,966 [MUSIC PLAYING] 633 00:31:25,966 --> 00:31:29,420 634 00:31:29,420 --> 00:31:32,845 So this is Melbourne Connect, where the First-year Learning Center is located. 635 00:31:32,845 --> 00:31:37,110 636 00:31:37,110 --> 00:31:41,095 So the Student Learning Center is located on level three. 637 00:31:41,095 --> 00:31:41,595 - Car G. 638 00:31:41,595 --> 00:31:43,415 [MUSIC PLAYING] 639 00:31:43,415 --> 00:31:48,972 640 00:31:48,972 --> 00:31:50,805 - It's located just down this corridor here. 641 00:31:50,805 --> 00:31:52,547 [MUSIC PLAYING] 642 00:31:52,547 --> 00:32:03,800 643 00:32:03,800 --> 00:32:05,750 And it's just here. 644 00:32:05,750 --> 00:32:07,525 Welcome to the First-year Learning Center. 645 00:32:07,525 --> 00:32:09,399 [MUSIC PLAYING] 646 00:32:09,399 --> 00:32:30,015 647 00:32:30,015 --> 00:32:30,598 [END PLAYBACK] 648 00:32:30,598 --> 00:32:32,077 [MUSIC PLAYING] 649 00:32:32,077 --> 00:32:32,813 650 00:32:32,813 --> 00:32:34,980 SHAANAN COHNEY: So one thing I wanted to communicate 651 00:32:34,980 --> 00:32:37,230 by showing you that is if you actually look 652 00:32:37,230 --> 00:32:39,330 at the physical space in that video, there's 653 00:32:39,330 --> 00:32:41,530 nothing inherently exciting about it. 654 00:32:41,530 --> 00:32:44,310 It's a room in an office building. 655 00:32:44,310 --> 00:32:48,510 But the idea of marking it out as a unique space for students 656 00:32:48,510 --> 00:32:51,660 and trying to create that sense of community through things 657 00:32:51,660 --> 00:32:53,940 like having snacks there on a regular basis. 658 00:32:53,940 --> 00:32:56,730 Through having our office hours in a consistent location 659 00:32:56,730 --> 00:33:00,330 has over the year, or the year and a half since I've been running it, 660 00:33:00,330 --> 00:33:05,970 created an enormous vibe of connectedness that previously was not 661 00:33:05,970 --> 00:33:08,170 part of our major altogether. 662 00:33:08,170 --> 00:33:13,320 Now, it's been incredibly useful from an educational perspective 663 00:33:13,320 --> 00:33:16,020 because we now have a drop-in center where 664 00:33:16,020 --> 00:33:17,820 students can get one-on-one support. 665 00:33:17,820 --> 00:33:21,030 But a very substantial outcome from it has 666 00:33:21,030 --> 00:33:23,610 been this notion of students feeling more tied to the place 667 00:33:23,610 --> 00:33:25,740 that they're learning, which, again, actually ends 668 00:33:25,740 --> 00:33:28,860 up improving their learning outcomes. 669 00:33:28,860 --> 00:33:31,260 Another way that we work to build community 670 00:33:31,260 --> 00:33:33,720 is by partnering with CS50 on some of the events 671 00:33:33,720 --> 00:33:37,010 that we don't necessarily have the resources to plan ourselves. 672 00:33:37,010 --> 00:33:39,090 One of these that's been a big success for us 673 00:33:39,090 --> 00:33:44,460 has been CS50 Puzzle Day, where the CS50 team puts together a sheet of puzzles 674 00:33:44,460 --> 00:33:45,970 that aren't coding problems. 675 00:33:45,970 --> 00:33:51,570 They're just logic puzzles and games that students compete in teams together 676 00:33:51,570 --> 00:33:54,510 to finish during a set time period. 677 00:33:54,510 --> 00:33:57,450 And having these kind of extracurricular activities 678 00:33:57,450 --> 00:34:01,770 helps to build community, which then plays back into the classroom itself 679 00:34:01,770 --> 00:34:05,580 and makes the internal classroom work a lot better, despite the fact 680 00:34:05,580 --> 00:34:09,659 that these things don't inherently have a connection to the material. 681 00:34:09,659 --> 00:34:13,770 So our outcomes for developing this center over the last year and a half 682 00:34:13,770 --> 00:34:15,070 are pretty incredible. 683 00:34:15,070 --> 00:34:18,840 So we've managed to get eight hours of student clubs coming in 684 00:34:18,840 --> 00:34:22,469 to help out other students, along with the 20 hours of formal support 685 00:34:22,469 --> 00:34:23,920 that I was able to arrange. 686 00:34:23,920 --> 00:34:28,230 And within just one year, we had more than 4,000 individual student visits 687 00:34:28,230 --> 00:34:31,550 to the space and 1,000 one-on-one support sessions. 688 00:34:31,550 --> 00:34:34,120 So these numbers are pretty substantial for our class, which 689 00:34:34,120 --> 00:34:36,460 means that each student was, on average, accessing 690 00:34:36,460 --> 00:34:41,810 this space at least a few times during the course of the year. 691 00:34:41,810 --> 00:34:44,810 So the last thing that I haven't quite solved, 692 00:34:44,810 --> 00:34:49,010 and I would love to say, you know, I have managed to get all the way to CS50, 693 00:34:49,010 --> 00:34:50,780 but the truth is all of us are striving. 694 00:34:50,780 --> 00:34:53,909 Even CS50 is making changes year-on-year. 695 00:34:53,909 --> 00:34:57,920 And one area where I still think there's a lot of work for my particular class 696 00:34:57,920 --> 00:34:59,130 is assessment. 697 00:34:59,130 --> 00:35:02,720 I am constrained a lot by the particular rules of my university. 698 00:35:02,720 --> 00:35:04,850 There is a nine-month long approval process 699 00:35:04,850 --> 00:35:08,620 for any change in assessment structure, which definitely slows down innovation. 700 00:35:08,620 --> 00:35:13,100 However, I have had success in moving from large exams 701 00:35:13,100 --> 00:35:17,720 to more CS50 style weekly assignments and weekly problem sets 702 00:35:17,720 --> 00:35:21,860 that we found really does ensure that students get started a little earlier. 703 00:35:21,860 --> 00:35:24,780 And that is generally considered to be good educational practice. 704 00:35:24,780 --> 00:35:27,320 So we often talk about, in the world of education, 705 00:35:27,320 --> 00:35:30,020 the difference between formative and summative assessment, where 706 00:35:30,020 --> 00:35:33,150 summative assessment gets students to show what they've learned at the end. 707 00:35:33,150 --> 00:35:36,680 Whereas formative assessment is about building their skills earlier on 708 00:35:36,680 --> 00:35:37,590 in semester. 709 00:35:37,590 --> 00:35:40,370 So by reducing the stakes and the amount of value 710 00:35:40,370 --> 00:35:43,230 that students place on the summative assessment 711 00:35:43,230 --> 00:35:45,960 and shifting some of that to the actual learning process, 712 00:35:45,960 --> 00:35:49,440 we found that students are feeling much more satisfied and indeed, 713 00:35:49,440 --> 00:35:51,900 their scores are increasing. 714 00:35:51,900 --> 00:35:54,610 So is this all worth it in the end? 715 00:35:54,610 --> 00:35:56,620 There have been a lot of changes that I've made. 716 00:35:56,620 --> 00:35:57,750 I've put in a lot of time. 717 00:35:57,750 --> 00:36:02,080 Even if it's just 5% of what CS50 is doing, we still have to ask, is it worth 718 00:36:02,080 --> 00:36:02,580 it? 719 00:36:02,580 --> 00:36:07,510 So are we getting this 80% of the outcome on 5% of the resources? 720 00:36:07,510 --> 00:36:10,930 So just like any other teacher, I've got the survey, 721 00:36:10,930 --> 00:36:12,860 end of semester survey things. 722 00:36:12,860 --> 00:36:15,740 So the results did go up very markedly. 723 00:36:15,740 --> 00:36:18,820 But as we know, surveys are not really the best way 724 00:36:18,820 --> 00:36:20,690 of testing whether students learn. 725 00:36:20,690 --> 00:36:24,070 If you make your assignments easier, your assessment, 726 00:36:24,070 --> 00:36:28,820 your end of semester scores go up no matter how good a teacher you were. 727 00:36:28,820 --> 00:36:31,090 So this only tells us so much. 728 00:36:31,090 --> 00:36:34,420 Most of the best outcomes that we find we're getting 729 00:36:34,420 --> 00:36:36,640 are actually outside of the traditional measurement 730 00:36:36,640 --> 00:36:38,840 instruments of surveys and grades. 731 00:36:38,840 --> 00:36:42,370 We've had a lot of very positive student feedback, 732 00:36:42,370 --> 00:36:45,850 including long thank you letters coming back years, 733 00:36:45,850 --> 00:36:49,390 students saying, you know, I took your course once three or four years ago, 734 00:36:49,390 --> 00:36:52,000 and now, I'm watching the lecture videos every year 735 00:36:52,000 --> 00:36:54,680 because I get really excited by the material. 736 00:36:54,680 --> 00:36:58,960 So even that, showing that they're still engaged in the class material years on, 737 00:36:58,960 --> 00:37:01,880 we consider it to be a substantial success. 738 00:37:01,880 --> 00:37:04,450 Another comment that we found particularly heartening 739 00:37:04,450 --> 00:37:07,220 was students feeling that they're now part of a community, 740 00:37:07,220 --> 00:37:10,920 whereas previously they felt like they were individual learners. 741 00:37:10,920 --> 00:37:14,460 We've also had some limited success on our YouTube channel 742 00:37:14,460 --> 00:37:16,950 where we've been publishing these for the world. 743 00:37:16,950 --> 00:37:21,690 So this was an example from last year where we had double the amount of views 744 00:37:21,690 --> 00:37:23,560 than we had students in the class. 745 00:37:23,560 --> 00:37:25,980 So that's showing that some people are picking this up. 746 00:37:25,980 --> 00:37:29,820 And that shows in our YouTube comments as well that are pretty positive. 747 00:37:29,820 --> 00:37:33,000 And it's great to try and share our educational innovations 748 00:37:33,000 --> 00:37:36,940 with the rest of the world, just like CS50 shares theirs. 749 00:37:36,940 --> 00:37:40,930 So a good way to assess whether what we're doing works 750 00:37:40,930 --> 00:37:44,080 is comparing my version of the subject against a different version 751 00:37:44,080 --> 00:37:47,480 that is taught in the alternate semester by a different professor. 752 00:37:47,480 --> 00:37:49,290 They have different teaching staff. 753 00:37:49,290 --> 00:37:52,930 They pre-record their lectures and do flipped classroom style and use a little 754 00:37:52,930 --> 00:37:53,450 more-- 755 00:37:53,450 --> 00:37:55,940 they lean more heavily on PowerPoint. 756 00:37:55,940 --> 00:37:57,930 But it's much cheaper for them to deliver. 757 00:37:57,930 --> 00:37:59,870 They don't have any of this production setup. 758 00:37:59,870 --> 00:38:02,630 They run the same slides and same videos year-on-year, 759 00:38:02,630 --> 00:38:05,340 but it scores very similarly on the survey. 760 00:38:05,340 --> 00:38:08,690 And so if it's scoring similarly on the survey, of course, I'd be asking myself, 761 00:38:08,690 --> 00:38:11,450 do I think this is actually worthwhile? 762 00:38:11,450 --> 00:38:13,610 Another thing that I might be worried about 763 00:38:13,610 --> 00:38:17,450 is that the better my videos become, the fewer students 764 00:38:17,450 --> 00:38:19,320 I might actually have in my class. 765 00:38:19,320 --> 00:38:22,790 But experimentally, empirically, I haven't really 766 00:38:22,790 --> 00:38:25,180 found that my attendance has changed. 767 00:38:25,180 --> 00:38:27,320 So let's think about student learning. 768 00:38:27,320 --> 00:38:30,650 Maybe this will help answer whether what I'm doing is worthwhile. 769 00:38:30,650 --> 00:38:32,750 So you remember the 2020 curriculum? 770 00:38:32,750 --> 00:38:35,770 It was six weeks of introduction to programming 771 00:38:35,770 --> 00:38:38,140 and then six weeks of basic algorithms. 772 00:38:38,140 --> 00:38:41,770 This is what it looks like for my version of the class in 2023. 773 00:38:41,770 --> 00:38:45,220 So now, we only spend three weeks on the C programming 774 00:38:45,220 --> 00:38:49,200 with talking about algorithms still during that process. 775 00:38:49,200 --> 00:38:52,740 Then I spend six weeks on algorithms and data structures, 776 00:38:52,740 --> 00:38:56,360 covering much of the same material as before. 777 00:38:56,360 --> 00:38:58,190 But that only takes us to week nine. 778 00:38:58,190 --> 00:39:00,210 And we've got 12 weeks in the semester. 779 00:39:00,210 --> 00:39:04,280 So then I can incorporate an additional two weeks of advanced computer science 780 00:39:04,280 --> 00:39:06,770 theory and present that in an exciting way, 781 00:39:06,770 --> 00:39:10,520 such that students get excited not just for the practical and engineering side, 782 00:39:10,520 --> 00:39:14,220 but also get excited for the theoretical components of computer science. 783 00:39:14,220 --> 00:39:16,280 And then we have one week for fun topics, which 784 00:39:16,280 --> 00:39:19,040 might be web development or AI or cybersecurity, 785 00:39:19,040 --> 00:39:23,390 just like CS50 includes extra fun topics throughout the course material 786 00:39:23,390 --> 00:39:24,930 to keep students engaged. 787 00:39:24,930 --> 00:39:28,550 And we've actually found that despite students now 788 00:39:28,550 --> 00:39:31,190 facing some more complicated material and indeed 789 00:39:31,190 --> 00:39:33,890 material that students often turn away from because it might 790 00:39:33,890 --> 00:39:38,390 be a bit more mathematically dense, satisfaction and grades have actually 791 00:39:38,390 --> 00:39:39,650 gone up. 792 00:39:39,650 --> 00:39:42,650 So we only assess nine weeks of content, which 793 00:39:42,650 --> 00:39:45,373 might be why students feel more comfortable with some 794 00:39:45,373 --> 00:39:47,540 of the more mathematical and more theoretical topics 795 00:39:47,540 --> 00:39:48,870 that we provide at the end. 796 00:39:48,870 --> 00:39:52,730 But we've still found that when we have conducted ad hoc assessments later 797 00:39:52,730 --> 00:39:55,460 that aren't part of their grade, that students have actually 798 00:39:55,460 --> 00:40:00,300 retained the material that we're teaching in the later weeks. 799 00:40:00,300 --> 00:40:03,420 So why do we teach this extra material and not 800 00:40:03,420 --> 00:40:05,290 assess it in a summative fashion? 801 00:40:05,290 --> 00:40:08,850 Part of the idea here is that we want to encourage students 802 00:40:08,850 --> 00:40:12,540 to be interested in computer science without the threat of them worrying 803 00:40:12,540 --> 00:40:14,970 about their grades at this particular point in time. 804 00:40:14,970 --> 00:40:18,480 So even by exposing them to material in an engaging way, 805 00:40:18,480 --> 00:40:21,060 I think we are achieving something as educators. 806 00:40:21,060 --> 00:40:22,810 And so that's something worth remembering. 807 00:40:22,810 --> 00:40:25,750 Not everything needs to be contributing to the final test. 808 00:40:25,750 --> 00:40:28,570 Something is just going to be interesting or extension material. 809 00:40:28,570 --> 00:40:32,050 And as educators, it's important not to lose sight of that. 810 00:40:32,050 --> 00:40:33,830 So we're almost at the end here. 811 00:40:33,830 --> 00:40:38,860 But think about the 80% in 5% of resources changes 812 00:40:38,860 --> 00:40:40,040 that you've already made. 813 00:40:40,040 --> 00:40:42,190 And the takeaway from this talk, the biggest one, 814 00:40:42,190 --> 00:40:45,790 is I want you to go home and think about what the changes that you could make 815 00:40:45,790 --> 00:40:50,980 are that would deliver similar larger outcomes with not so much input. 816 00:40:50,980 --> 00:40:53,500 So I'm going to give you-- aside from that biggest one, 817 00:40:53,500 --> 00:40:57,560 I'll give you a few smaller themes that are my takeaways at least. 818 00:40:57,560 --> 00:41:00,760 And the first is that you don't need an entire institution 819 00:41:00,760 --> 00:41:02,930 backing you to make very big changes. 820 00:41:02,930 --> 00:41:06,680 You can do a lot with only limited support for your definition of limited. 821 00:41:06,680 --> 00:41:12,040 Whether you are a single person teaching a small classroom in a town somewhere, 822 00:41:12,040 --> 00:41:15,110 or whether you are at a large university in higher education. 823 00:41:15,110 --> 00:41:19,240 No matter what the level of support is, there is always a lot you can do. 824 00:41:19,240 --> 00:41:22,120 And it's generally more than you'd expect and more 825 00:41:22,120 --> 00:41:24,280 than your colleagues might expect. 826 00:41:24,280 --> 00:41:27,800 It just requires you to be a little more creative. 827 00:41:27,800 --> 00:41:31,810 And the more time you spend putting into your creativity, the more return 828 00:41:31,810 --> 00:41:32,960 you'll get out of it. 829 00:41:32,960 --> 00:41:36,850 So make time to not just be doing the rote stuff of the class, 830 00:41:36,850 --> 00:41:44,970 to be making lecture slides or planning class or working on assignments. 831 00:41:44,970 --> 00:41:49,390 Spend some time explicitly dedicated to trying to come up with new ideas 832 00:41:49,390 --> 00:41:52,960 to try and think of ways to do things a little differently that step 833 00:41:52,960 --> 00:41:54,470 a little more outside the box. 834 00:41:54,470 --> 00:41:57,460 And these are often places that, if you make a change, 835 00:41:57,460 --> 00:42:01,480 will result in a larger outcome that makes students very excited. 836 00:42:01,480 --> 00:42:06,340 And then one other takeaway is that if the thing that you're trying to improve 837 00:42:06,340 --> 00:42:09,880 and the thing that you care about is end of semester survey scores, 838 00:42:09,880 --> 00:42:12,710 this is not going to be the best way of doing it. 839 00:42:12,710 --> 00:42:17,120 And so really, maybe we should think about our goals. 840 00:42:17,120 --> 00:42:19,880 It is not the best way to improve survey scores, 841 00:42:19,880 --> 00:42:23,640 but it does increase the joy in teaching and learning. 842 00:42:23,640 --> 00:42:25,880 And so let's throw away the surveys and think 843 00:42:25,880 --> 00:42:30,300 about what 80% changes we can make with just 5% of resources. 844 00:42:30,300 --> 00:42:31,260 Thank you very much. 845 00:42:31,260 --> 00:42:34,450 And it's a pleasure to be here with all of you. 846 00:42:34,450 --> 00:42:36,000