SHAANAN COHNEY: OK, hello, everyone. So because I've always wanted to do this, and I do it in my own classroom, lights, camera, and action. [MUSIC PLAYING] [BEGIN VIDEO PLAYBACK] [PHONE RINGING] - Hello? Oh my god. FOA is starting? OK, I need my puffer. Where's my puffer? - I think ChatGPT can be used effectively. [PHONE RINGING] - Oh my gosh. Sorry. FOA is starting now? Bye bye. Bye bye. [PHONE RINGING] - It's time for Foundations of Algorithms? I need to get my sonic screwdriver! - I'm really looking forward to my military coup of the university. - Me too. I can't wait to get rid of these hot desks. [PHONE RINGING] - Oh my god! FOA is starting! - What? FOA is starting? I should go get my parachute! - I've got to set my cryptowatch! [PHONE RINGING] - Wait, it's time FOA already? Let me get my cat. [PHONE RINGING] - FOA is starting? I've got to go meet the teaching fellows. [END PLAYBACK] SHAANAN COHNEY: So welcome, all of you, to today's class, which is Foundations of Algorithms with me, Dr. Shaanan Cohney, here at the University of Melbourne. So for the next few minutes, all of you are now my students. And if I look out over there into the classroom, I can see about 450 of you this semester. And were we in my other semester, there would be somewhere around 1,000 of you out there in the auditorium. All of you come from a variety of degree programs. In my university, in Australia, there are six degree programs that we're anticipating that all of you come from, from places as wide as music, commerce, and the business school to bio sciences and, of course, the faculty of science that many of you are coming to us from. We have 10 teaching fellows, also known as teaching assistants, or TAs in other countries, who are going to be helping out with our course. And then four first-year fellows, that I'll talk a little bit more about later, who help answer some of the questions that you'll give us throughout the semester. We have one production associate who is one of our very own students and is going to be helping out with the camera. Hello, Hrithik over there. I mean, he's back in Australia, so he probably can't see me all the way from Boston, but I'll give him a wave anyway. And then, of course, we have our two professors who are going to be delivering at least the lecture components of the class. There is Dr. Jianzhong Qi on my left at least, and of course, myself. So today, I'm going to tell you about the 80% outcome with 5% resources method, or what I call the 80 in 5 that I, Shaanan Cohney, have applied to my class, Foundations of Algorithms, that is quite similar to Harvard CS50. And how I tried to reverse engineer some of the components of CS50 and adapt them to my own university context. So a tiny bit about me. Now, we're back in the Boston, Cambridge, Harvard room, where I can now speak to you as stepping outside of our classroom and back into the educators' workshop. So I am responsible for the first-year computer science curriculum at the University of Melbourne. We have about 1,000 to 1,500 students who take the introductory course with us every year. And we have about 5,000 overall students in a university with about 40,000 students total. So a very significant fraction of the overall student body takes our class, just like the students who take Harvard CS50 in the college here. So the main theme of today's talk is going to be thinking about where the biggest room for improvement is in teaching. And while I've spent a lot of time reflecting on this, I want you as well to keep in mind today, where do you think you could make a small improvement that would be a large change in terms of the actual outcome? So is there a 5% change that you could make that would result in an 80% different outcome? And that's really today's talk. So as much as we're going to draw very heavily from CS50's curriculum structure, pedagogy, and design, at the end of the day, whatever context you're teaching in is going to be a little bit different. And so you're going to have to identify your own set of changes that you can make. Some of them will be a little bit smaller. Some of them will be a bit larger. But whatever your resources are, you've got to figure out the right things to do that will have the largest changes. So we already know a little bit about CS50 from all the discussion we've had so far. But what I teach is really a bit more like CS50 plus 1/2. So it's not the very, very intro class. They've taken a little bit of programming before in Python. And now, they're moving on to something slightly more difficult. And, of course, because I'm operating in a different university framework and particularly in higher education, I have to fit within the structure of the larger university and can't just teach CS50 directly without some longer term changes. So CS50, as we know, is a lot of resources, plenty of staff, administrative support, certain financial support. And also from a practical perspective, there's a lot that actually makes it work the way it's taught here at Harvard College. They also have a lot of TAs and just like me, a very large enrollment. However, unlike me, they also have to cater to all these very substantial external actors. And so taking that aside for a minute, catering just to my classroom or just to my university, it gives me a little bit more flexibility around what I do. I don't have to worry about my 1.4 million YouTube subscribers. So what does my class look like as opposed to what CS50 is like? What changes do I have to make? Well, firstly, we've only got two professors, and we're very, very part-time. So I spend half of one semester I can dedicate to this class. And it's only about 0.2 of a full-time position for each of us. This equates to 400 hours of work per semester, give or take, which is 36 hours of lectures, where we're actually standing in front of the class. And then we also do have a little bit of TA time, but not very much. So beyond the sections that we run every week, we are given two hours of TA time per student that's enrolled to make sure that we get all the grading done and all the additional help that that student might need. There are no other resources that the university provides. So everything that we have to do, we have to do on this smaller budget. And this is really important because almost none of us are here at CS50. I'm very lucky to be here today, but the rest of us are in our own scenarios where our resources are going to look different. And so whatever your resources are, you're going to have to think, OK, how do I spend my resources to get the best bang for buck? What are the things that CS50 does that are really effective, that you don't need a large budget for, that you don't need a giant staff for, that you can do in your own classroom? So here's where we take seriously one half of CS50's invitation. So CS50's invitation is to adopt, but it's also to adapt. This means that you don't have to necessarily take the whole course as is. You can take the pieces that work very well for your setting. These pieces might be the lectures, but they might be some of the pedagogy, some of the class policies. Maybe it's the way they handle extensions. Maybe it's some of the tooling. All of these different things are components that you could consider incorporating to your class, whether you're teaching CS50 itself or something a little different, like the class that I teach. So our theme here is, can we get 80% of what CS50 is doing in each of these different realms with only, let's say, 5% of the inputs-- 5% of the staff time, 5% of the finances, 5% of the practical support? So what makes CS50 work? What are the different pieces that we could potentially take? You've heard a lot, particularly in Guy's talk yesterday, he spoke a lot about what makes CS50 tick. And this is my version of the different pieces that I think you could potentially select from. And we'll go through these one by one and talk about the pieces that I was able to take for my class and how they were effective, even in a lower resourced environment. So we'll start off with curriculum, and we'll talk first about where my curriculum began when I joined the university adopting a class that already existed. And then where we've ended up today. So the CS50 curriculum looks something like this-- a week of very basic introduction to programming in Scratch, followed by some substantial work in C, talking a bit about algorithms in that component. We all remember the Tiedeman assignment. Then a week to get us ready to go to Python and then three weeks that really take the skills that students have learned earlier and putting them in a setting where students can build a real web app. So there are three weeks dedicated to learning SQL, Flask, and then the project that allows students to really take that foundational knowledge they've used before and demonstrate it in a very broad and flexible way. The Melbourne curriculum before I joined-- so I joined in 2021-- looked a little similar, but also a little different. So there was six weeks of Introduction to programming in C and there was a lot of discussion of algorithms. And then the final six weeks of the class in our 12-week semester was spent dedicated to various different types of data structures and algorithms, which is appropriate given that this wasn't their very first class. They'd already had some exposure to programming before. So the next six weeks spent a little time going through some basic data structures and basic algorithms. But CS50 seems to cover more in less time during the semester than we were covering in a full 12 weeks. So in just 10 weeks, CS50 is cramming in a lot of content. It covers a lot of the algorithm's content that we did, perhaps not quite as far, but then it also managed to get all the way through teaching Python, substantial amounts of web development, and all that in really just 10 weeks of lectures. So how is that done? Well, you could say all we need to do is we should take exactly the things that CS50 teaches and then put them in our classroom, and everything will instantly be fixed. Unfortunately, the reason that CS50 works is not just because of the things they've chosen to teach. And so it's a mistake to adopt someone else's content whole scale without understanding how it's delivered and why it works in their context. So you also have to think about the other components of CS50 that make teaching this class in 10 weeks a viable proposition. So this includes the pedagogy, the way they think about how they're teaching and how things are structured, the quality of the delivery, and the excitement of the delivery that engages students and gets them to put in that necessary effort, as well as the assessment components, which is really where students do a lot of their self-driven learning. And ultimately, where they're going to pick up the skills that they see demonstrated in lecture. So if we don't have students putting in the same amount of effort and time, we're probably not going to be able to cover the same content in 10 weeks because students will fall behind. So now that we've taken at least a first look at curriculum, and we'll come back a bit later to where we landed, we should talk about some of these other ingredients that will enable us to make this curricular shift. So one thing that students often say about CS50, and something that I think I've managed to accomplish in my classroom, is to make the class feel a bit like a movie. So we've all watched the CS50 lectures. We see how it's presented in super high production quality that's filmed in 4K, ultra high definition, high dynamic range, all these little buzzwords that end up giving you a good picture and good sound quality. And particularly with the sound, students notice when what they're seeing in front of them, if it looks good or looks bad. Now, many of my students end up watching my lecture videos instead of actually coming to class, which is a common phenomenon in many parts of the world, as students are increasingly using the flexibility of being able to pause lectures, slow down, speed them up. So this is what CS50 looks like. And this is what Foundations of Algorithms looked like my very first year when I joined and was using the built-in projector system and lecture capture in the room. And depending on what kind of institution you're looking at, you may have this, or you may have nothing whatsoever. So you can't really see what's going on there. So I didn't think my students would rather learn from that. But within about a year, I was able to get something that looked much clearer and much better. And depending on how much you like playing with cameras and toys, you can go on a wide spectrum from I'm going to have a nice webcam and I'm going to shine a light appropriately on my face so that students can see me all the way to a multi-camera production. And so because I'm one of those more techie people, I thought, you know, let's figure out how we can do this slightly more sophisticated recording setup on a much smaller budget. Now, again, what I'm about to talk about I did with not a tiny budget. I spent a little bit of money and built a medium budget. But you can do this on your own sliding scale for what's appropriate for you. You don't need to go all the way to CS50 the way Harvard does it. You don't even need to go all the way to what I do it. Again, the message is, what are the small changes you can make that will have the big outcome? So our goal here in this component was to make the lectures memorable, accessible, and high quality by using video and audio production techniques to take another step up from what they were getting before. And this is something that CS50 has thought very heavily and very carefully about, including in this one paper that goes beyond just the fancy cameras to talk about elements of theatricality that get brought into the classroom. How to take techniques from drama, production design, staging, that sort of thing, to make students feel like they're more immersed in what's going on and not just sitting watching someone stand at a lectern, as you might be doing for the next half hour, giving a bit of a talk. So to actually bring them into the world of the lecture. So one way that CS50 does this, and that we do this fairly affordably, is to use props. And these props don't have to be super fancy stuff made by a special prop designer. They can be small. You can see in this photo, potentially, one of the students is holding up a number two. And I'm teaching sorting using house numbers that I bought from a local hardware store for about $5 each. Now, while, of course, you could go bigger and better, even this small change, students already felt like they were more immersed than just drawing the numbers on the whiteboard. On the right here, you can see the big blue brain. And this was part of an art exhibit that was being thrown away after the art exhibit was being done. And I thought, you know, I could bring this costume into my classroom. And our students get excited whenever we bring out the big blue brain because it means that a student volunteer is going to be selected to come up to the front of the classroom and do some exciting demonstration. So the big blue brain itself doesn't have any inherent educational value. But by incorporating some elements of design and creativity into the classroom, you can get students excited about the content itself. And eventually, these theatrical elements become representative of substantive course elements. So by creating that link between the fun stuff and the substantive stuff, you can begin to create links in the students' minds that actually enable them to learn the material better and to get excited about the material. So this is what it actually looks like up close from my little control panel. Now, I don't have as good a way of hiding what's going on, but this actually makes it handy to explain how I'm running my class. So you can see the same kind of thing that's happening in CS50, just with the cables hidden away a little better. So essentially, what's going on-- this is a very schematic stick figure diagram of it-- is I have two laptops, one of which is controlling the code and one of which is controlling the slides. And then sometimes I'll plug an iPad in. And then I have a cheap little device-- you can get these for as little as a few hundred dollars-- that enables me to switch what's on the screen at any given time. And just by pressing a button, then I can quickly switch between the code and the presentation material. And this ends up actually saving a lot of class time. I calculated it saves me probably six minutes per lecture of futzing with things and switching between things. And this is an extraordinary amount in a lecture that's only 55 minutes long. So by investing a little bit in technology, not only does it make the presentation look cleaner because students see these fancy transitions between the slides and the code, but it actually saves time and allows me to focus on the teaching and a little less on the technology, even while requiring a little more thought to begin with. But you don't necessarily have to do that. But if you do want to, it does cost a little bit at the end of the day. So this was my personal investment into the class with a mix of camera equipment. Now, again, what I really, really want to emphasize here is you don't have to spend the $14,000. You don't have to spend $140,000. You could do this with $140. And just the types of things that you buy, depending on your situation, will be a little different. And so just like me, you might soon, too, be lugging a suitcase full of items from your office to where you actually end up teaching, whether that's props, whether that's cameras. Bringing extra stuff into the classroom can help enhance the experience. As well after the lecture, one thing that I do that I find very effective as an educator is actually going back and watching a recording of what I've done. Now, I do spend a lot of time editing what I've done and finding the bits that worked. And one of the surprising things is I find sometimes saying less actually results in students learning more. So sometimes I'll have an explanation that missed the mark. And if I cut that out and make a note to myself the next time-- next time, don't give this explanation-- sometimes that's just as good as adding an explanation. So while the editing process is useful for producing this very polished finished product, just like happens at CS50, the more important part of that process is not actually the video editing. It's watching back myself and then making changes based on what I see. And this is, again, as someone says in the comments, less is more. So figuring out what are the important things that are actually getting things across. I also see a question. What software do you use for video editing? I use DaVinci Resolve, and I believe that's also what the team here uses. So there also are time costs that I pay to make this happen. And this is some of that extra 5% investment, might be a little more than 5%. Oops, there we go. So I do invest substantial extra time in my classroom getting set up for it every time. Right now, this looks like an hour of additional set up and tear-down time. There's me setting up all the cables myself and my student production associate. And this results in about an extra $3,500 dollars of staffing costs, which might be relevant if you're at a university budget where you have to think very carefully about your TA time. And on top of that, about two hours of my time for every hour of class time in doing post-production, so doing this video editing. Now, one thing that I should note is while this is my current version, when I started doing this, I did not have an extra hour of setup and tear-down time. I had the traditional five minutes in between classes. The hour that I have now is just so I can do it in a relaxed and less frazzled way. And I've already lost more hair than I would like doing it the five-minute way. So giving myself an hour is now my preferred method. And then, of course, there's the fun aspect of creating catchy-- these are YouTube thumbnails that I've used for some of my slides. But more of the point here is creating a fun theme for every class that kind of hooks students in. And there are different ways of doing this, whether this is a demo at the start of the class or a fun thumbnail that you post to students to get them hyped before the class begins. This is another method of drawing students in. [BEGIN VIDEO PLAYBACK] [MUSIC PLAYING] - This is the magical star of dereferencing. The power of the magical star of dereferencing is to open Bruce's mailbox. So open Bruce's mailbox, take out what's inside. [MUSIC PLAYING] OK, what do you see there? - 0x50. - The other thing that happens when you use the star of dereferencing is you now go to the location on the paper. [MUSIC PLAYING] So 0x50 matches 0x50. And we read out what you find there. So what do you find? [MUSIC PLAYING] [END PLAYBACK] SHAANAN COHNEY: So again, this is just another example of the type of content that you can actually put together pretty quickly if you are recording your lectures and then get students very excited. So I'd say that was about another 20 minutes of effort after the fact. And students then started sharing that around with friends and encouraging more people to come to the class, which is, again, beneficial to the environment overall. So while I've spoken a fair bit about production and technology, I've said, you know what if you buy this other camera, what if you film your lectures. These are all cool, and they're definitely added bonuses. But I would say that they're definitely not enough on their own. You can have the world's most beautiful looking lectures, but if you don't have the other ingredients, you haven't actually done anything other than create a very slick looking YouTube video. So here is an example of actually taking some of these elements and combining them for pedagogical value. Now, CS50 has done a number of similar exercises in the past. I'm going to let it run for one minute so you can see what kinds of things you can do in the classroom with relatively simple props that actually use the props as a way to hook students into the material you're teaching, and really deliver pedagogical value. [BEGIN VIDEO PLAYBACK] - So first card and then the second card. The first card will close, then the third card. And then the-- - Second card closes. - Fourth card. And then they will switch. And everything-- - Over here. - OK, then the first card. - OK, you guys are getting the hang of it. Let's speed it up slightly. We're almost through. - Third card, fourth card, fifth card, sixth card. That's right. - Yep, looks good to me so far. - Card, and eighth card. [LAUGHTER] All right. OK. First, second, third. They switch second and third. And then first and second switch. Yeah, that's sorted. I think we need to go through and verify. So one, two, three, four, five, six, seven. - OK, everyone turn around. - And that's right. - Can we have a big round of applause for Disan? [APPLAUSE] [END PLAYBACK] SHAANAN COHNEY: So while that is a simple sorting exercise, bringing in these giant human-sized playing cards suddenly made the experience a lot more tangible, a lot more exciting, and a lot more memorable through this technique of theatricality that CS50 uses that we could bring into our own classroom, despite not actually being CS50 itself. And you can do this kind of thing, whether it's with sheets of paper, like David talked about yesterday, with giant playing cards, house numbers that you buy from the hardware store, or whatever it is you use. The techniques of theatricality can be brought into your own classroom with only a little bit of extra creativity and a little bit of extra effort. So one thing that I do that's different from CS50 is I do use a lot of cold calling in my classroom, trying to remember students' names. And I find that for my classroom, and at least with the number of students that actually show up, this is a very helpful approach for actually getting student involvement and bringing students to the front of the class. So this is one of my ingredients that's very highly debated and educators have very different approaches to this because students will sometimes have a negative reaction. But I've found it's valuable in my classroom at least. So how do you actually do all of this? How do you learn how to do all of this? How did I learn how to do all of this? Well, there's one surprisingly simple answer to it. I spent a lot of time on YouTube University. So this is whether it's looking at CS50's material or looking at other educators or looking at content creators on the internet who are talking about how to set up a camera so you don't get glare in your glasses. There are so many free resources out there for teachers now. And I would encourage you to actually spend some of your time as professional development going out there and actually looking at what other people do and then taking the best elements that are going to fit your classroom and bringing them in. I've also found that the listserv for teachers, the email listserv, has been an incredible source of support. And CS50 staff have been very helpful if you do start running against challenges with adopting some of their tools or their pedagogy and have been very helpful and very responsive. So that covers the pedagogy and the engaging delivery and active learning components. We spoke a little bit about active learning yesterday in Guy's section. But there's also some of the tooling that CS50 uses. And this can be adopted and adapted as well. So CS50 has their website. And they have, again, much like many of the other things for their class, made the source code available. And so I was able to adapt that and produce a similar looking website for my own class building off their code, which now enables my students to have the same kind of positive experience accessing the material beyond the learning management system built into the university. Now, again, your mileage may vary depending on your institution's rules, but I found that adopting some of the code that CS50 uses to run their infrastructure very, very helpful. One other thing that's different from what CS50 has done that my group has done is tried to do some more automated style checking, where CS50 is traditionally relied on humans, even though the AI grading is coming more and more into a part of their infrastructure. So this was a paper that my team presented at SIGCSE, which is a computer science education conference in the last year. And we built a tool that aims to provide students helpful error messages to understand not just how to fix their bad style in code, but also what might be a pedagogical learning from it. And so while the comments that I've shown here are reduced in size, if you were to click on any of these, you would see an expanded explanation explaining to the student what this element of style actually means. So again, this is an indication of where tooling can actually make you more efficient as a teacher, giving you more time for the things that matter. So a 5% investment in technology in this case allows us to spend more of our TA time on the one-on-one student interaction and the things that are harder for computers to deliver, even while AI tools are advancing and actually allowing us to do more and more of the substantive hands-on stuff and replacing more and more elements of what TAs would have used to do. So, again, with this, the feedback that we got from TAs was universally saying, I now don't have to spend as much time on the menial tasks and the things that don't need me, and I can spend more time on the things that are going to make the real changes to the students. Helping them through whatever their substantial struggles are. Helping them cover gaps in understanding and sometimes even being a helpful ear to them when there are things going on outside the classroom that are impacting their learning. So another thing that you have just heard about that I've managed to integrate into my own class and I want to give a positive report of that is using CS50's AI tools to help answer questions on our class discussion board. Now, you might be surprised, you went through a session before, and you spent a little-- it took about an hour of building code. And you might not feel super confident that this is something that you could roll out tomorrow. But one of my experiences was CS50 gave this talk, a workshop at SIGCSE earlier in this year as well, and I was very excited. And within eight hours of hearing the talk, I was able to roll it out into my own class. And so this was a very powerful-- again, adopting what CS50 did, but using slightly different configuration on the back-end so that I was able to roll it out with a little less effort and a little less time than they necessarily did. So this was again, one of these very effective 5% effort, but very substantial outcome very quickly. And we found our outcomes were quite similar to CS50 adopting this tool. This is also true of the programming environment, kind of CS50 Codespaces, that has been incredibly effective. So we've experimented with Codespaces. Ultimately, in our environment, we decided to go with edstem, which is a tool that CS50 has used in certain contexts. One thing that I'm actually a little worried about as an educator being direct is I don't like having too many of my tools reliant on infrastructure that I don't control. The reason is, no matter how good a company is, I'm always a little bit worried that one day that something might happen to that company, and they won't be able to support me anymore. So that's one thing that I very much appreciate about CS50's tooling is that even when they use something, like Github, which is incredibly helpful and has been a great partner, they also retain a lot of control over the infrastructure and are able to deploy it somewhere else should they need to. So edstem has been very useful for me, but I am looking towards adopting some of CS50's cloud tools. So beyond the technology and infrastructure, another thing that makes CS50 very, very powerful for students is this sense of community that's generated through many of their activities. [BEGIN VIDEO PLAYBACK] [MUSIC PLAYING] - Hi, everyone. Welcome to the University of Melbourne, and welcome to the School of Computing and Information Systems. I'm Jud from CISA. And today, I'll be taking you on a tour to the First-year Learning Center located in Melbourne Connect. Every day, there'll be tutoring for Foundations of Computing and Foundations of Algorithms, and snacks will be provided. Come along. [MUSIC PLAYING] So this is Melbourne Connect, where the First-year Learning Center is located. So the Student Learning Center is located on level three. - Car G. [MUSIC PLAYING] - It's located just down this corridor here. [MUSIC PLAYING] And it's just here. Welcome to the First-year Learning Center. [MUSIC PLAYING] [END PLAYBACK] [MUSIC PLAYING] SHAANAN COHNEY: So one thing I wanted to communicate by showing you that is if you actually look at the physical space in that video, there's nothing inherently exciting about it. It's a room in an office building. But the idea of marking it out as a unique space for students and trying to create that sense of community through things like having snacks there on a regular basis. Through having our office hours in a consistent location has over the year, or the year and a half since I've been running it, created an enormous vibe of connectedness that previously was not part of our major altogether. Now, it's been incredibly useful from an educational perspective because we now have a drop-in center where students can get one-on-one support. But a very substantial outcome from it has been this notion of students feeling more tied to the place that they're learning, which, again, actually ends up improving their learning outcomes. Another way that we work to build community is by partnering with CS50 on some of the events that we don't necessarily have the resources to plan ourselves. One of these that's been a big success for us has been CS50 Puzzle Day, where the CS50 team puts together a sheet of puzzles that aren't coding problems. They're just logic puzzles and games that students compete in teams together to finish during a set time period. And having these kind of extracurricular activities helps to build community, which then plays back into the classroom itself and makes the internal classroom work a lot better, despite the fact that these things don't inherently have a connection to the material. So our outcomes for developing this center over the last year and a half are pretty incredible. So we've managed to get eight hours of student clubs coming in to help out other students, along with the 20 hours of formal support that I was able to arrange. And within just one year, we had more than 4,000 individual student visits to the space and 1,000 one-on-one support sessions. So these numbers are pretty substantial for our class, which means that each student was, on average, accessing this space at least a few times during the course of the year. So the last thing that I haven't quite solved, and I would love to say, you know, I have managed to get all the way to CS50, but the truth is all of us are striving. Even CS50 is making changes year-on-year. And one area where I still think there's a lot of work for my particular class is assessment. I am constrained a lot by the particular rules of my university. There is a nine-month long approval process for any change in assessment structure, which definitely slows down innovation. However, I have had success in moving from large exams to more CS50 style weekly assignments and weekly problem sets that we found really does ensure that students get started a little earlier. And that is generally considered to be good educational practice. So we often talk about, in the world of education, the difference between formative and summative assessment, where summative assessment gets students to show what they've learned at the end. Whereas formative assessment is about building their skills earlier on in semester. So by reducing the stakes and the amount of value that students place on the summative assessment and shifting some of that to the actual learning process, we found that students are feeling much more satisfied and indeed, their scores are increasing. So is this all worth it in the end? There have been a lot of changes that I've made. I've put in a lot of time. Even if it's just 5% of what CS50 is doing, we still have to ask, is it worth it? So are we getting this 80% of the outcome on 5% of the resources? So just like any other teacher, I've got the survey, end of semester survey things. So the results did go up very markedly. But as we know, surveys are not really the best way of testing whether students learn. If you make your assignments easier, your assessment, your end of semester scores go up no matter how good a teacher you were. So this only tells us so much. Most of the best outcomes that we find we're getting are actually outside of the traditional measurement instruments of surveys and grades. We've had a lot of very positive student feedback, including long thank you letters coming back years, students saying, you know, I took your course once three or four years ago, and now, I'm watching the lecture videos every year because I get really excited by the material. So even that, showing that they're still engaged in the class material years on, we consider it to be a substantial success. Another comment that we found particularly heartening was students feeling that they're now part of a community, whereas previously they felt like they were individual learners. We've also had some limited success on our YouTube channel where we've been publishing these for the world. So this was an example from last year where we had double the amount of views than we had students in the class. So that's showing that some people are picking this up. And that shows in our YouTube comments as well that are pretty positive. And it's great to try and share our educational innovations with the rest of the world, just like CS50 shares theirs. So a good way to assess whether what we're doing works is comparing my version of the subject against a different version that is taught in the alternate semester by a different professor. They have different teaching staff. They pre-record their lectures and do flipped classroom style and use a little more-- they lean more heavily on PowerPoint. But it's much cheaper for them to deliver. They don't have any of this production setup. They run the same slides and same videos year-on-year, but it scores very similarly on the survey. And so if it's scoring similarly on the survey, of course, I'd be asking myself, do I think this is actually worthwhile? Another thing that I might be worried about is that the better my videos become, the fewer students I might actually have in my class. But experimentally, empirically, I haven't really found that my attendance has changed. So let's think about student learning. Maybe this will help answer whether what I'm doing is worthwhile. So you remember the 2020 curriculum? It was six weeks of introduction to programming and then six weeks of basic algorithms. This is what it looks like for my version of the class in 2023. So now, we only spend three weeks on the C programming with talking about algorithms still during that process. Then I spend six weeks on algorithms and data structures, covering much of the same material as before. But that only takes us to week nine. And we've got 12 weeks in the semester. So then I can incorporate an additional two weeks of advanced computer science theory and present that in an exciting way, such that students get excited not just for the practical and engineering side, but also get excited for the theoretical components of computer science. And then we have one week for fun topics, which might be web development or AI or cybersecurity, just like CS50 includes extra fun topics throughout the course material to keep students engaged. And we've actually found that despite students now facing some more complicated material and indeed material that students often turn away from because it might be a bit more mathematically dense, satisfaction and grades have actually gone up. So we only assess nine weeks of content, which might be why students feel more comfortable with some of the more mathematical and more theoretical topics that we provide at the end. But we've still found that when we have conducted ad hoc assessments later that aren't part of their grade, that students have actually retained the material that we're teaching in the later weeks. So why do we teach this extra material and not assess it in a summative fashion? Part of the idea here is that we want to encourage students to be interested in computer science without the threat of them worrying about their grades at this particular point in time. So even by exposing them to material in an engaging way, I think we are achieving something as educators. And so that's something worth remembering. Not everything needs to be contributing to the final test. Something is just going to be interesting or extension material. And as educators, it's important not to lose sight of that. So we're almost at the end here. But think about the 80% in 5% of resources changes that you've already made. And the takeaway from this talk, the biggest one, is I want you to go home and think about what the changes that you could make are that would deliver similar larger outcomes with not so much input. So I'm going to give you-- aside from that biggest one, I'll give you a few smaller themes that are my takeaways at least. And the first is that you don't need an entire institution backing you to make very big changes. You can do a lot with only limited support for your definition of limited. Whether you are a single person teaching a small classroom in a town somewhere, or whether you are at a large university in higher education. No matter what the level of support is, there is always a lot you can do. And it's generally more than you'd expect and more than your colleagues might expect. It just requires you to be a little more creative. And the more time you spend putting into your creativity, the more return you'll get out of it. So make time to not just be doing the rote stuff of the class, to be making lecture slides or planning class or working on assignments. Spend some time explicitly dedicated to trying to come up with new ideas to try and think of ways to do things a little differently that step a little more outside the box. And these are often places that, if you make a change, will result in a larger outcome that makes students very excited. And then one other takeaway is that if the thing that you're trying to improve and the thing that you care about is end of semester survey scores, this is not going to be the best way of doing it. And so really, maybe we should think about our goals. It is not the best way to improve survey scores, but it does increase the joy in teaching and learning. And so let's throw away the surveys and think about what 80% changes we can make with just 5% of resources. Thank you very much. And it's a pleasure to be here with all of you.