[MUSIC PLAYING] BRIAN YU: Hi everyone. Welcome back to Web Programming with Python and JavaScript. Today we're joined by some special guests from GitHub and Travis, which you'll recall we've been using in order to do version control and CI and CD. And they'll be here today to continue those conversations about what we can use version control and CI and CD for. And so first up, we have John from GitHub who has joined us today to talk to us about how we can use GitHub in order to work on our web applications. JOHN BRITTON: I'm looking forward to spending some time with you, building some cool features. So just an overview of what we're going to do today, I'm going to walk through the development flow that we use at GitHub to work on web applications. So we're primarily a software as a service company. We build web apps. So I'm going to walk you through what a typical day would look like for me in working on some Ruby on Rails code for GitHub Classroom. So GitHub Classroom, as you probably know, is an application for teachers and for students to use GitHub for learning how to code. And I think you're using it in CS50 already, or in this course. So yeah, let's get started. The first thing that I'm going to do is just hop into GitHub and go to the repository. This is the GitHub Classroom repository. So it's an open source code, Ruby on Rails app. You can see there's different stuff going on here. You can watch it. There's a bunch of stars. All the code is down here. And then you can check out the Read Me. If you're interested in developing, getting involved in open source, I chose this project because it is an open source project. I'm a maintainer on this project. And if you're interested, you can send a pull request, open issues, and I'll be able interact with you. Also, it's great because it's open source, I can easily use it as an example here. This code powers a website, GitHub Classroom, which, if we log out here, is just a tool for creating programming exercises and distributing them to students, or people in your course to do coding exercises on GitHub. So what we're going to do is we're going to walk through the process of developing a feature using Git, making a branch, running our unit tests, sharing with our peers, getting feedback via a pull request, deploying to production, and also doing something that we call feature flipping, which is an approach to bringing out features to a live production application and slowly rolling them out over time or to specific people. So the first thing that we need to do-- and you can kind of follow along with this. I'm probably going to move too fast for you to keep up, but I want this to be kind of a tutorial like what you could do on your own time. So the first thing to do here is in the top right-hand corner of the repository there's a Fork button. And if you go and click that, it will let you create a copy of the repository in your personal account. I've already done that, so if I go to my browser and just type in my username here, you'll see that I have a fork. It says forked from education classroom. So this is the public open source main repository. And I have a copy of this repository in my personal GitHub account. Then what you would do from there is go to this Clone button right here and click on that, and just grab the URL and hop over to your terminal. In your terminal, you would do git clone and paste in that URL. So now you can get a copy of the repository on your local machine with all the code for the app so you can run it locally and get started. I've already cloned it, just for the speed of this demo. So if I move into my source repository location, we can go ahead and run git status. And you can see that I have the master branch, everything is working. And if you want to look at the files, there's all these files and gems and cool stuff in here. We follow a pattern on our repositories of scripting everything. So we want it to be, for a new contributor coming to our project, really easy to set up. So within the project, there's actually this file in the Scripts directory. So in here, we've got script. And inside there, we have a file called bootstrap. You run that and it will just set up your entire machine. The only prerequisite for this app is to have Docker installed. So you install Docker on your machine, you run script bootstrap, and you should be off to the races to get this running locally. In my case, I'm just going to go ahead and run script server. So after you've set everything up, you launch the server. You can see Docker is booting up the different services and different containers. And then if things go well, we'll have a locally running version of the application. Takes just a moment. So there we go. It says it's done. It's starting the Docker services. Connection succeeded. And if we switch over to our browser-- I'll put this over here, put the browser over here-- we can do HTTP localhost 5000. Once that finishes loading up. You can see the different services running and putting out their output. And then the Rail server is running. Now it's loading. Great. So we have a locally running copy of the application. Doesn't look like much right here. It's just the landing page. So we go ahead and sign in and just see how it works. You can see a lot of stuff happening in the background in my console, but basically what's happening is it's connecting to my GitHub account, and it's logging me in as my user on GitHub on this application. And it shows-- this is kind of what the teacher sees when they're using GitHub Classroom. It's kind of overview of all their classrooms. It shows them what students they have, what assignments they've created, who submitted what when. And we're just going to add a really simple feature. We're going to go ahead and add a link to Harvard CS50 in the navigation. So we're not going to do any actual programming. We're just going to add a little bit of HTML as a sample of how this process works. Now, once we dive into the code, we're going to use a feature branch development strategy. So at GitHub, and commonly on teams that work on software as a service web applications, it's very common to take your project, make a branch, do a little bit of development off to the side just on one specific feature, and then merge it back into master, where the master branch, the main line of development for your project, represents production. That's what we do on GitHub Classroom. The master branch is what's in production. We do a single feature branch, and we go off to the side and we do some work, and then we merge it back in and we deploy it right away. So this is the idea of continuous deployment. We can deploy GitHub.com hundreds of times a day. We use the same flow on GitHub.com, but I use this as the example because it's open source. A different way of doing this is kind of release-based development. And I think this is really common in desktop applications or any kind of software that goes in a package and is shipped out. Something you might see, like version 1.0, version 2.0, version 3.0 of the software. And generally what happens in those cases is you have a main line of development. You branch off for a new release. And you'll say, this is the 1.0 branch. And your developers will be just putting lots of stuff into the 1.0 branch. And eventually, that 1.0 branch might get merged back into master. But in the meantime, there could be a 1.1 branch started so that you can simultaneously be working on the upcoming release, and like maybe one release down the road for six months later. But in our case, I'm going to focus just on this feature branch development mode where we just make one branch for each feature or one branch for each bug report, and we merge that in and just kind of keep going. So I'm going to open up another terminal, and you'll see that I have my repository clone down locally. On the right-hand side of my terminal, that just says what branch I'm on and the current status of my repository if I ran git status. So every time I do something, that will automatically update. And the first step here is to get onto another branch. We need to make a branch to work on the side. So we'll use the command git checkout -b. git checkout allows you to select a branch and make it active, and the -b flag lets you create the branch at the same time. So I'll just call this Harvard. I'll make the branch. You'll notice on the right-hand side, it's updated. I'm on the Harvard branch. The next thing that I'm going to do here is open up my text editor. So I'm using Atom, which is GitHub's text editor. And I'm going to open up a file called header.html. So this is on every page of-- let's see here. On every page of the site, you'll see we have this header. And it has like a GitHub Education link, a link to our forums, a link to some video tutorials, different stuff up here. And all of this is contained in this file. So I'm going to make a change that. What I'm going to do is just copy this GitHub Education link, and I'm going to just paste another copy of it and save it and refresh. And so now you see I have the link twice. We're editing the local development environment. I'm going to go ahead and call this Harvard, and then I'm going to link to Harvard.edu. Now obviously, this isn't a features that we really want in the app, but you can imagine that I'm making some new feature, like the ability to grade assignments, or the ability to invite people, or different kinds of things. The important part here is that we're doing some kind of development on the app in its branch on the side, and that we can discuss it and share it with people on GitHub. So I go ahead and save that. And the next thing that I'm going to do is go back to my terminal and do git status. git status, I know you've covered some of Git in this course already, but git status, just as a refresher, tells you what's going on in your working directory and in your staging area, and what kind of the next thing you should do. So it gives you some pointers. It says, right here, use git add file to update what will be committed. So what we need to do here is we need to use git add, and then app views shared header partial dot html dot erb. So what this is saying is I want to put this file that I've modified into my staging area and prepare it to be committed as a new commit in my repository on the Harvard branch. So I do that. git status reports that it's modified. And then the last step here is to do git commit. When you commit, you include a message. And I'll say, add a link to Harvard. I should probably check and make sure that the link actually works. If I click it, it does, in fact, take us to Harvard. All right, so now we've made our first commit on this new feature development that we're doing. And I want to show you a little bit about how to switch back and forth. So the first thing we did, we made a branch. We moved over to that branch. We made it active. We can, at the same time, use the command git checkout and specify a branch master to go back to our main line of development. This is the thing that's currently in production that everybody's using. I do that and I refresh this page, and you'll see that the link to Harvard disappeared. It's not on this branch. If I go git checkout Harvard and I refresh, it's now on this branch. And this is pretty accurate as to how I would be working normally. I would make a branch, do some stuff, run in development, see how it's going, and just keep checking things as I go until I get to the point where I'm like, yes, this is the thing I want. I want to push it out to production. It works. Now I need to get code review and run some unit tests to make sure that what I'm doing is right. So again, we started by forking the repository and we had it in our own account. And so what I'm going to do is I'm going to push it to my copy. I'm not going to push it to the main GitHub Education copy. I'm going to push it to my copy. So what I'll do is say git push johndbritton harvard. And in this case, johndbritton is the name of my remote that represents the repository, my fork of the repository. So it's here. It's this one. So that's what that means. So I'm going to push to that. And I'm going to push to the branch called Harvard. So this branch, I'm going to push it to a branch with the same name remotely. So you can see it pushes the code up to GitHub on my repository. And if I refresh the page on GitHub, you'll see that-- let's see here. If I go to the homepage. There you go. This yellow box says that less than a minute ago, I pushed a branch called Harvard to GitHub on my personal GitHub repository. And I can click this button here, Compare and Pull Request. So I'll do that. What a pull request does is it lets you start a conversation about some code, about some changes that you made. So in this case, I'm just going to say I want to add a link to Harvard. And up at the top, because this is a fork, you'll see it saying, I want to merge from my version of the repository on the Harvard branch to GitHub Education, the official version, on the master branch. I'm going to change this just to be my own version. So I'm going to merge my Harvard branch into my own version's master branch, just because I don't actually want to deploy this to our production instance. So I'm going to add a link to Harvard, like that, and just say, this is a simple example of a feature branch development for the Harvard course. And I can just say, create pull request. What happens is it automatically starts a build. So it goes off to Travis and it says, John made this branch. He made some new code. I want you to run all the tests. So what does that mean? Let's look at our test. While this is kind of doing some stuff in the background, I'm going to hop over to my editor. And if we go here and go to spec. So in here, we have-- let's zoom in. Within our repository of this folder-- So it's a Rails application. We're using RSpec. All of our specifications, all of our unit tests are within this one folder called spec. We have all these different kinds of specs. We have controller specs. We have helpers. We have background job specs. We have model specs, service specs. And each one of these things-- so if I say, for example, I'll go into my model specs. And I'll go in here and I'll check like the deadline specification. What this does is makes sure that when a teacher sets a deadline that we close down assignment submission on the deadline. And if somebody tries to submit, it gives an error. So there's all these different checks. In our case, our change was very minor. We just added a visual change. But we could have accidentally introduced some kind of bug, some kind of error. And so by running the tests, we're able to guarantee that nothing has changed, nothing has broken. Nothing important has broken anyways, because we have tests for everything that's important. So there's hundreds and hundreds of these tests. They're all running. And if I go in here and click on the Travis CI link, and I go to our build, it's building on different versions of Ruby. So I'll open up this one. And you can kind of see it's setting all of our environment variables, it's installing all kinds of-- all of our different-- oops. There we go. Yeah, what it actually says here-- we're not going to go through every one of these tests, obviously. But it's running all the tests. And then you can see as they run that they're passing. It's kind of live updating. So this takes a couple of minutes. Once that's done, I'll get a green status update on my repository on the pull request saying that, yes, in fact, this did pass all the tests, so you should feel very confident about deploying this to production. In the meantime, what would commonly happen is my coworkers would come in, or other people on the open source project would come in and review the changes. In our case, the change is rather trivial, so I'll just come in here and just say, this looks great. And I will zoom in on that. This looks great, let's ship it. Add a comment. And so we can collaborate on that. What would be pretty common to see here is say you did something that might be a security concern. Somebody from your security team could come in and give comments about what you need to check, make sure that you're following best practices, kind of give you a second set of eyes so that you end up in a position where you're writing the best code you can. It's going to take a couple of minutes for this to run, so I want to jump to-- jump back to the code. So we're going to cover a few things. We did branches. We made a pull request. We're doing automated build. We got some review. And then we're going to talk about deploying. So every setup is different, how you're going to deploy your project. In our case, we use our chat tool. So we have a shared chat, kind of chat room thing. And every time we make a change, those changes get propagated to our chat tool as kind of a notification. And then we can run a command in our chat client that says-- we have a bot. His name's Hubot. And we tell Hubot, Hubot, go deploy GitHub Classroom to production. And we can even specify, go deploy this branch of GitHub Classroom to production. So I could say, for example-- where is my-- here we go. So for example, in here I could say, for example, deploy classroom/harvard to production. I'm not going to run this right now, but essentially, if I wanted to deploy this out to production, that's exactly what I would do. And as you can see from before, I deployed classroom, the master branch. Hubot replied to me that I was deploying. I got a success option. And then it said it was finished. Let's check in on Travis. OK, our build-- wait, this is not the right one. We had one build fail and one build pass. I don't know why we failed. This one will restart. All right, so anyway, we go back to-- where did our thing go? Oh, I have to press the back button. I lost my tab. So if we go back here, anyway, this pull request build passed, so let's check that one out. So ultimately, we ran all of the tests. We've seen them all passing here. So I feel pretty confident that the change I made by adding a link to the navigation, I didn't break anything. So I could go ahead and merge this in, into the master branch, and be totally happy. However, this is a common path to take when you're doing smaller features. You build it. You test it. You make sure it works. You get code review. You put it out there. But what happens when you're doing like a really big feature? If you're doing release-based development, you might lump all of those changes together into one big release and say, this is the new release that supports this new thing. But when you're doing continuous deployment with a web application, you can't really do that. Diverging from the production version is your enemy. The longer you have a branch open where you're making changes, making changes, making changes, and not re-integrating that back to the main line of development, the harder you're making your life for the future. So what we like to do is use feature toggles. So we'll create kind of a break, a logical break in the code, where you can choose two code paths based on certain variables. And that allows us to deploy new features to the public production version of our website or our app without impacting all of our users immediately. So practically speaking, what does this mean? This means that I want to be able to deploy this new feature that links to Harvard to production, but I want it to be disabled by default. So I want nobody to see it. I just want that code to be in production and ready to go whenever I need it. And I want to be able to enable it for one person at a time. So I want to be able say, when John accesses his web page, then it should show the Harvard link. But when anybody else accesses it, it should not. So that's called a feature toggle, and I'm going to show you how we build those and deploy them into production. So I'll go back to my header. And it's pretty straightforward. We have one line here, this line number 7, which links out to Harvard. What we want to do is add a conditional around it. The conditional should be based off of the current user, and it should be based off of if the Harvard feature is enabled or not enabled. So the first thing I'm going to do is I'm going to use some Ruby syntax to say, I'm writing logic code. And I will say, if current_user.feature_enabled question mark, and then I'll call the feature Harvard. Simple IF statement. If the feature's enabled, I want to link to Harvard. And that's it. And then I just say end to end the statement. If the feature's not enabled, do nothing. OK, so I'll save that. Now if I go back to my browser and go to my local copy, you can tell that this is my local copy, because it's got this red bar across the top, which is kind of development mode for us. And you can see the Harvard link is here. If I refresh this page, it's gone, OK? That's because the feature is not enabled. Up here in the top, there's this Site Admin button. This links into a backend for developers who work on this app. So if you're a developer working on the open source version of this app, you would have access to this through your copy of the app. Or if you were a GitHub staff employee working on the production version, you'd be able to access this as well. Within this interface, we link out to this thing called Features. And Features, this tool, is actually an open source Ruby gem called Flipper. So it's GitHub Flipper. So here's the gem. If you're interested in building apps that have feature flippers and you're using Rails, this is definitely a thing to check out. It's probably too in-depth for me to go into all of how this works, but we didn't build everything that I'm showing you uniquely for this app. Every Rails app that we use has the ability to do this feature flipper thing through a third party library. In JavaScript, there are other libraries that do this. Different frameworks have different libraries for this kind of thing. So it's a common pattern in deploying production web applications. And also desktop applications. We do this in some of our desktop apps as well. So anyway, I'll click on the Features toggle. And it says there are no features. You need to add a feature. So I'll go in here and click Add Feature, and I'll call it Harvard right here. And then click Add Feature. And now I have this nice interface that shows me ways to enable the feature. I can enable the feature for a percentage of actors. That means if you have 100 users, enable it for 1 in every 100, 2 out of every 100. What's interesting about the percentage of actors is that that is based off of the fact that when a feature is enabled for a given user, it stays enabled for that user all the time. It's very important that if somebody goes your website and the feature's enabled on one page and they navigate to the next page, it shouldn't disappear. So you can think about this as taking a numerical ID for the user and doing like mod something. And if it has a remainder or not, it's enabled. So for a given user, it will always be enabled or not be enabled. The other option is percentage of time. This allows the feature to be enabled, basically, for every request, 1 request out of 100. So for the same user, it might be enabled sometimes. It might not be. It just depends on what you're implementing. And then lastly, there's individual groups and individual actors. So I can enable a feature just for one user or just for one group. Since this is my development environment, I know my user ID is 1. So I'm just going to table this for user 1. I'll just enter this in here. Click Add Actor, and now the feature is partially enabled. You can see it's yellow. And if I switch back-- where did my thing go? Oh, I have to go back here. If I switch back to my local host and load the page, you'll see that now the Harvard link is here. Now, I'll open up a separate window, incognito window, and I'll go localhost 5000. And this time I log in as a different user. Uh-oh, what'd I do? Oh, it was disable this feature. Let's do this. I'll log in as a different user. And then in this case, I just have a student, like a test account. I'll authorize this. So now I'm logged in in this browser window, which has the gray bar, as a different user. Go back to my code and fix that. And for this user, you'll notice that the Harvard link is not enabled. But for this user, the Harvard link is enabled. So I basically have a feature toggle in there that allows me to specifically say which users get what. Why is this useful? This can be very useful if you do-- when you're making new features, you can enable it just for your friends, or you can enable it just for people who work at your company, or you can enable it just for beta testers. So what this means is like the problem we were trying to solve before is that you can diverge from your main line of development, and the longer you stay diverging and you keep spreading apart, the harder it is to reintegrate your code later. This means that you can add a feature toggle, put in an unfinished feature, or a beta feature, merge it in, and every time you improve it, just make a new pull request, a new branch, doing it bit by bit by bit, and you're always re-integrated. So you never have this big scary day where you say, we made this huge new feature for the app. Now we have to integrate it with all the other work all the other developers have been doing. It's been integrated all along, so now you just click a button and it's enabled. Then once you're happy with the feature and you know it's been working well, you can delete the code that lets you choose two paths and let it just have one path. So if we go back to our feature toggle board and click on Harvard, I'll disable it for my user. And instead, I will enable it 50% of the time. So if I go to my private user and I refresh, it's not there the first request. Not there the second request. Oh, there it is. Now you see it showed up right here. If I refresh it again it might disappear. Eventually comes back. So 50% of the requests. And that's true across all users. So if I go to my normal account, it would do that as well. Put it to zero. And then ultimately, you can just click Enable here and it will enable for everybody. All the time, enabled for everyone. So now we're able to create a feature flag, push it out to production, and see it in action. So here we go. In my case, what I want to do is look at the code. We're happy with this feature toggle. Then we say git status. We have modified this file with a git add app views shared header. And then git commit, which says enable harvard feature behind a feature flag. All right, and lastly, I'll say get push johndbritton harvard. So this will push it up to my forked copy of the repository. And that will run the builds again. It'll run all the unit tests again, make sure it's good. Now, I'm also going to go ahead and push this. I'm not going to create a pull request for this on the main public project, but I do want to show you how deploying works. So what I'm going to do is I'm going to say git-- I want to do this in here. I need to add one more thing. If current user and. So I need this line, because you saw the error I got before. Because I was logged out, I wasn't a current user, so I caused the app to crash. I don't want to cause our production app to crash, so I'm going to add this if current user and if the feature is enabled, then do this thing. Otherwise, if there's nobody logged in, don't do anything, or if the feature's not enabled, don't do anything. So git add-- what's it called? git status, git add app views shared header, git commit -m, check for current user, git push harvard. And I'm also going to git push origin harvard. So in this case, I'm actually going to push the change up to the real production application's repository. And then what will happen is if I go in here, you'll see in my chat-- so this is really important to the way GitHub works and the way we work with teams. You can see I created the branch and I got a notification in the chat room. All of the other people on my team also got this notification. And they'll see it in the chat room. What's interesting is like I'm doing all this work on my command line. You can all watch and you can see because I'm screen sharing. But my coworkers can't see any of the commands I'm running. The reason we have all of this stuff publish out to our chat channel is because it gives visibility into what commands we're running to the entire group. So by using Hubot and using a chat tool, we're able to have a shared command line where people can see what's happening in the deployment process. So in our chat tool, I can run CI build classroom harvard. And what that's going to do is it's going to say, go to my CI tool. Uh-oh. That's not good. Well, it runs automatically as well. GitHub.com education classroom. If I go here, now you'll notice this is not the forked version of the repository. This is the actual version of the repository. I can go to this branch and I'll see the Harvard branch. And you can see the build is pending on Travis. So I can click this and see what's happening. Once this build finishes, I'll be able to actually deploy this out to production and try it out. So while this is building, does anybody in the audience have any questions while we wait for this for a sec? Yes, in the back. AUDIENCE: When you said your command line was pushed out to the chat, I didn't see what you meant by the command line, what-- JOHN BRITTON: Yeah, so the question was about my command line being in the chat tool. It's actually not that the command line is in the chat tool, but what I wanted to show is that while I was working on my command line, all my git commands, they were private to me. Nobody could see them. But when it comes to deploying a product, deploying a service, it's important for the team to be in communication and know what's going on. So we don't actually deploy from our command line. We deploy from chat. So I'm about to demonstrate that as soon as the build's finished, but I'll be able to go into our chat room. Actually, I could do it with the master branch while this builds. So I can go to our chat tool and I can say-- OK, you can see the build was successful. But I can say, for example, deploy classroom/master. So I'm going to deploy the master branch of my app to production. And so now when I do this, everybody on my team can see that I'm currently doing deployment. Nobody else can deploy. The deploy queue is locked, and you can see that my deployment is done. It went out to production. I literally just deployed the app to production as we were talking. But everybody else on the team can see that I'm doing this. Does that answer your question? AUDIENCE: Yes. And how would you set up-- were those hooks that were set up? JOHN BRITTON: Yeah, so this is all set up via Hubot. So it's an open source tool you could use to kind of set up your own thing. There's also this thing called the GitHub app, which it's a Slack integration. You could enable this for your Slack, which is giving us all these notifications. All right, so now we can see right here it says, Travis CI build successful, education/classroom - ref harvard. So this is the branch that I-- this Harvard branch is the one that I just made. So we can actually deploy it now. So I'm going to go ahead and deploy it to production. So deploy classroom/harvard. Now, hopefully I don't break the whole thing. It's deploying. And then once it's finished, I can actually go to the public GitHub Classroom website and take a look. Presumably nothing has changed because we didn't actually change anything in the-- we didn't turn the feature toggle on. So we just put some code out there that's now integrated with our codebase, but it's not publicly-- it's not running for anybody. And then we'll go and enable it and kind of walk through the steps. Takes the moment. So in the meantime, while it's doing that, I'll kind of walk through-- oh, it's finished. There we go. John's production deployment of Harvard is done. Great, took 80 seconds. So I can say classroom.github.com. App is running. It's not broken. I can sign in. I can see my normal classroom. There's no link about any of this stuff. Because I'm a staff member, I can go in here and click on this and go to the features thing. You can see we have some real features in here about deadlines, student identifiers, different kinds of features we've been adding recently. I can add a feature called Harvard and I can enable it for user 1. And now if I go to classroom.github.com, you'll see-- did I break it? Maybe I'm not user 1. We'll enable it for staff instead. Oh, no. The live demo failed for this, but I will-- you should believe me, it does work. We saw it work in development. So I'm just going to go ahead and do deploy classroom/master and put things back how they were. So that unlocked production. It redeployed the master branch out there. And this is actually like the demo didn't work, but it's a very similar experience to what would normally happen. I would test it on my local environment. It would work. I'd have my test run. The build would succeed. I would deploy it. I would go out and test it, make sure nothing is broken. If something's broken, I redeploy master. I go back into my development environment and try and figure out what's wrong. I don't know what I did wrong, but I'm not going to debug that right now. So that's basically the standard flow going from just taking an open source project or a project working on with coworkers, doing a feature branching model, committing that stuff, sending it up to GitHub, and then deploying it. I think what we didn't do in the case of deploying to production, which I just wanted to reduce the amount of noise for my collaborators, was normally what I would have done is made a pull request in the production repo, had a discussion about the feature, but then that would create noise for the hundreds of people who are on that repository. So I just kind of went around that for demo purposes. But like we did on my fork, we can do a whole conversation about what the features were and how it worked. So that's what I had for that stuff. I wanted to go and offer up the opportunity for some questions around this flow, why we might choose certain things. So I'd kind of throw it out to you all. No takers? There we go. AUDIENCE: Can you recommend a good resource for-- I see there's a lot of components in here, a lot of setup. JOHN BRITTON: Yeah. AUDIENCE: [INAUDIBLE] all of this-- JOHN BRITTON: Like the end to end thing? AUDIENCE: Yeah, is there like a book or does GitHub-- JOHN BRITTON: So I don't know about a book. I would point to Travis, which you're going to hear about next, as kind of the go-to place. I mean, there's other tools. I like Travis, but there's other tools out there. But go to them and you can basically see how to set all this stuff up. What's cool about it, too, is I'll just show you my Travis CI account for this. So I can go to my Travis CI account and basically I can list all of my GitHub repositories. And I just click a checkbox and it just automatically works. If you have any of the standard frameworks, so say you're using a Rails app, or a Node app, or basically every language framework combo has a testing system, if you use the standard thing, all you need to do is have one point of entry to run the tests. So in our case, we do everything in scripts. And we have script CI build. And that's just-- if you have that one file that will run the test, then Travis can figure everything out automatically. And you don't have to configure GitHub at all. You just log in to Travis, you click the button, and Travis configures GitHub for you. And I think that's the best way to get started is like go through their guides on CI/CD. There's also-- and I guess I should plug this as well-- developer.github.com. So if you want to be a bit more sophisticated, I know a lot of teachers are using this stuff for customizing the way that they have workflows for their students and stuff. But if you want to use this for your own projects and you want to customize it, you can just go to developer.github.com. Check out the API docs, and specifically this section here on the webhooks. What that does is it gives you basically a way to listen to events happening on GitHub.com and respond to them. So the most common event that people listen to is the push event. Anytime somebody pushes code to GitHub, GitHub will send a web request to a URL of your choosing and notify them of what's changed. And then you can build an integration. It does all kinds of stuff. So that's like if you want to get more custom, I would start on developer.github.com. If you want to have just standard thing, go to like Travis or another one of those services. And I should repeat the question, I guess. Or is-- the question that that answered was, is there a resource where I can get started? Yeah. AUDIENCE: Just trying to understand your deployment flow right there. You are sending a message from Slack to Hubot which is running some kind of a script which is kicking off your test in Travis and then deploying-- JOHN BRITTON: Well, the test happened first. So the question was, I want to understand the deployment flow, like what are the steps and what are the pieces? So first we write the code and we get the code hosted on GitHub. Then when the code gets pushed to GitHub, GitHub uses webhooks to trigger a build. That's all automatic, I don't do anything. Once the build is finished, or even before the build is finished, I can go into our chat channel. And that's actually one channel, education ops. It's all about this education stuff. And I just say, period deploy and the name of the app. And we have Hubot in the room listening to commands. So any person who's authorized to use Hubot, if they type a command in the channel, everybody can see it and Hubot sees it. And when Hubot sees it, that triggers a flow for Hubot. Hubot doesn't actually do the deploying. So this app in particular is hosted on Heroku. So what Hubot does is it tells Heroku to get this new version of the app and have it be deployed into our production pipeline. So Hubot is kind of like a little bit of glue there. And the part we need Hubot for is so that we can do it in the chat room. I could do this without being in the chat room. I could do it automatically when I deploy. And in fact, if I made a pull request with the changes and I just clicked Merge and I didn't talk to anybody about it, as long as the build passed, it would get deployed to production automatically without even talking to Hubot. When I said that you can actually deploy before the build passes, what'll happen is if you type in deploy the app name before it passes, then what will happen is Hubot will wait until the build passes. If the build fails, Hubot will tell you, sorry your build failed, fix your problems. Or your build succeeded, I'm deploying your app now. So it kind of always does that. Another kind of interesting component about that whole flow is GitHub is getting to be a larger company. There are a lot of engineers there. A lot of people working on GitHub.com. There can be a limitation on the number of people that be deploying at one time. So we've kind of developed like a queuing system, where it's like you get in line for your deployment. So basically, GitHub is constantly being deployed, like nonstop. Just deploy, deploy, deploy, deploy. Whenever the first deployment finishes, Hubot notifies the next person and starts their deploy. And there's just a whole process. They're always getting out there as fast as possible. Do we have any other questions from folks? All right, well then I think that we may be finished up a little bit early. But thank you very much. Do you want to come up? BRIAN YU: Yeah, thanks so much. JOHN BRITTON: Thank you. [APPLAUSE] BRIAN YU: We'll go ahead and take about a 5 to 10 minute break. And then when we come back, we'll have Anna from Travis talking to us. OK, welcome back. Next up we're going to be talking a little bit about Travis CI, which you may recall from last week when we talked about testing in CI and CD for a bit. And so we have Anna here from Travis, who's here to speak with us about Travis CI. Turn it over to Anna. ANNA NAGY: Hi, I'm Anna. I'm a product manager and heading up involved in our education initiative. So I'm here to talk about continuous integration, which if you are involved in software already, you certainly probably have come across this. And I hope to go into enough detail that it's useful. If you haven't, we're going to start from the top. So I work at Travis CI. We are a leading continuous integration platform. It's essentially a tool for software engineers to maintain quality and collaboration. We started in 2011 in open source, moved to paid and enterprise over time. It's entirely GitHub-based. So your GitHub repositories talk to Travis if you would like them to. A bit of a side note. Microservices, Ruby Go, Ember.js, it's cool stuff. You can find us also on GitHub.com/travis-ci. Also, we do have an education program, just to plug it real quick, education.travis-ci.com. If you're on the GitHub Education pack, which you all should be if you have a dot edu, go sign up. We have an organization club. Contact us. And there was a question earlier, there are a couple of docs down there for getting started, kind of background and stuff. So all that stuff is there. But docs.travis-ci.com, it'll get you all there eventually. We also are involved in other education initiatives, so through Rails Girls Summer of Code have like a paid fellowship for women and non-binary engineers getting started. So highly recommend. Some general stats about us. We are smaller than GitHub, but it's pretty cool. So right now, 101.3 million repositories have built on Travis. About that many CI jobs, and that much compute time per month. So what exactly is happening in all of that? So to back up, why do you need CI? So imagine you have a project where you have to implement some stuff. Maybe it's homework. Maybe it's a product you're working on. You get this feature request. OK, we need to write this. You write it all at once. You try and fit it back in. And it always goes great, everything merges, it's fantastic. Generally there's some testing that needs to happen. Generally you end up with checks. Maybe you end up doing more and more shorter and shorter components, especially if you have multiple engineers working on a codebase together. Everyone's changes happening all at the same time is totally not a recipe for chaos. So this is where CI comes in. So continuous integration, integration being like let's get our changes in. Continuous being happening all the time. That said, as I go through this talk, there's a bit of phrase overloading here. So CI is both the process and the concept of continuous integration, like the best practice of moving stuff into the codebase over time, as well as like literally the tool. So when we say this project has CI, they probably follow CI best practices using a CI tool. So it's a bit of conflated terminology, but both refer to kind of the same thing. The biggest thing about CI that you talked about last week is testing. So I'm going to talk from an industry perspective. John Britton showed a little bit before. But testing. So code is tested, or it should be. If it's not, please write tests. It helps. It's a good idea. Or my favorite is the YOLO merge, where you just sort of send it to master and hope. Also good. But all modern testing language, in fact, even Fortran have testing frameworks. So use them. In JavaScript, there's Jasmine, Mocha, Jest, a bunch of others. Python has unit tests built into the standard library, but there's an interesting test runners. Nose test is one I've used. py.test is another. All sorts of good stuff. Many languages go, have tests built in. Ruby has RSpec. They all exist, so you can go and find them. Also, look up Fortran. It's kind of neat. So types of tests-- you can look at it from a few different angles. And I am going to compress all of quality engineering into about five slides, so apologies to any quality engineers in the audience. It's a great spot if you're into automation, so your tests can go both into automated or manual tests. I'm going talk about automated tests, functional versus non-functional tests. Functional being like, we're going to test a function. Nonfunctional being like, reliability, security, sort of operational things. So automated-- the machine follows a script that you've written or someone on your team has written. And you assert an expected result and verify that the assertion is true. Sort of like a programmatic version of checking a hypothesis, only unlike physical and social sciences, we're going to go back and change our results until we get the hypothesis. Please don't do that, except here. So we're going to set like, oh, this string, this output is true, we expect it to be true. It's not true, I'm going to go and fix my code until it is true. So we're checking-- we'll write tests beforehand or after-- beforehand's a good idea to check that things are working the way they should. As a side note, manual tests, this is literally someone manually going in and checking. Good idea to do it at times, but CI is really concentrated on automated testing. Then functional tests also are broken down into different categories. So the root word function here is like programming functions. We're going to test a function. So there are subcategories, unit tests, integration tests, component tests, others too. This is a classic diagram I just felt like I had to throw in there. You start with your unit tests. Fail fast on the unit tests. And then work your way up, such that a component test tests like a package, a module, a larger part. Integration test tests how things work together. And end to end is like an entire flow. So if you're writing a testing suite, a good idea is to start with unit tests. Does this method work the way I expect? So examples. Here's a method out of our front end. I will look at it later. It's generating the build status badge. I'll show it to you, but literally, the build is passing. Let's make this into a markdown thing you can put into your GitHub read me. That's our method. Here's our test. In a great irony of software engineering, the test is longer than the method. But it's good. What it's doing is saying, OK, here's the assertion. It generates a markdown image with a particular string that looks like this. If this is changing, then we're going to fail for whatever reason. So if this test passes, we haven't broken anything. This is great. If it hasn't, then maybe we need to go back and take a look. Good things to test for, just backing up in general. Things work the way they should, like a user can see the things they are supposed to be able to see. The string is generated the way we want the string to be generated. Or things fail the way they should be failing. A user can't see the stuff they're not supposed to be able to see. That's a good one to keep in mind, because users should not be able to see admin things and vise versa. So the idea with testing is stuff should change only when you intend it to, and you update the tests accordingly. So if you write your tests beforehand, this is a whole framework called TDD, Test-Driven Development. Lots of folks use it. So that way you know the code you are writing matches the output you are expecting out of it. Just a general statement is, it's really important to keep in mind that unlike student work, or unlike personal work even, software engineering tools are for teams. So the testing is to help engineers keep stuff from breaking things that impact other people, breaking production in unexpected ways so that when the deployment goes, we don't-- this side issue has happened. It can also function as a kind of documentation. This is what I did. This is what this method is supposed to do. These are the results we expect. And this is the stuff you can rely on. So if there is no documentation in a repository, shake your head and go read the tests. So separating out-- so after we've gone through tests in-- tests happen when we're doing a software build. So we're going, I have this code. I've written it. I've written the tests. We're going to then build this code, compile it maybe, prepare it, and turn it out to the world. So this is just-- the next section just on builds and building. Any burning questions in the meantime? OK. So builds and building. Build can both be like a verb, like build this source code into something. And this is really interesting, and I'm going to skip over this a lot, but compiled versus interpreted languages. So some languages need to be turned into a source code that is executable. Others it's less clear cut. So builds versus interpreter languages may jump straight into testing. Sometimes there are optimizations that happen first. This is some really cool stuff in computer languages. This is also true. So builds can get rather long. This is the classic XKCD and I felt like I had to include it. A build is also like the end result. So the build from yesterday was broken, but today the build passed. That refers to all of the change that happened between now and then. So we can say the build as the end result of a build process. So we went through the 10 minute build process, and then we have the build. It functions as a kind of audit log. So here you can see me going back and forth with our technical writer, talking about-- like this is actually a documentation discussion in which, apparently, I had a really hard time with the passive voice. So it was thought to look great. There, in specific, it's highlighting the GitHub status checks. So when we post to GitHub, we'll see both the positive status, like yes this passed, the negative status, no this did not. And it functions as kind of like a yes, this is what this commit did, this is with that commit did. GitHub is really nice and gives this in our UI, but the general concept stays anyway that builds are highlighted and you-- the statuses are highlighted. You can see where it changed. And I think the last one was a merge into master in which we also fixed a hyperlink. So why CI in general as entire systems as opposed to like, why don't I have a single machine that just sort of we test on every so often? It's to solve this problem. Hey, it works on my machine. It's going to be fine. It's always fine. And there are many different flavors of this problem. So reproducibility. Standard environments mean we can eliminate special customizations and hacks from influencing the results. So maybe you've installed additional dependencies, maybe you have special scripts that when you know one thing happens, go do the other thing. If someone else doesn't have that, or even if your production environment doesn't have that, then what? Software is built on top of the entire ecosystem of dependencies below it, so if those things don't exist, you might have a failure in production that's really hard to debug, because it works on my machine. It also helps people check other people's work. If we know the environment is standardized, we have this clean installation, we know the environment is standardized, oh, this build failed on this separate machine, it must be the code or how we're working with this clean environment, which we can fix. As opposed to, say we're working on an individual laptop. Oh, let's all crowd around and try and figure it out and hope it works in production. So reproducibility is a big deal around CI systems and why do we have external CI systems. Also just a shout out-- collaboration is really important. Because if we have these environments that we sort of can trust, or at least can predict, people don't need access to the exact same things. So I can go and work from wherever. I can go and work from headquarters in Berlin, or I can come back here, or whatever. Because we're not necessarily looking at the exact same installations, we have some reasonable, predictable environments, other contributions can be included. Also, this is a big deal in open source and corporations, because of this wide distribution. I also touched on it, really tidy deploys are a big deal for CI. So a standard environment that looks like the production environment, or even a standard environment that we can configure the production environment to look like, any flavor of that, it gives a better view of what will happen when the code is merged in. You don't actually want to test on production most of the time. It's one thing to feature flag. It's another thing to do real live tests on production. We would rather get our testing environments as close to production as possible, as opposed to just, say we have customers, hope that they don't see the problem. It also helps, again, prevent forgotten customizations. Oh, I had this machine. Before the last Mac update, I installed this Ruby environment. This happened to me. I updated my operating system, now it's all gone. Thank goodness I have another-- I'm not testing on my own machine. Otherwise that would have taken forever. We also see things like faster development with CI systems. So this was a really interesting study that we heard about, which is that CI both improves confidence in the code in an application and the pull request submitted to it. So people, when there is CI, people feel better about the code they are maintaining and they feel better about the pull requests they are receiving. So projects with CI will release about twice as often and pull requests are accepted almost 2, but 1.6 hours faster, which is a big deal both in open source-- that means people can contribute better-- as well as in business. That means you can move things along and people's valuable time is-- you can get more distance from it. There's a link there. There are a lot of other interesting studies on CI out there as well. My email is at the end of the slide. I would love to send links later, but that's a good one. Also, CI will then function as kind of this automation hub. So I have this big, long list here of coverage tool, of different things that will happen. Code coverage being like, how much of our code is tested? You may or may not want to test all of it. Some of it is illogical to test, but some of it really should be tested. Other things, linting does this following the best practices for syntax. Some languages will let you get away with syntax that you shouldn't, but it does make it harder to maintain and harder to read. Language run times-- does this run on Python 2.7, does this run on Python 3.6, does this run on Python 3.5? For libraries, this matters a lot. So I have a library of Python tests. Going to make sure this works. Everyone is using everything in Python 3-whatever, so I'm going to test that. I'm also going to backport it and just make sure this runs on 2.7, because maybe someone's on it. So you can do a lot of that with CI. Then dependency management, configuration management, these are all kind of, again, the libraries you're building on or the deployments you're doing, and various other things. I threw in some documentation generation. There's some really cool frameworks for that. Demo generation, really cool frameworks for that. So this is where we start hooking up CI to CI, which we've talked about more. So CD being either Continuous Delivery, this is ready to be deployed whenever, or Continuous Deployment, this is deployed all the time, GitHub style. One way or the other, you need CI doing regular tests, because if we say every commit is deployable, conceivably, we could deploy every commit. And if it's untested, we're back to the YOLO situation where we just hope. That's the CI/CD interaction. Also, CI jobs will often spawn automated deployments, right? So hey, this passes. I like it. It's on the branch that we allow to deploy, let's just go an deploy. That tends to be more in the continuous delivery category. So tests ends up being really, really important. If it's deployable every change or if it's deployed every change, we want to have high confidence in the deployments we are doing. So a couple examples, and you will see some familiar faces. These are just literally some build configurations I pulled out of open source so we can have a look at what exactly is happening. So CS50 manual, it was a fabulous little Travis YAML configuration to look at, because OK, you can see the merge here. David was merging in pull request about last week, I think. And it was passing. And this was the Travis YAML. You can see the language, what version we're on, any sort of caching that's happening. The script is building that, making sure that the Jekyll build runs. And does this Jekyll project properly turn into the web page it's supposed to do? If it works, we're going to do a deployment. We're going to go to Elastic Beanstalk using this deployment process. We're going to use these use secrets, which you can't see because it's open source and it's hidden. We're going to use these secrets to this part of AWS. App is named this, like the whole thing. And then in the last bit, we're going to notify a Slack room. I assume there's a deployment Slack room or some interesting Slack room in which everyone is cheering on the manual. So we see CI both being like, hey, do this build, make sure it works. Do this deployment, and also tell people it's done. This is one of our projects. This is one of the biggest Glue projects at Travis. This is Travis Build, which takes this Travis YAML-- this being a YAML configuration. YAML is like this sort of key value configuration language. So we're going to take Travis YAML and we're going to turn it to Bash. Bash being a shell language which the build is actually executed in. This is a big, long Ruby repository in which a bunch of things are happening. This is the key part of this build. So if we go back here, we have these three different tests. We're going to deploy it to key.io. And then there's an additional check these rate tests that happens after that sadly didn't fit on my slides. So again, here we're going to run these tests. Beforehand, we're going to precompile and clean everything. We're going to run the test, the spec tests. Do an additional validation test that we've written ourselves. And after deployment, we're going to send it to a Code Climate. Code Climate being another tool for checking the health of your code. There are all sorts of awesome tools out there, by the way. I'm shouting at this one because it's in our repository, but go find there's some awesome ones out there. And then before the deploy, we're going to clean up. And then ultimately, we end up deploying it. So we have this project that goes from these individual steps, test test test, to you clean up and get ready, deploy if it's the right kind, and take it from there. So I have a quick demo next. Any questions before we get started? OK, and let me make sure this works. I thought I could only do one thing at a time, so I prerecorded it. So I can pause if anyone has questions along the way. So here we see my Travis account, my travis.org open source account. I'm going to go and enable my demo Travis web project. So I did this cooking show style because the build is a little bit long. So here's the beginning of my pull request. And there's the pull request itself. I'm adding features. I'm adding an emoji to the string here. So you can see the first project. I actually updated the test first, again, so that I could know what the output was supposed to be right there. There's two tests needed that updated, because they're checking the string. Take it from there. I'm going to go back, see if I'm remembering my demo. I'm going to go back and head back over to my editor there and actually update the code. So now that I've done the test, I'm then going to update the code and say, this is what is supposed to happen when you generate a markdown string. You're supposed to also have that emoji. And I pick tools because we're building things. And see what emojis I like. And now that we've done that, we're going to save it. And I'm going to come over to my terminal. That is my branch status. You can see what's happening. I'm going to add that file I have a terrible habit of get add dot, which adds everything in the current directory. Don't do that habit. And then I'm going to commit it with a, hopefully, clear commit message. I'm going to push that up. This is just sort of the general commit, git commit and git management tools. There are desktop apps. I like my command line. We're going to take it from there. And now the build is going to run. Build is running. See my pull request. That's my active branch. There's my build history. Apparently, I like gesturing with my cursor. And it's coming in. Travis will show the build in this current page once it has started. So right now, the build is going from like a request coming into the Travis, beginning of the pipeline, being parsed through things like Travis Build, until the worker's ready to start. And we take it from there. And here is the build coming in. So the first thing it's going to do is pull get chrome stable. Like Chrome for the browser, but in headless browser mode so we can test like that. Taking a while. And then it's going to set up assets where we turn this source code. This is our entire front end, by the way. If you go to Travis-CI/travisweb on GitHub-- Travis-web on GitHub, I forked that project and have done this on our open source front end. So now it is building the front end, downloading the build archive, which is the-- some of these modules, some of these node modules that were set up in the prior build. There's a whole concept of, if you have dependencies you can rely on, let's send them up to somewhere stable, in our case S3, and pull them back down for the next build so we don't have to pull it every time. Now we're getting the Node version we need, which is the underlying language here for this build. Getting Node 5. Installing the NPM modules, the Node modules that we need again. And now we're linting. So we're checking, does this exist the way it's supposed to, is the syntax correct, all that good stuff? Does it match the best practices? And now we're building the website into something that we can test automatically in the build. So it is going from source code to testable situation. We're doing that. For better or for worse, I picked Travis Web because it is a nice, clear JavaScript project you can go and look at. It does have a fairly long build, so it will build for a little while. For some background history, we picked Ember because it works really well with our Ruby applications on the backend. And we have this nice Ember project. There are a lot of great JavaScript frameworks that have come out recently, more recently. Embers need to feel like Ember. Highly recommend all these things. Again, these are all ways of building applications, building JavaScript applications, performant web apps in the browser. So now we're generating-- yes, question. AUDIENCE: So are you a contributor to Travis Web [INAUDIBLE]?? ANNA NAGY: Um, let me pause real quick. Oops, too much of a pause. And we're back to being-- I'll figure it out. I contribute in different parts. So I'm on the product and support side. So odds and ends pull requests. I do a lot of reading our code, because I end up explaining it to people. So I pull request it to different configuration things I've set up, different infrastructure things with people. But I personally don't work on the Travis Web project. A lot of cool people do. But I'm happy to try and answer questions if you've got questions about it. So, yeah. AUDIENCE: If your project was not on GitHub, can you use Travis? ANNA NAGY: You cannot use Travis without GitHub. Though you could if you have another source control project, you could add a second remote and you would then be testing off the GitHub remote. It's a little iffy, but that is a way to do it. But right now it is a continuous integration for GitHub projects. We actually grew out of the GitHub open source community, so we've stayed super true to that. Other questions? Otherwise I'll try to find my spot. OK. Let me see. And I think it was like half way. Oh, wow, that was pretty good. So we're linting, going back to linting. The next thing it will do is it will run all of the tests, or after it builds it will run all of the tests. I think I counted about 800 plus tests for all of these different things in this entire front end. I'll skip a little forward. Oop, backward. Sorry about this. Let's skip to the end. So I've skipped to the building step. The dots here are the tests that ran and passed. The stars are the ones we skipped. Sometimes you want to skip tests, because whatever reason that component's still under work. It's not the best practice, but it is something that happens every so often. So now we are building all these tests, all 847 last I checked, and taking it from there. Yes. AUDIENCE: So here your [INAUDIBLE] is not actually Travis and you're using some tests [INAUDIBLE]. ANNA NAGY: Travis running the tests in-- like running the NPM tests, the Node tests here. There is a test runner that is executing these tests. We are automating calling that test runner. OK, so I'm going back to my pull request. I'm saying it worked. I'm checking it, all my checks have passed. Because this is an individual pull request, I'm going to be wild and crazy and merge my own pull request. Highly recommend you get a review. But I added three emoji. It is OK. It will survive. So there is my pull request. And went pretty quickly. But in review, builds and testing in general, so the main goal is to provide a way for tests to be run as development progresses, rather than at the end. CI came in as a theory in the '90s, but it's still a best practice that is being adopted. So we want tests to be run as the project is going so that we don't get to these great merge days where we go, OK, let's see if everyone's code fits together. Oh my goodness, it doesn't fit together, now what? Let's go back to the drawing board and figure it out. You have all that time that both requires, like we need to figure out this problem, but also, we did all this code and maybe it needs to be changed. Maybe it needs to go away. If you do these small increments, test them as you go, merge them in as you go, it's much more-- you don't waste time in quite the same way. It's much easier. You can also predict what other teams are going to do, because if you know everyone's tests are going to pass or they're not going to merge it in, then you know that there's a-- side effects are generally under control. New features are tested with assuming the best practice of writing the tests with the features. And historical builds become a kind of living audit log for tests. So yes, my build last month did this. This is what the output is like. Oh, we've changed it a lot, let me go back and compare. Or the last build that was successful did this, that, and the other thing. My tests, my project no longer is successful. What specifically has changed here? Let's go back and compare. When were these things done? Which commits had what status? There's tons of good data, tons of good information. And again, as you're doing your build, there are lots of different types of tests to include. So like the unit tests at the bottom, these are individual methods. Component tests, integration tests, working your way up until you're testing the bulk of an application. Oftentimes people, again, will start with unit tests. If the unit tests don't pass, you know particular methods are wrong. Go and look at those. And then it connects back into CI and CI/CD, both in open source and in business and in industry. So CI and CD are getting more and more common. They are more or less considered a best practice, though of course, every individual environment, every individual corporation, every individual project has different requirements. So I don't want to be like CI is for everybody, but CI is for everybody. So it's both like a trend and a best practice. If you know your code is reliable at any given point, and is tested and under control at any given point, you can do all sorts of interesting things around deployment, continuous deployment, other types of deployment, changing around infrastructure, especially in cloud native and all these cool stuff. So teams will use CI and CD for testing and deployment, also just general productivity. CS50 is posting to a Slack room. They're also all getting emails. They could be posting somewhere else. They could be notifying other things. We could be going back and saying, oh, we had an incident on our infrastructure. This is not happening with the manual. But say in another product we did an incident. What happened to the tests most recently? Can we fix this? Oh, we need to fix this really quick. Let's push this feature, make sure the test passed before we end up with an incident on top of our incident. So CI and CD are really deeply connected in that way, either by making sure you can deploy whenever, or deploying always on passing CI, like CS50 does. So any questions? This was like very high level, but I would love to talk more about it. Questions, concerns either about Travis or-- yes. AUDIENCE: Is there a particular reason that you rebuilt the environment every time a new commit is pushed versus keeping something hot and ready to go? ANNA NAGY: Yeah, so we do a bit of a mix of that with the caching. But the rebuilt environment every time does deal with changes to those environments. So either-- like from our infrastructure side or from like your libraries and dependencies? AUDIENCE: Infrastructure. ANNA NAGY: Right. So we actually have a fairly continuous infrastructure project, infrastructure deployment, so we do that. It also works so that as we are changing our infrastructure, as we are changing-- like we'll change infrastructure and Travis Build at the same time, Travis Build being the runtime side. So if there's a change to infrastructure, we need to change Travis Build. And so we have pretty continuous deployment. We do have pre-built images. So we kind of have these catch-all images of like ci-garnet covers like-- I don't remember off the top of my head. But it covers half a dozen languages and their various types. ci-amethyst covers all the stuff on the JVM and their various types. And so we will kind of have these catch-all images. And then when you select a language, we'll pull in all the specific stuff to it. It also lets for better community involvement. So we'll have like the D language contributes to Travis Build. I personally have never written D, but it's a supported language. Other questions? Yeah. AUDIENCE: We've had some issues, we also use a different CI. So basically, if you make a change to say a read me file, is there a way for Travis to know, oh, that's not going to break any tests, so we shouldn't really be running all the tests based on this [INAUDIBLE]?? ANNA NAGY: You can do like-- it depends on the workflow. So with Travis-- I'm going to speak from Travis. From Travis specifically, you could just skip the entire thing by adding bracket skip space CI to the commit message. So you could skip the entire thing. You could set up a matrix build where you would set the environment variable. So this type of change would have these kind of tests. And then if we don't see any change or we want to pass it along, we could do something different. You could also get rid of it out of the read me on your pull request, and then just-- or get rid of it out of the Travis YAML in your pull request and then put it back. I would still say run the whole test suite though, because you don't know what changed. So different stuff. But also, it will depend on project. So we can talk more too afterward. Other questions, other concerns about testing CI? Other things? I can talk a little bit about how we use Travis at Travis, which is kind of fun. We've had different situations where in order to fix the builds we have to pass a build, but the builds are stalled. That's really fun. Highly recommend that situation. We dog food all of Travis, both Travis open source, close source, and enterprise. So you can't push a change to say Travis Build unless your Travis build passes. So if you're changing it around, it gets super interesting. Questions, concerns? Otherwise I can call it early. It was fast. BRIAN YU: OK, well, thanks so much to Anna. [APPLAUSE] And with that, we'll wrap up for today. And next week we'll dive to talk a little bit more about scalability, and thinking about taking the web applications we've been building just on our own computers and how we might scale them to be larger applications used by some number of more users. Thank you so much.