BRIAN YU: OK, welcome back everyone, Web Programming with Python and JavaScript. And we'll pick up where we left off last week, which was exploring Django, which was a new Python web framework that we were able to use to continue to design more sophisticated, more interesting, more advanced web applications and do so more easily. It gave us access to things like a model already in place for how to authenticate users, how to log them in, how to log them out, how to remember that a user is logged in. It also gave us access to a really nice and easy to use administrative interface which made it such that if we wanted to very easily add new things to our models it was very easy to do that. And it also gave us the ability to enforce relationships between different models in ways that was easier than we would have had to have done previously, whereby if we wanted a relationship, for example, between flights and individual passengers, rather than have to create for ourselves another table that would relate an individual flight ID to an individual passenger ID we were able to leverage Django in order to simply say that we want there to be some sort of many to many relationship between flights and passengers. And taking advantage of that relationship, we could then use that in order to have Django do the hard work for us. So figuring out that we need some sort of intermediary table, and Django goes ahead and creates that. Well, we don't need to worry about how exactly Django's doing it. We just need to worry about what relationship we want. And so before we dive into the main focus of today, which is going to be about testing and also continuous integration and continuous delivery, we're going to talk a little bit more about Django and the features that it gives us to touch on a couple of things that we didn't quite get to last time in order to answer a couple of questions that people have had. And so the first thing we'll take a look at is inside of the same airline example that we've been using, we'll take a look at how we might customize the administrative interface. And so a couple of questions were raised during the last lecture about whether or not we could change anything about the administrative interface, that thing that's automatically created as one of the built in Django applications. And in fact there are things that we can do in order to customize it to our liking. There are in fact, many different options that we have available to us. But I'm just going to show you a couple just to give you a sense for what's possible. So if I look go into Admin.py, which is the place where, in last lecture, we really just had these three lines at the bottom where we registered our airport model as something that we should be able to edit in the admin interface in addition to the flight and passenger model as other things that we wanted to be able to edit and modify in the admin interface. What we can do is extend the existing model admin class in order to add additional features that we might want. And the full list of all of these features is available in Django's documentation. But I'm just going to show you a couple that might be interesting or of use. So one, for example, is that you can determine how users are able to interact with particular data points. So for instance, I'm extending the model admin class to create a new class called passenger admin. And this is going to be a special set of configuration settings that I'm only going to use when I'm editing passengers. And I use that down here when I register the passenger class for the admin site. I'm saying, use this special passenger admin class which is going to contain additional settings or additional information about how I want the user of the admin interface to be able to interact with passengers. And in this case, I'm using this special variable called filter horizontal which is going to give me a really nice way of taking a passenger and manipulating what flights they're a part of. So I'll show you what that looks like. Filter horizontal is one of those built in settings that I can manipulate using Django. And if I run the server now, and go to this URL, and I go to /admin to get to the admin interface. What I can do is, now when I go into passengers and I click on an individual passenger like Alice, what I get is, this is what the filter horizontal administrative window looks like. And I'll get a really easy way of controlling what flights that they're on. Before, if you recall from last week, I was clicking and there was Control or Command clicking on different flights to either select or de-select them. Now I can determine what flight Alice is on just by clicking on a flight, moving them back and forth. Now Alice is not on any of the flights. If I click on another flight and move it over, now Alice is on both of these flights. And so these are just small changes I can make to change the appearance of individual flights. And one other thing you might have noticed from last time. And before I do that, let me go back to the way it was before. If I were to go into Admin and go and edit an individual flight, for instance-- I want to edit flight number one-- you'll notice that normally what I get are just the options to edit the properties of that flight. I can edit that flight's origin, I can ediit the flight's destination, I can edit the flight's duration. But I can't, built in here, edit the flight's individual passengers. And why not? In the passenger side I was able to edit the flights, but I can't seem to do the reverse. Well the reason, if we look into models.py here, is that if we look at what properties the flight has we only defined origin and destination and duration. Whereas for passengers we defined their first name and their last name and this additional property called flights that was associated with passengers. So therefore when we go into the passenger admin interface, I can edit what flights that passenger is on. But when I go into the admin interface for editing flights, I can't control what passengers they're on. And so this is something we can also solve just by doing some customization of the administrative interface. So what I'm going to do is first define a new class which extends StackedInline, where StackedInline is just one of Django's built in classes that lets us, in a stacked format, add new relationships between objects. And so I'm going to call this one PassengerInline, and you'll see why in a moment, where this is going to be the class that represents the place on the admin interface where I would like to be able to add and modify passengers. I need to specify what model I want to use, what data I want to be associated with this class that I've just created. And in this case I'm saying passenger.flights.through. This additional .through property refers to that intermediary table. So recall that we have a passengers table and a flights table, but there is some sort of relationship table between them-- some table that connects individual passengers to individual flights. And this .through is how I sort of access that intermediary. And extra equals one just means that I want to be able to add one additional passenger at a time. And you'll see what that looks like in a moment. And then in the special flight admin class that I'm extending from the existing admin class, I'm saying I want to add this additional inline section of the Admin page called PassengerInline, which is the one that I created up there. And so if I now register the flight model to my admin site, specifying that I want to add these additional settings in the flight admin settings, now when I go to edit an individual flight in addition to getting to edit the origin, destination, and duration, I also see this view, which is the stacked inline view, where it's just a stacked list of all of the passengers on this flight. Whereby if I want to add an additional passenger to the flight I simply need to go down here and select which passenger that I want to add. Of course, in this case, both passengers are already on the flight so it wouldn't make much sense for me to add one of those passengers again. But that would be the sort of thing that right now our application can't handle but you might want to add additional constraints that help enforce. And so the goal of this is just to show just a very brief teaser as to how you might be able to manipulate or customize the administrative interface to look and feel exactly the way that you want it to do. And if you take a look at the Django documentation for the admin interface, there are a whole lot more settings than just the couple that I've shown you here that you can get a taste for, that you can experiment with. And I would encourage you to experiment with it as you work on project three just to get a sense for the different ways that people can begin to interact with this administrative interface that Django has really built out for you. The other thing that I wanted to touch on last week that we didn't quite get to was how to deal with static files. So a couple of people have asked about how you use static files in Django, like CSS files or JavaScript files that are external. So the way that you would do that in Django is, inside of our HTML file, so one of our Django templates, what we would first need to do at the top is use this additional Django command called load static which loads static files into this page. And then if we wanted to reference a CSS style sheet, for example, this is the specific Django syntax for doing that. So it looks a little bit different than the way it looked in Flask, for instance. But inside of the curly braces and percent symbols, we'd have the static keyword just to say we want a static file to be used here. And then which static file, well, it'll be the flight's directory and then styles.css. And if you look at the structure of our application here, inside of the flight's app, I have inside of it not just a templates directory that has flights and then based on HTML along with all these other HTML files, but I also have a static directory, inside of which is the flight's directory, inside of which is this styles.css file, which is the static file that I want to load in all of my HTML files. The question also came up last week as to why it is that we need this additional flights directory. In the same way that we had an additional flights directory inside of templates, we also have this additional flights directory inside of the static directory which feels a little bit redundant because we already have a flight's directory, which is the name of the entire app. And the reason, again, is the same exact thing as before. It has to do with the fact that if I had multiple different apps, each of which had a styles.css file for instance, a pending flight slash in front of it helps to namespace that file, to make sure that I know how to reference it in order to access it. So just wanted to show you those things just to give you a sense for a couple of final things in Django that are possible that you may be interested in doing as you go about working on your project, though it may not actually, in the end, come up. But the main focus of what I wanted to talk about today is ultimately going to be about testing. And so as we begin to develop more and more sophisticated, more complex web applications, it's going to be increasingly important to test that code in order to verify that it's accurate. And so reasons why it might be good to test that code are to make sure that if I'm making changes to one part of the code base, making changes to one function that might be used in a dozen other places across the application, I want to make sure that the changes I'm making to this one function aren't going to cause some other part of the web application to break, for instance. And it might also be important because I'd want to anticipate the fact that functions may behave differently based on different types of inputs. And so I want to account for the fact that no matter what sort of input I provide, I want to know that my application is going to be able to handle them well. And so what we're going to spend a lot of time today focusing on is how to build out a system for testing. First of all, how you would even test things on your own. But then looking at what tools and technologies exist in order to help facilitate the testing process which ultimately becomes very, very important as we begin to develop more complex web applications. So to take a just very simple example, I wrote inside of-- let's go to the source and go to prime.py, which is a function that I wrote. Inside of a file called prime.py I wrote a function called is_prime. And what this function does is, it determines whether or not a number is prime. And this has nothing to do with web applications just yet, but I'm going to show you this example as a function that I might want to test to see whether or not it's accurate. And then we'll go from there in order to later explore, how do we apply these same ideas of testing a function in command line Python to going on to how do you test functions in a Django web application, for instance. So this function, is_prime, is going to take a number n, a non-negative integer, and tell me whether or not it is in fact prime, returning true if it's prime, returning false otherwise. And the basic logical flow for this function is that if the number is less than two, well, it's not prime because two is the smallest prime number. So we'll go ahead and return false. And then I'll run a loop going from two up until the square root of the number-- math.squareroot is a function that gets me the square root. And if it's the case that n mod i, where mod is an operator that gives me the remainder when I divide n by i-- if I divide the number I'm looking at by this i, this variable in the loop, and that number comes out to be zero-- in other words, n is evenly divisible by i-- well that means that the num n is definitely not prime because i is one of its factors. And so I'm saying, return false. The number is not prime. And then finally, if I made it through checking all of these factors up to the square root and none of them divided n evenly, then I can say that this number is in fact prime. And so at the end I'm going to return true. And so this is the basic logic for the is_prime function that I wanted to create. And now my goal is going to be, how do I test this function? How do I evaluate whether or not it's going to work or no? So one thing I can do is to test it is just to test it on my own. So I can open up the Python interpreter and say, from prime import my is_prime function. And now I have my is_prime function and I can test it. I can say, OK, is_prime of 23, well 23 is a prime number so this should return true. It should be prime. And it does in fact. And likewise I can say, is_prime 28. Well, 28 is not a prime number so this should be false. And so I press Return and indeed it does turn out to be false. And then I could try, is_prime 25, for instance. Let's see, well, 25 is not prime either because 5 is one of its factors. And OK, something seems to be wrong with my is_prime function. My is_prime function thinks that 25 is in fact prime when in reality it's not prime. So I found some sort of bug or error inside of my is_prime function. But all of this is ultimately going to be pretty tedious because as I think about using this function in the broader context of a larger web application where I'm constantly making changes to different parts of the codebase where I might be changing different aspects of the function, I don't want to have to, every single time I make a change to the program and I want to make sure that the is_prime function is working, to go in here and type is_prime 23 and 28 and 25 and try a whole bunch of examples to make sure that it's going to work. So what might I do instead? What are some other things that I could do just to simplify this process or make it so I don't need to manually go into the Python interpreter every time and try out the is_prime function on a bunch of different inputs? AUDIENCE: [INAUDIBLE]. BRIAN YU: Great, exactly. I can write some sort of program that's just going to run a whole bunch of these for me in order to make sure that I have the right idea. And so let's try doing something like that. If we open up testzero.py, what I have in testzero.py is a very simple Python application, or really just a Python module that contains a single function called test prime, that is going to test whether or not I get the expected value that I want. So if I feed into this test prime function a number n and a Boolean value of what I expect the result to be, this expected value, then I can check if-- so run the is_prime function on n, because I imported it up here at the top. And if I check the is_prime function and it's not equal to the expected value-- in other words, when I perform the computation it wasn't what I expected it to be based on what I thought the test should result-- then there was some sort of error. There's some sort of error in my is_prime function. And so I'll print something like, there is an error on is_prime n, where n is whatever the number. We expected it to be whatever we expected it to be. And so this function is going to simplify the process of allowing me to test whether or not a number is prime. And so in order to use this, now I just need to call the test prime function. So I could write a Python file, for instance, to just use the test prime function over and over again in order to test whether a number is prime. Another thing you'll sometimes see is people writing just a bash script, just a script that's going to run inside of the terminal shell without anything more. And so the way bash scripts work is that they'll generally just be a bunch of lines of commands that I might run on the command line. And in this case, Python-C is just a way of saying, run this particular Python command. And so on line one I'm saying, run this Python command. First import the test prime function. And then test to make sure that one is not prime. And likewise, I'm doing the same thing on this line saying, test to make sure that two is prime, and make sure that eight is not prime make sure 11 is prime. So on and so forth, testing all of these possible input values. And now if I wanted to run this I could run that script. And what I get is for the two for which the number did not match the expected result, where I ran is_prime eight, I expected eight to not be prime but there seems to be some sort of error here. I get that there is some error that's listed here. And so that's helpful. Now, any time I make some sort of change to my web application or my application in general, I can run a sample test file that's going to run a whole bunch of these tests and then give back to me some information about any line of output that I get here is a sign that something wasn't quite right, that something went wrong over the course of me trying to test this function or this application. So if we go back to prime.py, I can begin to start debugging this, try and figure out what exactly went wrong. Does anyone have a sense for what might be wrong, out of curiosity? Why it seems to be thinking that more numbers are in fact prime when they're not? If you played around with it for long enough, you might find that it has to do something with this loop. That for some reason I'm not testing enough possible values in the loop. And in particular, it has to do with the fact that when Python does loops in a range, it includes the first thing in the range but it excludes the last thing in the range. So for instance, if I do for i in range zero to five, print i, what I get is, I start with zero and it goes all the way up to for but it doesn't actually include five. So it includes the first thing in the range but it doesn't include the last thing in the range. This is a common source of what we call off by one errors in Python and other languages, where we had the right idea but we were just off by one number in something that we were trying to calculate. And so the problem in this is prime function just happens to be another one of those off by one errors that really, in order to include that final number, that square root that I want to test, I additionally need to test that last number, the actual square root itself, or the number closest to it if it doesn't have an integer square root. And so now that I've fixed the function and I want to test it, rather than go back into Python and then saying from prime import is_prime and then testing all of these individual values again, all I have to do is run that test file again. And this time the test file produced no output. Remember output was only going to be produced if there was some sort of error, if there was a line that was incorrect. And now when I run the test file I get no errors which tells me that, at least for the inputs that I provided, the test seemed to be working. Now this doesn't mean that my function is necessarily totally correct because it just means that whatever test I decided to give to this program, it passed those individual tests. But that's where it starts to become important to write good comprehensive tests. To make sure that if you're going to test a function or a particular part of your application, that you're really testing it under all possible conditions. That we have tests for even numbers and odd numbers and different types of numbers to really try and get a sense for, is this function really correct by testing it on as many different things as we can. And that's going to be important as we start to build larger web applications that are getting more complicated. We want to make sure that our tests are testing as many different possible situations that might come up as possible. Questions on anything so far? OK. So we'll take a look at a couple other features of Python that might be useful as we go about testing. And one thing is Python's built in assert command. And so the Python assert command really does one thing and one thing only. It takes the expression that comes after the word assert and it asserts, or just states, that this expression is going to be true. And if it's not true, then Python will throw an error, the same way Python throws an error if you try to divide by zero for instance, or access something in a dictionary that doesn't exist. It'll throw a particular exception known as-- it'll throw an assertion error. And so if I assert like, the statement true, for instance, nothing happens. It just states that it was true. And it's not actually changing or modifying anything. But if I assert something that's false, then I get this assertion error exception. The whole program quits and I get this error that says the expression that came after the assert was not true so there was some sort of problem. And so this can be useful if I'm trying to see whether or not a program is working correctly or not because if I want to state definitively that something is supposed to be true, I can assert that it is a true. And if it's not true, then my program is going to give me some sort of error. And so if I look at assert0.py, for instance, here's a very simple file that is just going to define a function called square that takes a number x and returns x times x, returns the square of x. And down here on line four, I'm just going to assert that the square of 10 is equal to 100, because that should be true. The square root of 10 should be 100. And I want to assert that that is in fact the case. So now if I try to run assert0.py, nothing happens, right? Assert doesn't print anything to the screen or have any other side effects. It just asserts that the thing that comes after it is in fact true. If it's not true, in an assertone.py for instance, if I try to assert a fact that is not true. I try to assert that the square of 10 is a 101, for instance, instead of 100. Now when I try to run assert1.py, now I get this assertion error because the thing that I was asserting was in fact false and so the Python program is going to give me some error. It's going to quit and say that something went wrong. And in particular, all programs when I run them give me some sort of exit code that indicate whether or not the program ended in a particular state or not. And generally speaking an exit code of zero means the program ended and everything went well. And an exit code of something other than zero, like one or something else, means that something went wrong as I was running this particular program. And in bash, in the terminal environment that I'm using, if I want to look at the exit code I can type echo $?. And here I see that the exit code of this program was one. Something went wrong and therefore this program exited with a status code other than zero. Whereas meanwhile if I had just run assert0.py, and I looked at what the exit code of that was, that exit code was zero because everything went fine. There was no problem. And so we'll return to this idea of exit codes and how we can use the fact that exit code zero means everything's OK and exit code one or something else means something went wrong a little bit later. But this just gives you the sense that we can use these assertions to either have some sort of error if something goes wrong, or do nothing if everything was OK. But of course it's going to again be tedious if we continually have to write these assert statements after assert statements all the time in order to verify that things are correct. It would be nice if there were some easier ways to do this. And Python does, in fact, allow us some tools that make it easier to do testing. And in particular, they have a framework called unittest which is a Python library that's designed to help make it easier to test your programs. And so we'll take a look at unittest. And then we'll look at how we can apply unittest to actual web applications. And so what we're building up to now is in tests1.py. And so this will look a little more sophisticated. And what we've done here is, first, import unittest. And now I'm defining an entire class. This class is going to extend unittest.testcase. So it's a class that's going to contain a whole bunch of individual tests that I want to run on my code. And each test is just going to be a function, or a method inside of this class. So I define a test called test1 and this docstring here, this comment, is just a label for the test such that if the test fails, then I'll know something about it. I'll be presented with this label for it. And so the name for this test, my description of it, is that it's going to check that one is not prime. And unittest has a whole bunch of these built in functions for asserting things. Rather than just a plain old assert that says assert some expression, there is a special assert false, for instance, that I want to assert false is_prime one. In other words, assert the fact that if I evaluate is_prime of one, that that should be false. And likewise, in test2, where I want to check that two is prime, I can likewise call is_prime applied to two and assert that the result of that expression should be true. And so I can define a whole bunch of these functions, each of which is just going to be an individual test that I want to run. So here is checking that eight is not prime, checking that 11 is prime, checking that 25 is not prime, checking that 28 is not prime, for instance. And then at the end down at the bottom I'm just going to run unittest.main, which is going to run all of my unit tests. In other words, run all of these individual functions. And so if I now run tests1.py then what I get is that it says it ran six tests in 0.001 seconds and, OK. Everything was fine. So nothing went wrong in this case. I didn't get any errors. If I change my prime function and put that error back in, such that now the is_prime function is buggy, and now I try to run tests1.py, now I get a whole bunch of these interesting errors. So up here at the top it's telling me what happened. This dot just means this first test passed, it passed, it passed. This one failed, then it passed, and then this one failed. And now it's going to tell me which tests specifically failed. So I failed the test 25 check, which is the check that's going to check that 25 is not prime. Again, this was that description line that I included in that triple quoted docstring. And then it's going to tell me, all right, what was the line that caused the assertion error? Well the error with here where I was asserting the fact that 25 should not be prime, that is_prime25 should evaluate to false and it was an error because true is not false. And likewise I got the same error on this test eight function, where I wanted to make sure that eight is not prime. And likewise I got another assertion error there as well. And so that seems like a little bit of overkill for a function this simple that's just testing whether a number is prime. And it seems like we probably could have done this without using unittest, and in fact, we did. But as we begin to want to test more sophisticated, more complex programs we'll see how the value of unittest really comes in and how we can begin to build on this in order to make our tests more comprehensive, make the process of running tests even easier. But questions about anything we've seen so far with just unittest before we move on? Yeah. AUDIENCE: [INAUDIBLE]. BRIAN YU: Good question. The echo command is just a Unix command that basically just prints something out. And in this case I wanted to print out to the screen the return value, or the exit status, of the previous command that I ran. Good question. OK, so that was unittest in the context of just a basic Python file, a Python module that contained a function. Now let's try and apply this to unittest in the context, or unit testing in the context of an actual web application. And so let's go into airline1 for now. And we'll take a look at inside of our flights application. We've been looking at all these various files that we've had before. So we had this admin.py file, which we saw before helped us to create the admin interface however we liked. The apps.py file was a file that just defined basic configuration settings. Models.py was on we were using to define our different models, whether they were airports or flights or passengers. And we dealt with URLs in view last time too in order to manipulate what people see and what URLs they have to go in order to see it. But the one file we really didn't touch last time was this tests.py file. So Django comes built in with its own testing framework in order to make it easy for us to run tests on our web applications because it's designed with this idea that we should make sure that our applications passing our tests before we deploy it for users to use. And so if we open up test.py, here's some tests that I've written in order to allow and facilitate for the process of testing this web application. And so the first thing you'll see up at the top is that I'm importing from Django.test, I'm importing TestCase. TestCase is an extension to the unittest framework. Django has taken unittest and built onto it in order to make it easier to test some Django web application specific things. And you'll see examples of that later. And so here's my class that's going to contain functions for each of the tests that I want to run. And inside of this class, the first thing that I do is set up the tests. So this is a function set up, built into the TestCase framework, that is going to run before any test ever runs. So before any of my individual test functions that we're about to see ever actually happen, the first thing that's going to happen is that this set up method is going to run which is going to do a couple of interesting things. And it's just going to set up the tests such that I can test things that I care about later. So maybe I care about testing things about airports and flights. So in order to really test this, the first thing that I want to do is set up those tests by creating some airports and creating some flights inside of my database such that I can then run some tests to make sure that they're behaving the way that they should behave. And so on lines 11 and 12 here, you see that I'm creating new airports. So this is a slightly different syntax for creating objects than we saw last time where before we just use airport and then all of the properties and then we saved it. This is another way of achieving the same thing. Just wanted to show you both. And so we can say airport.objects.create, create a new airport with code AAA and city City_A. Create another one with code BBB and city City_B. And then create a whole bunch of these individual flights where this has origin A1, A2, duration 100. And a couple others have different origins and destinations and different durations. Now what am I actually going to test? Well, let's take a look at an example of a function that I might be interested in testing. So I've added to models.py. I've added to the flight model this additional function called is_valid_flight. So the issue came up during last week's lecture of, well, right now based on the way that the flights are set up it's not at all clear that a flight is necessarily a valid flight. It might be the case that someone could create a flight that goes from London Heathrow Airport to the same airport and that wouldn't really be a valid flight but we'd be allowed to do it based on the way that our database is set up. And so maybe we want to add a function that is going to take a flight and check to make sure that it is, in fact, valid. And so that way when we display the page that displays information about that flight, it would also include information about whether that flight is valid or not. And if it's not valid, then we could do something about it. We might also, likewise you can imagine we could extend this to have a page that would just show me all of the currently invalid flights, all the flights that don't satisfy some constraint that we want to have on flights. And so what does my is_valid_flight function do right now? Well right now it returns a Boolean value, true or false, indicating whether or not a flight is valid. And what does it mean for a flight to be valid? Well in this case I'm saying a flight is valid if the origin of the flight is not equal to the destination. In other words, it starts somewhere and ends somewhere else. If it's starting and ending in the same place that's not a valid flight. And likewise, the duration needs to be greater than or equal to zero. I don't want a negative length flight because that wouldn't make a whole lot of sense. Questions about is_valid_flight? OK, so now what I might want to do is actually test this, make sure the is_valid_flight method is actually going to work because if I'm going to use it inside of my view, for instance, then it better work. And in fact I do use it in my view because if I take a look at the flight.html page, this was the page that before showed me just information about the flight number, origin, and destination. I've added an additional list item here that's just indicating whether or not the flight is valid. So valid is just going to be true or false by accessing the is_valid_flight property of the flight, which will now tell me whether the flight is valid. And so you'll notice that if I run the Django server now and go here, and I click on an individual flight, then what I get is this additional property on the flight page that's says valid true, meaning this slide is valid. It would be valid false if it were not valid. And so now that this valid function is something that's integrated into the logic of my web application I want to test to make sure that it is, in fact, going to work. And so I've created two airports, airport1 and airport2. This flight goes from airport1 to airport2. This flight goes from airport1 to airpor1, which would be a sign that that should be an invalid flight that I might be interested in testing. And here's one that goes from airport1 to airport2, but it has a negative duration, a duration of negative 100 for instance. And so now here's an example of a sample test. I'm testing the departure count of an airport. And so remember that last week when we set up our models for airports, we also allowed for an individual flight to be related to its two departures such that I can take an airport and access all of the departures from that particular airport. And so here what I'm testing is I'm making sure that that departure's relationship really works. So I'm getting an airport by saying, get me the airport that has code AAA. And now I want to assert that these things are equal to each other. I want to assert that the count of the number of departures should be three. And so if the count of the number of departures from A1 is three, then all is well and my test will pass. And if something's wrong there, if the departure count is not three, then this test will fail. Let's see the other tests that I've written. Here is one to test the arrival count in the same way. There's only one flight that lands at City_A, and so I'm testing to make sure that if I count the number of arrivals at City_A, that should only be one as well. And now here I'm starting to test whether or not these flights are actually valid. So let me try and get the airport1, airport2 in the flight. And I want to make sure that this is the flight that goes from airport1 to airport2 and has a duration of 100 because I might be trying to get some other flight. And then I'm going to assert the fact that F, this flight here, this should be a valid flight because it's going from airport1 to airport2, it has a duration of 100. That's a valid flight and I want to assert that that is in fact true. And so I'm testing that. Here, now let me test other things as well. So in addition to testing whether a flight is valid I want to test whether flights are correctly invalid. And so if there's an invalid destination where the destination is the same as the origin, that should not be a valid flight. So if I grab the flight whose origin is city one and whose destination is city one, if I run F.is_valid_flight well, that should be false because that flight should not be a valid flight. And likewise I can do a test for an invalid flight duration, that if I query for that flight that I created that has a duration of negative 100 and then assert that F.is_valid_flight is false, well, that should work because this flight that has a duration of negative 100 minutes is very clearly not a valid flight. And so when I run the is_valid_flight function then that should result in false. Questions about any of these tests or how they work? Yep. AUDIENCE: [INAUDIBLE]. BRIAN YU: Yep. AUDIENCE: [INAUDIBLE]? BRIAN YU: Good question. What is the .get doing? .get is just a simple way to query for a single thing. So normally I might try and query something by filtering to try and get only the things that are from CityA, for instance. And that might give me a query set of multiple different possible responses. But if I only care about getting one thing, if there's only one city that has city code AAA, then I can use .get to just say, get me the thing that has city code A. That only works, though, if there is a city with code AAA in the database. And the reason that's there is because of this setup function where I initially add a city here by creating a new airport that has code AAA. So if I were to do like .get equals CCC, for example, that wouldn't work because that does not already exist in the database. And so now that I've created these tests I can begin to use Django's built in testing infrastructure to actually run these tests. And this is quite powerful for a number of reasons. One, because it makes it very easy just to quickly run all of the tests that I've created across all of my different applications. But secondly because Django knows that my tests are likely going to involve manipulations of the database. They're going to involve me creating objects and testing them. And it would be probably not a good idea if any time I ran these tests, it would just actually run these commands on the database. Why? AUDIENCE: [INAUDIBLE]. BRIAN YU: Yeah. It's going to mess up the data in a number of different ways. And one, it's going to create these fake cities that aren't actual airports inside my database. But secondly if I ran the test once and then ran the test again, for instance, it would run through this whole set up function again, it would create a second city with code AAA and BBB. And that's going to really mess up the data that I have in my database. And so Django is smart enough to know that when we're running tests, go ahead and don't use the original database. Create a separate test database that we're going to use purely for the function of running these tests. And it's going to be blank and it will start from scratch. And that will allow us to just create these objects without needing to worry about what other data might already be in the database, without needing to worry about messing up the data that's already there. And just allow us to really test the things that we're interested in testing in a clean and isolated environment. So how do we actually run those tests? I'll go back here and I'll run Python manage.py and test. And what you'll see here is that it's first going to create that test database. It's creating a brand new database. It's going to be an empty database where I'll run all of these tests. And then you'll see a very similar interface to what we saw on unittest. It's really just running unittest on these tests because it gives me these five dots indicating five tests all successful. It ran those tests in 0.013 seconds. Now it's OK. Then it gets rid of the test database because presumably I no longer need it. And now we're all set. And so using that, I was able to very quickly, by running Python manage.py test, run all of the tests that I had previously created inside of this web application. Questions about anything so far? Yeah. AUDIENCE: [INAUDIBLE]. BRIAN YU: Good question, yeah. So we want this setup to only happen once before each test. Each test, though, is operating in isolation to each other test such that we never want it to be the case generally that what happens while running one test might influence the running of a second test. Because generally speaking, good tests are written independently, where one test is not going to influence something else. And if something goes wrong in one test it's not going to affect the second test. And so these are all operating independently. And they all first set themselves up and then run the test without ever interacting with each other. And so this set up code will happen one time for each one of these individual tests. But the setup code for one test is not going to influence the setting up of a second test, for instance. So what we've done now so far is create tests that are testing the database, really. We're testing the back end side of our web application involving the models that we have, the functions associated with those models, and making sure that all of that is working correctly. But that's not the only thing that we can test because if we think about the relationships between different things on our web application, if we think about it in Django's model view template format where we have a model that has all the data, our views.py file takes that information from the model and figures out what to put into the template, and then the template displays all the information that the user cares about. Well, we've tested the model side of things. But we really haven't yet tested the template side of things. And so what we'd like to do now is test the template side of things. Test not just the back end, but make sure that whatever information we're passing into our template, that we're able to test that, too. So let's take a look at that by looking at airline2. So we'll take a look at test.py. And in this case, notice we're doing a couple of things. In addition to importing TestCase, we're also importing Client. And what Client is going to do is simulate a web client, a client that is able to make requests and get responses back from the server such that we can simulate web requests to different pages and make sure that the information we get back is what we expect it to be. And so these first couple of tests are the same. But let's now take a look at test_index. And so what did the index function do, first of all? Let's make sure we remember. What our index function did, when we first go to the default flights page, is that it sets up some context information where we set flights equal to query for all of the flights. And then we take that flights information and we pass it into the index.html page. That way the index page, very simply, would just show a listing of all of the flights because of the context that we pass in, this context variable here, includes within it this flights key that's associated with a value that represents all of the flights that we have inside of that context. So how do I test to make sure the index function's working? That if I had two flights in my database, that when I pass it into the template I'm passing in both of those flights? Well, here's the test_index function, which is just one of these tests where the first thing I'm going to do is create a new web client-- I'm just going to call it C-- which is going to be able to make requests. And now I'm going to do C.get, send a get request to the slash route, that default route that should just load the default index page, and get back the response. And so a couple of things I want to do about that response. First thing I want to do is make sure that response was successful. Recall that when I make an HTTP request I get some sort of exit code where generally 200 means everything was OK. I want to first of all assert these two things are equal. I want to make sure that response.status_code, in other words the status code the came back from that web HTTP request, is equal to 200. That those two should be the same. And if, for some reason I try to access the index page and there was an error and I didn't get a 200 response, well then this test should fail. And the second thing that I want to check is I want to check the context of the response. Remember the context was that object that I passed into the template. And what Django allows us to do in order to make it easier to test our templates is it gives us direct access to the context when we're testing the response. So if we want to make sure that inside the context that came back there were two flights, because up above I created two flights, then what I'll do is say, check the response context, get at the flight's key, count them up and make sure that there are two of them. And so the reason this works is that if we compare it to the views.py over here, in views.py I define this context Python dictionary. And I pass that context dictionary into index.html. Then when I run the test, I'm able to actually access that context dictionary, get at the flights information in there, which is just going to be one of those Django query sets, count up the number of responses that came back from that, and make sure that that number is equal to two to verify that the index function is in fact working the way that I expect it to. So other things I can do here. I can test to make sure that if I go to a valid flight, then the flight page is going to work for me. That if I try and access a flight it will work. And if you recall, what does my flight function do in this views file? I try to get the flight in question. If it doesn't exist I raise a 404 error saying there was some sort of problem. This flight number doesn't exist. If I went to my URL/28 but there is no flight 28, I'm going to get a 404 error saying, sorry the flight doesn't exist, or at least I should. And then I'll render this context, rendering information about the flight, passengers on the flight, passengers that are not on the flight in case I want to register them for the flight, so on and so forth. So if I'm now testing to make sure that I can get a valid flight, well let me first grab a flight that goes from A1 to A1. So this is actually not a valid flight but I want to test to make sure that this flight page is going to work. Then what I'll do is get a web client, try to go to slash and then whatever this flight ID is, going to F.id, saving that response. And then, down here, asserting that the status code is equal to 200. In other words, if I go to slash some ID, where that ID is the ID of an actual flight, then this page should work. I should get a 200 status code as a response from it. On the other hand, what happens if I try and access a flight ID that doesn't exist? So here on line 60, I have an interesting query which is a slightly more complex Django query but I'll try to explain what's going on here. I want to get the maximum value of any ID of any flight. Because what I want to test in this function is, I want to test what happens if I go to an ID number that doesn't exist. If the IDs only go up to ID three or four, what happens if I go to /5? So the first thing I need to know is, what is the maximum ID of any of the flights in my database right now? And so to do that I'm going to go to flight.objects.all, query for all the possible flights. And Aggregate is a special Django function that helps to, as the name might suggest, aggregate data by some particular attribute. And in this case I'm aggregating it based on the maximum ID value. So I'm going to get the maximum ID value of any of the individual flights. And then when I actually do this test I'm going to get slash whatever the maximum ID is plus one. So I know that whatever route I'm trying to access now, it's not going to be a route that exists because of the maximum ID was flight number five, then when I access /6, as I'm doing here, then there is no way that this should be a valid flight which means that the status code that I get back as a response should be status code 404. I should get that 404 error if I go to a flight that doesn't exist. And so this function tests to make sure that is going to work. And then you can see I also I have a couple of other tests here as well. So here I'm testing to make sure that the passenger page is going to work correctly. So I first start by adding a passenger. And then I check to make sure that when I go to slash this particular flight, that if I count up the number of passengers that I have that one passenger, for instance, on that flight. And I also do the same sort of test to make sure that non-passengers work. So if I have a new passenger but I don't register that passenger for a given flight, then when I try to check the number of non-passengers on this flight, that should be one. And so what I'm doing here is by accessing the response context and getting at particular aspects of the response context, I can effectively test what information is getting passed to my template, test what information actually gets displayed to the user when they go to a particular page. Which means I can now automatically run these sorts of tests to make sure that my page is going to behave the way that it should behave such that later on down the line if I'm making changes to the flight's page and I change different things about what information is in the context. And I want to make sure that this original functionality of displaying the passengers, displaying the people that are not passengers, is still working. I can just use tests like this in order to make that happen. Such that now if I go back and run Python manage.py test, now I just ran all 10 of those tests. Everything was OK. And so now I can feel pretty confident that that is, in fact, working. If I mess something up, though, for instance, if in here rather than passengers being all the passengers and non-passengers being only the passengers that are not on this flight excluding the people that are on this flight. If I accidentally switch them up, for instance, and this was my non-passengers and this was my passengers, for instance. And now I try to run that. Well now I'm going to get the fact that I got two failed tests. I missed a couple of assertions. And here were the tests that I failed. I failed the test flight page in non-passengers. And I also failed the test flight page passenger's test. And then I can go back and figure out why is it that exactly I failed those tests? I can fix those individual errors. I can go back. I can run the test again. And now I can see that I passed those individual tests. And so writing tests in advance and then running those tests when you make changes is often a good way to verify that your code is, in fact, working. That you haven't made a change to one part of the code and therefore messed up some other part of your application. And that can be good for making sure the code is well integrated, that you're avoiding bugs wherever possible. Questions about things so far? OK, so thus far we've seen a bunch of different ways to run tests. We saw just running sample assertions. We took a look at unittest. And unittest has a bunch of other functions that we can use in order to assert the things that are true are not true. We saw assert true, assert false, we saw assert equal. There's also assert not equal if two things are not the same, assert that some item is in a list, for instance, or not in a particular list, and a whole bunch of others that you can explore. And so unittest is useful for quickly running a whole bunch of functions that are each going to test a particular aspect of our program. And then we saw how Django was able to expand and build upon those basic functions in order to allow us to test individual routes, or test things about our database, or test the templates that come back and looking at the context of the information in those templates to make sure that those work too. And I'll just show you one other example of testing in order to give you a sense for the different types of testing that's available. What about something like a JavaScript application? So if I had a JavaScript application like counter.html, which is a simple JavaScript application similar to things we've seen before that just has a number and I have an Add button that increases the number and a Minus button that decreases the number where I can just press the plus button over and over and over and the number goes up and up and up and up, and then the minus button decreases that number. How do I test to make sure that's accurate? I can't just use like the Django framework that I was using before, because it's not just a matter of request this page and see what the response is. Because the response will just be the number zero. And I can't use that to say, test the plus and minus button because that's not something about Django's server. It's about something in the web browser. It's about the JavaScript code that's really running in the web browser. So how do I test that? So there's an entire set of testing tools known as browser testing tools that are designed for really testing things happening inside of the web browser. And one example of that is Selenium, which is the one that I'm going to show you know. And what Selenium does is it uses a web driver to allow us to use Python code to effectively control what's going on in a web browser, to pretend to be the user programmatically, in order to manipulate different things about the page. And so how might that work? Well, I'm going to go ahead and from Selenium import web driver. And I'm also going to import from this test file a function called file URI. What file URI is going to do is it just takes an HTML file and turns it into the URL that I would actually want to use if I were to open something like this up. So for instance, if I take this URL and open it in Chrome, I get to the page. This is just a file URL that now takes me directly to the page that is located on my system. And so that's all the file URI function is doing. But I'll go ahead and save that inside of a variable called URI. And now what I'm going to do is create a web driver by going webdriver.chrome. So Selenium has a whole bunch of built in web drivers for different browsers. Chrome just happens to be the one that I'm going to use now. And what effectively this is going to allow me to do is control Google Chrome through the usage of Selenium's Python API. And so what I see is this Google Chrome window that now pops up that says, Chrome is being controlled by automated test software. I've installed something called Chrome driver inside of Chrome that lets me now run these automated tests. And so now if I say, well, URI is the file that I want to open. So if I say driver.get URI, well that's going to, in here, open up the page that I wanted it to open up. And so now if I want a test, like I want to test these individual buttons, the plus and minus buttons, how do I do that? Well inside of counter.html, if I took a closer look at it, and I-- let's go ahead and open up. Let's look at counter.HTML. Inside the HTML I have these two buttons, Plus and Minus. The Plus button has an ID of increase. And so if I want to get at the Plus button, well that's as simple as me now just saying, driver-- or lets me call the variable plus. I'll say plus equals driver.find_element_by_ID increase. So I want to get the thing that has ID called increase and save that as Plus. And so now I can say, plus.click, for example. I've extracted the element that has ID increase. And now I can programmatically, in Python, using Selenium, say plus.click to click on the plus button. And ideally that should have had the result of causing the number to increase. It looks like that didn't quite work in this case. What is-- plus is this element. And I wanted to say plus.click. No elements CLS. So when I-- quick question, how does it know that it's a button? When I'm extracting a web element, it knows information about the web element that I'm attempting to extract. So in this case it does know the fact that it is a button. Let me give that another try. So we'll go from Selenium, import webdriver, and we'll say from tests import file_uri. So URI equals file_uri of counter.html. And now driver equals webdriver.chrome. And now I have this Chrome web driver. And I wanted to say driver.get this particular URI. And now plus equals driver.find_element_by_id increase. And now plus.click. All right, not sure what happened last time but this seems to be working now. So now I found the element that has ID increase, saved it inside of a variable called plus. And now I can actually manipulate that button by doing something like plus.click, for instance. And I do it again. And that-- didn't seem to have the same effect. So maybe there's some sort of bug in the web application that I can look for. But the idea here is that we can begin to take advantage of the interface that Selenium gives us in order to manipulate the page and make sure that it's working the way that we want it to. And so let me show you an example of those tests in action. And so one thing I might want to do is test different aspects about the page. So I might write a whole bunch of unit tests where I first try to get the counter.html file and then assert that the title of the page that I got is equal to counter, make sure that the title is working. And likewise I might test the increase button by first getting the counter.html file, finding the increase button, clicking on the increase button, and then making sure that when I get the H1 tag, that heading at the top and check the text, that the text should be one-- that that should have the result of updating the counter. And likewise, I would want to do the same thing for the decrease button-- make sure that when I click the decrease button, it has the effect of subtracting one. And then down below, I might test that the increase button works multiple times, which seems like maybe it didn't based on the opening of the file just then, such that if I get the increase button and click on it three times, then the result of that should be that if I check the contents of that heading then that should be three. And so what Selenium allows us to do is it allows us to do some of this browser type testing where I can really begin to test whether or not things are working in the web browser in response to things that users are actually doing. That if they click on something and that has the effect of something changing on the page, that's not something that's very easily testable by Django's testing software, for instance. But it is something that I can do by taking advantage of browser testing. The goal of all that was to show you a bunch of different types of tests that we can perform, whether it's testing functions just by using asserts, using unittest, using Django's framework to test databases and templates and contexts, or using Selenium to test user interfaces and testing the way that users actually interact with the web browser. All of that is going to be very helpful as we begin to build larger and more sophisticated web applications. And what we'll see after the break is how we can actually implement some of these tests and use some of these tests in our own building and deployment process for web applications as we take a look at CI and CD-- Continuous Integration and Continuous Delivery. We'll take a look at that after the break. We'll take a short break now. OK, welcome back. So where we left off was talking about testing and how we can write tests in order to make sure that different components of our application work and then run those tests in order to make sure that anytime we are making a change to one part of our web application we're not inadvertently causing some other part of the web application to stop working, such that if we're writing tests comprehensively and thoroughly across all different parts of our application it becomes pretty easy to make sure and verify the different components and aspects of our application are working. And so this is ultimately just part of good software development workflow. And that's what we're going to be spending some time with the rest of today talking about, which are just good software development design practices and goals that we should bear in mind as we go about designing software in the modern era. And one thing you'll hear commonly when we deal with this topic is CI/CD, which stands for Continuous Integration and Continuous Delivery. So what does that actually mean? Well, continuous integration has to do with consistently and frequently integrating code together between different people working on the same project, for instance. Where in an old model you might imagine that once upon a time there would be one central place where all the main code was. And different developers working on the same project might be working independently, separately on different features. And then only at the end would they then try to integrate their code all back together with the main original branch in order to make sure that everything's working. But that leads to a lot of different integration problems where it's not necessarily immediately clear that when you try to integrate all the things together that it's just suddenly going to work. And so what continuous integration is all about is frequently merging different changes back to some main branch that either you are working on your own project or that multiple different people, if you're working in a group, are working on together in order to make sure that changes that you make are continuously integrated. And using software version control tools like Git, this is actually quite easy. And a second aspect to continuous integration, which is sort of added onto this, is that any time a change is made and we want to integrate it in with the rest of the codebase, we'd like to verify that that change is actually going to work. In other words, it's not going to break anything. It's not going to cause any of the existing parts of the codebase to fail. Because if you have one software developer who's working on feature A in your web application, they might not be thinking about features B, C, and D. And when they integrate feature A into the branch, for all they know it might be causing some problems to happen in other parts of the web application. So one important part of continuous integration is going to be automated unit testing. This idea that any time we make some change to the web application and want to integrate it with the branch on which we're working on developing this application, we automatically run some set of tests and then we get some result. And any time I push code to GitHub, for instance, I'd like to know whether that code I just pushed actually passed the test or not by automatically running those tests every time I push code so that I don't have to worry about doing it myself and I might forget to do it, for instance, in order to really just verify that this is going to work. Continuous delivery, on the other hand, take this one step further. In addition to just continually integrating code into the code base consistently we also want to continuously deliver code to our ultimate web application. So as opposed to making a whole bunch of changes and only after a couple of months of building up changes to release of those changes in one go, continuous delivery is all about making incremental deliveries to our web application. That if we make one small modification and push that modification and it passes all of those tests, then let's go ahead and make it easy to then deploy that small change to our web applications so people can see those changes right away. And so these are growing in popularity as practices when it comes to software development. And we'll be looking at some of the tools that help to make this a little bit easier today, which are tools that you might see an industry quite frequently working on larger software development projects like this. And so there's many different CI tools for continuous integration who are meant to serve the purpose of helping to quickly integrate code and to run tests on code. And you can explore a bunch of them. They all do very similar things but have their slight differences and advantages. The one we're going to be focusing on in this class is Travis, which just happens to be one of the more popular CI tools. And the basic way that Travis is going to work-- Travis is the little guy in the lower right of the screen there-- is that when I push code to GitHub, when I've made changes to a web application and I push that code to GitHub by pushing some new change to my repository, what GitHub is going to do is GitHub is going to notify Travis, our CI software, that we've made some changes to our repository, that there has been a new commit, a new push to the repository, and to let Travis know that there has been this new change. And what Travis is then going to do is it's going to pull that code from GitHub and then run some tests on it. Travis can do other things as well and we'll touch on that as well. But one thing that it's quite good for is for running tests. If Travis knows that there's been a change that's been pushed to GitHub, and Travis can then say all right, let's pull that new version of the code, the latest version of the repository, run the tests, make sure that the tests are all passing. And either way no matter what the results of the test are let GitHub know the results such that on GitHub I can now see, did the test pass? Did the test fail? And that can tell me something about whether or not the change that I just made needs to be looked at again or needs to be reworked, for instance. And so before we actually look at specifics as to how that works, questions about the general idea of what we're trying to achieve here? Yeah. AUDIENCE: [INAUDIBLE] into the branch [INAUDIBLE]?? BRIAN YU: Good question. Question is about how you're merging and what branches we're merging into. Different web applications will have different practices for how this generally works. But frequently there will be one branch on which the latest version of the project being developed is on. And as people work on individual features, they'll work on those features individually on different branches for each individual feature such that if you make a new change to a feature that gets pushed on to that feature branch. And only once you're done with that feature, or at a place where it can be merged in with the rest of the code and it's working, then you'll want to merge that back in with the rest of the code. But what Travis can do is it can run these tests no matter what branch you're on. So even if you're not on the main branch, it can test your code, make sure that it's going to work such that you can be confident before you merge back in to the main branch on which all of the code is. Because you want to know that when you merge it in that those tests are going to pass, that you're not breaking anything when you go about making that change. OK, so how does Travis work? How are we going to be able to take advantage and use this? Well we're going to need some sort of configuration file to tell Travis what to perform, what tests to run, how to install all the things that need to be installed, for instance. And to do that we're going to use a file format called YAML. YAML is just a common file format that's used for creating configuration files usually-- files that are configuring some service to behave in a particular way. We'll see it for Travis now. We'll see it again later on today in a different context. But the general idea for YAML is that a file in YAML is going to look something like this. It's going to be a set of keys and values, just like in an JSON object, for instance, which we saw before. There were keys and values where we had key colon value. Same general idea here-- key colon value. And if we had a list of items, for instance we had one key associated with multiple values, it would look something like this, where a list is indented and has a hyphen in front of it. And so that would be what a YAML file would look like. And what a Travis YAML file is going to look like, configuration for Travis, our CI service, is it's going to look something like this. This is going to be the configuration file telling Travis what to do in order to test our code to make sure that it's working properly. So we'll first specify what language we should be using, in this case Python because our application is written in Python. We specify what version of Python we want to run, 3.6. We are going to specify what command or commands to run in order to install the things necessary before we actually run our tests. And so in this case, if we had all of our dependencies and requirements stored inside of a file like requirements.txt, for instance Django would be an example of a requirement that we would need to install, we would include a line like pip install -4 requirements.txt to say, go through all the lines in requirements.txt and install all of those dependencies, because Travis by default is not going to have Django installed, for instance. So if we want to run our Django tests, then we need to tell Travis that you need to install Django first before we can actually run any of that. And then finally here we're specifying what script we want to run. In other words, what should we actually do in order to run these tests? In this case run Python manage.py test. Same command we ran in Django ourselves in order to run the tests but by putting it inside of the Travis YAML file, we're going to have Travis run those tests for us. And so let's see what that actually looks like. So what I have here is airline3, inside of which is this .travis.yml file, which is exactly the contents of what was inside of the slide before, specifying the language, specifying what version of Python, saying install all of the dependencies first. And then after that, go ahead and run Django's built in test. This is what I want Travis to do any time I make a change. So how do I actually configure Travis and set it up? Well if I go to GitHub, you'll notice that I have on my profile this repository called Airline which is going to be the repository where I store my code for this airline application. If I go to TravisCI.org now, which is where Travis lives, and I go to my profile, I'll see my repository here. Here is my repository, Airline. And I'd like to enable Travis for this repository. I want to start using Travis on this repository. So after I've linked it up with my GitHub account, which is fairly straightforward to do, I'll go ahead and enable that. And what that's going to do is set it up so that any time I make a commit or a push to GitHub, it's going to notify Travis And GitHub does this through web hooks, where GitHub has this built in idea of hooks on particular events on my repository where I can say, anytime someone pushes to a repository, GitHub should know to do this. And so in this case I'm saying, anytime I push to my Airline repository, GitHub should notify Travis, the CI software that I'm using, and let it know that there's been some new change that you should run these new tests. And so now on the command line I'll go ahead and add .travis.yml. Because I want to now include this file inside of my repository. I'll go ahead and commit and say I've added .travis.yml. And I'll push those changes. So now I've pushed those changes to GitHub. And if I go here and go to this repository page, what we should see is this build. So a build is just Travis's way of saying it's detected the fact that I've made a new commit, here's my commit, this add .travis-yml. And now it's running a whole bunch of commands in order to actually run these tests. So it's first just doing some set up. Here you can see that it's actually cloning the repository by saying get clone, for instance. And now it's checking what version of Python I have. I have Python version 3.6. The next thing it's going to do is right here, it's running that command that we told it to run, pip install -r requirements.txt, to say install all of those requirements. And now after it's done installing those requirements it's going to run Python manage.py test. That line of code that I told Travis to run in order to say, I want to run all these tests now and make sure that they actually pass. I didn't have to do this myself. Travis is running these tests automatically as soon as I make a push to the repository. And so what I see here is, OK, I failed a couple of tests. So I failed the test flight page non-passengers test and the test flight page passengers test, in both cases because zero didn't equal one. And it told me I ran 10 tests and that two of them failed. And therefore my build exited with status code one. And so remember these exit codes that we talked about before, where an exit code of zero generally means everything was fine. Exit code of one, or some other non-zero number, means something went wrong. If I run Python manage.py test and I pass all of the tests, that command is going to return with exit code zero meaning everything is fine. And Travis will see that exit code and say, all right, exit code zero. Everything must be fine. We can therefore say we've passed this build. This was all good. Whereas on the other hand, in this case, if I failed some of my tests for instance, and therefore Python manage.py test exited with a status code, an exit code, that was not zero, like one for instance, then Travis sees that. It notices that something went wrong. And therefore it reports that this build has failed. So if I scroll up to the top, I see this red X. That means this build failed. I didn't actually pass these tests. And if I go back to GitHub and actually click on this commit history, I can see that on this commit, this commit that I just added, this add .travis.yml, on the right side of it now I see that red X that actually indicates to me on GitHub that I did not pass all of the tests. That Travis tried to run the tests and this build failed and therefore it's reporting to me the fact that this build failed. And we can use that information to our benefit when it comes to things like pull requests and merging between branches where you can start to impose constraints in GitHub and say, in order to be able to merge from whatever branch I'm working on to the original branch, this better not be an X. I better have passed all of the tests before I actually go about merging whatever change I made back into the master branch. And so what Travis is doing for me is automating the process of running these tests such that now I notice, OK, it looks like there was something wrong with my code. So I can go back to my code now and say, OK, if the problem was with my flight page I can check here. What seems to have been the problem? Well, it looks like I screwed up which one was passengers and which one was non-passenger. Just a mistake between which key was which. This one should be passengers, this one should be non-passengers. And now if I go ahead and add those files and say fix passenger list bug and push those changes to GitHub. And now I check GitHub's commit history. So far I don't see anything there because the build's not done. You'll notice that changes to this circle to indicate the fact that the build is currently running, that it doesn't know yet whether or not I've passed or failed those individual tests. But if I check back here I can see that now Travis has detected the fact that I've made this new push, that I've tried to fix the passenger list bug. And now it's going to go ahead and, again, run all the requirements and redo that exact same set of steps that it did before. It's automating the process of building my code, running my tests, and making sure that they pass. In this case, it looks like it ran all 10 tests. Nothing seems to have gone wrong. It says OK. Manage.py test exited with zero, so now we're done. And if we go back and check this commit history, now we see next to each one of these changes, this add .travis.yml commit that I made, something went wrong. I didn't pass the test here. But now here when I tried to actually fix the passenger list bug, then that did in fact pass. So all is well in that case now. And so what that's ultimately allowing us to do is automating this process. Using continuous integration to say, any time I make a change, let's just push it to some branch. Maybe it's not going to be the master branch. Maybe it's going to be some other branch. But then let's run the test suit on it. Make sure that the tests are all passing in order to make sure that everything is in order before I try and merge things together. With the goal being such that when I do merge things together, I don't run into problems when I try and integrate everything all together, that I've been checking all along with every change that I make, am I going to break anything? Am I going to break anything? If I do I have the opportunity to fix it. And if not, then I can feel confident at the end that as long as my tests are thorough and comprehensive, that my change is not going to cause any problems with the rest of my web application. Questions about any of that so far? About what Travis or other continuous integration tools are used for? Travis certainly isn't the only one. Just happens to be one of many. But they're all with the ultimate goal of doing things like this, of automating the process of integrating code together. And they can be used for other things as well. Travis can be used in order to make sure that we are sending notifications, for instance, such that Travis can notify a slack channel, for instance, to say that this succeeded or this failed. It can also be used for automatically deploying an app. So if you're using Amazon AWS in order to deploy your application to a server somewhere, Travis can take care of that for you where it can say if all of the tests passed, then let's go ahead and deploy this application to the internet and automatically deliver it. And if the tests didn't pass, then let's not do that. Let's not actually push those changes because we want to make sure the tests are passing before we go ahead and make those changes. Questions about any of that? OK so what other issues might we run into when we're thinking about developing software, about building applications, deploying applications to the internet? Well, one thing that you may have already come across is just making sure the application even runs in different environments. If I took one web application that I've just written and put it onto a different computer and then try to run that Django application, there is no guarantee that that application is going to behave in the same way. Maybe the computer that I put it on doesn't even have Django installed, for instance. And so I'm going to get an error if I try and run the server because Django doesn't exist on the computer that I'm using. Or maybe it's using an older version of Django whereby as a result some of the newer features of Django that I'm using aren't going to work on the older computer. And so in web application development you might run into all sorts of these problems where the computer you're running things on, or the computer you're developing locally on, might be different from the computer where you're actually running the application in production, where things are actually running that users are really using. And as a result you might run into all these sorts of compatibility errors between your machine and some other machine because there is no guarantee that different machines are behaving in the same way. When really we would like to make sure that when I'm running my application it's going to be working in the same environment the whole time. That if I have these dependencies and libraries installed on my application on my computer, then someone else running my application on a different computer will have all of those same things installed, will have the same set up ready to go without any of those issues. So how might we go about achieving that? What are some strategies we could use to make sure that the way I'm developing my application is going to be in the same sort of environment to someone else? Multiple possible things you could throw out here as ideas. Yeah? AUDIENCE: Virtual machine. BRIAN YU: A virtual machine, yeah. And for a long time, a virtual machine was really the best way to do something like this. That you would, in a virtual machine you would set up a virtual operating system, the guest operating system that was operating on top of your existing operating system where you could virtually run your own computer that had whatever configuration settings you wanted it to. And you could therefore say, this virtual machine is going to have these dependencies installed on it. It's going to be running this operating system. And so when I run my application on that virtual machine then I can be sure that if I'm running it on this particular virtual machine then everything is going to behave exactly as expected because the machines are identical. We're running the same virtual machine on my own computer and on wherever I'm trying to run the application. What might be some drawbacks of the virtual machine that we can think of, for running a web application like a Django web application on the internet? Yeah? AUDIENCE: [INAUDIBLE]. BRIAN YU: It's slower, yeah. It tends to be pretty slow because I need to start up this entire virtual machine with an entire operating system of its own, with its entire any libraries that are necessary for the operating system. And generally that can be overkill if all I want to do is run a web application, for instance run a simple Django application that just needs to have certain dependencies installed. And so one solution that people have come up with in order to deal with that problem is containerization. The idea of creating containers that are not quite virtual machines in the sense that they're not fully blown operating systems on top of which are all these additional layers but are just isolated containers that have just the things we want to have installed on them. Such that if we define what should go into a particular container inside of what we'll call an image, we can make sure that the image that I'm using on my computer is the same as the image that's being used on the server that my application is eventually running on. And therefore I can be confident that whatever environment I'm running my application on my local computer will be the same for anyone else who's also running the application as well. And so one of the most popular tools that I'll introduce now for containerization is called Docker. And the way that Docker works is-- we can use this as an example by comparing Docker to a virtual machine. So on a virtual machine we have the original computer on which you're running your own operating system. And on top of that, for each application you would run, you'd have a virtual machine which has a guest operating system on it on top of which are all the libraries that you might need. And then the application is running on top of that. Whereas the way Doctor is going to work is that Docker is just going to build directly on top of the operating system on which you're running. It's going to have direct access to the operating system that your computer is running on. And it's just going to add an additional layer that helps to keep these containers isolated from each other so that each container can still have its own libraries and dependencies. I can say, inside of this container I want to install version two of Django, for instance, and then run my application on top of that container. So it's a slightly different model but it's still trying to get at the idea of, separate out our applications into these containers where each container has just the things that it needs to have installed in order to run that application in such a way that it's consistent. Such that so long as I know what the instructions are for creating one of these containers, then when I need to run my application somewhere else I can just create one of these containers and then run the application. And this will also be useful when it comes towards applications that have multiple different parts. So for instance in project one where you were using a PostgreSQL database and a Flask web application, there were sort of two different parts to that web application. We had the Flask application itself. And then we also had your database running in some separate server elsewhere. And what Docker is going to let us do, and what we'll see in a moment, is let us take all of those different services and compose them together, so to speak, in such a way that so long as you have the instructions for how do you start up the web application server? How do you start up the database server? It can do all of those things on its own such that you don't need to worry about whether or not you have all the services connected in the right way. So we'll take a look at some concrete examples of that by taking our application for airlines and Dockerizing it by using Docker images and containers in a moment to explore how we can actually take advantage of Docker to make it easier, ultimately, to deploy our application. The goal of all of this is going to be to make it easier to test and deploy our application in such a way that everything is all in one place, that we don't need to worry too much about compatibility issues that might arise. So let's take a look at airline4 now. And we'll open that up. And so the first thing that you'll notice that's different about airline4 is that I have this file called Docker file inside of my repository. And so Docker file is going to be the file that's going to define a Docker image, where a Docker image you can think of it as just the instructions for how I'm going to create one of these containers that my application is going to live in. So here are the instructions inside this Docker file for how to create a container that I want to run my web application in. So what I'm going to say is, from Python:3 it's just saying, from is going to tell me sort of like inheriting from some existing Docker image. There's an existing Docker image called Python 3 that's just a default image for creating a container that runs Python 3 applications. And I'm going to use that. I'm going to set my working directory, the directory that I'm in, to a user source app. So this app directory is just going to be the directory where I'm running my application. I could have used some other directory but this just happens to be a convention to put everything inside of this app directory. And now what do I want to do to make sure that whenever I'm running this application I'm always going to be running it in the same environment so I have the necessary dependencies installed? Well the first thing I wanted to do was make sure that I've got Python 3 installed because I want to be using Python 3 for this application. But I also want to make sure that my requirements are installed. That if I need Django installed in order to run this Django application, then I better first add requirements.txt to my app directory. And then I'm going to run pip install -r requirements.txt, to say go ahead and run the installation of those requirements. Such that now if I were to ever create a Docker container based on this Docker file what that's going to do is install all the requirements. Where if I look at my requirements file I have a Django version two and also a Python library that's going to be useful for dealing with PostgreSQL if I have a Postgres database, for instance. And finally, the last step is going to be add all the contents of my current directory, dot meaning the current directory, to that app directory as well such that all of my Django files-- my application files, my settings files, URLs and views and templates and all that-- gets added to my application directory. And so that is going to be how I define-- this is going to help me define my Docker image such that when I create a container it's going to run all of these instructions, first installing those requirements and then adding the entire contents of my application to this app directory inside of my container. So how do I then say what my application needs to do? So let me first show you settings.py. So settings.py is one of those files we took a look at in Django before that lets us define various settings for our application. Last week inside of settings.py we specified that our database would be SQLite, which is a simple database and very easy to use when testing. But in reality if you were to develop a web application that was going to be used by a large number of users, SQLite's probably not the best one to use. You'll probably want to use a database that will scale a little bit better. And for instance, Postgres is a good one for that, which is what we used in project one. And so what we might do is change our database, for instance, to instead of using SQLite as our database that we defined in settings.py, we might instead specify inside of our settings file that we want to use PostgreSQL instead. So instead of engine, which with SQLite before, now I'm going to specify Postgres. And I can use name and user, Postgres is just the name and user name of the database. It might be something else. But in this case, we use Postgres. And specifying the port on which that database is run. And so depending on where your database is, these might be a little bit different. But you'll see how this information is going to be useful now because what I want to do, now, is tell Docker that when I start up this application there are a couple of different things that I want to do. On one hand I want to start up the database. I want to make sure that my Postgres databases is up and running, because my Django application is going to need to talk to it. Secondly, I'm also going to want to actually start up the web application itself because I want to make sure Django is up and running, that that server is up and running for when I want to interact with it. And finally, there's actually a third thing that I might want to do which is I might want to, anytime I run my application, to automatically run any of the migrations that I need to apply. Such that if I bring my web application to some new place where it's got some new database, I want to make sure that any of those migrations that I had-- migrations being ways of applying changes to the existing tables that existed inside my web application. I want those migrations to be applied to my new database. And so I would like it for whenever my application starts up, go ahead and migrate everything over, any changes I've made to those tables, such that I make sure that my web application is running on the latest version of my database. So those are all of the different services that I want running when I run my Python application. And the way to make all of that come together is by using what's called Docker Compose. And so DockerCompose.yml will be the last new file that we'll look at today. Which is going to be a YAML file in the same file format as before, which is going to define all of the different services that make up my web application. And so the first service that I'm going to have is a database. And I need to specify what Docker image should I use in order to construct this database. We'll go ahead and use Postgres, which is just one of the default ones for just creating a Postgres database. In addition to the database, I also want to set up a service for migrating everything over such that when I start up an application, whether it's on my local computer or on a server somewhere, I automatically migrate any changes. And so build dot means I'm going to build based on the Docker file in my current directory, because I have this Docker file here that's going to tell me how to install all the requirements, for instance. The command is going to be what command to actually run to make the service happen. And so I'm going to run Python manage.py migrate in order to run all of those migrations because that's what I want to happen every time I start up the application. Volumes helps to link between different files. So I'm saying, link between the current directory that I'm in with this app directory, which is where in the Docker file I said my container was running. And then depends_on is saying, what should be done first before I actually work on this service? And in particular, I need this DB service running, this database service running, which is my Postgres database to be working before I do my migration because I want to make sure the database is there before I try and migrate anything there. So now I've set up the database. Now I've set up the migration such that when I start up my application using Docker it's going to create the database as its own container, then migrate any changes that need to be made. And now let's add another service just called Web, which is going to be the one that's actually running the web application. So again that's going to be based on the Docker file in my current directory. What command am I going to run? I'm just going to run that same Python manage.py run server. And specifying what IP address and port I want it to run on, just setting that explicitly though you might not necessarily need to do that. And then the only different things here are what's happening here in ports. What this is saying is, my container has its own set of ports where things are running on. And what this is saying is, take port 8,000 that is running inside of my container and map that to port 8,000 that's running on my own computers. So that if I, on my computer, go to this URL I can actually access the web application that's running inside of that container. And then finally, I'm saying this depends on two things. It depends on the database being there. I want to make sure the database is there first. And I also want to make sure that the migration was successful. So make sure the database is there, make sure that I've migrated everything over. And once that's done, then I can start up this web application by running the server. And so this is a lot of new syntax. The general idea-- don't worry about memorizing this-- is just getting a sense for what Docker is designed to let us do. Where the idea is that it's a way to allow us to easily create containers that have all of the requirements and dependencies that we need by just automatically installing whatever requirements we have, making sure we have the right version of Python installed, and then, in Docker Compose, defining the different parts that make up my web application-- defining the database that I have, any migrations that I might need to perform, and the web application that I actually want to start up. And so now in order to start up my application, I can run something like Docker Compose up, meaning start up my web application using Docker Compose. It's going to look for that DockerCompose.yml file. And what you'll see is the first thing that it's going to do is start up the database. Then it's going to start up the migration. Then it's going to actually start up the web server. And now I can go to 0.0.0.800, and now I see that I have this interface where right now I don't have any flights but now I'm running my web application using Docker. And so all I've done so far is just start up the web application, which doesn't seem like I've gained a whole lot. But what I have gained is, I've gained at these two files, the Docker file and the Docker Compose file, that define exactly how to start up this web application. And this is using a Postgres database which I didn't need to specifically download Postgres and install it and figure out how to use it. Docker is taking care of that for me by just creating a container that's running a Postgres database. And what this means now is that I can give this repository to anyone. And so long as they have Docker installed they can run this web application and expect it to behave in exactly the same way. And so this is very useful where, in web applications, it can often be a challenge to make sure that the way an application is behaving on your computer will be the same as the way the application is behaving on different computer. By taking advantage of containerization, we can A, simplify the process of combining these services together, but B, simplify the process of making sure that the application is going to behave the same way across different places as well. Questions about what Docker is, what it's used for, the general ideas here? Yeah. AUDIENCE: [INAUDIBLE]? BRIAN YU: Good question. So who is actually going to be using Docker? So, if your project uses Docker any developer working on the same project is likely also going to be using Docker. Whereby instead of running Python manage.py run server, which is going to run a server based on whatever libraries you have installed on your computer, you will instead use something like Docker Compose up to start up the Docker container and then run things inside the Docker container. And that way if there's something missing from your Docker configuration, if you're missing a dependency for instance, it won't run in Docker and you'll know that you need to add added dependency, for example, to requirements.txt. And once you do, then it will behave the same way on someone else's machine. Then on the deployment side, all sorts of application Deployment Services now are able to use a Docker image and build the application based on that Docker file. Such that some common ones include Heroku, for instance, is a common service for deploying applications to the internet. Amazon AWS is also a very popular service for taking web applications and deploying them. And normally, you might have to worry. Do the servers that Amazon's running things on have the same version of all the software that I'm running things on? There are ways now with Amazon AWS to say, let me just give Amazon AWS the Docker configuration for my application and now Amazon can use Docker to ensure that when they start up the application on their servers, that it's going to behave in exactly the same way because we defined exactly what should be inside the container. And therefore we can be confident that that behavior will be emulated when the application is run on the server. Yeah. AUDIENCE: [INAUDIBLE]? BRIAN YU: Great question. Are all of the services running in the same Docker container? They're not. So in fact, if I go ahead and open up a new tab and I run Docker ps, that's going to list for me all of the Docker containers that I currently have. And I'll just shrink things down so it's all on one line. And what you'll notice is that I have two containers that are up right now. Here's one container that's based on the airline4 web image, which is the image based on the Docker file I showed you a moment ago. And then I also have this Postgres image, which is going to be the database that I have running. So I have two different containers, one that's running the actual Django web application and one that's running the Postgres server. But they're able to communicate with each other. And if I wanted to actually get inside of one of the Docker containers in order to run commands in it, for instance, that would look something like this. I could say, Docker exec-- meaning run a command-- -it to say I want to be interactively working in a terminal. Paste in the container ID of the container that I want to get into. And then I want to run bash, just run a shell. And now you'll see that I'm inside of the container now. I'm inside of the container, inside of this user source app directory. And now I can do all the same things that I could do before, where if I want to go into Python manage.py shell, I'm now inside of the shell of the Django application that's running inside of this container. And so if you ever need to get into a container in order to run commands in order to make modifications so on and so forth, you can do something like that in order to get access to it as well. But the long story short answer to your question is that each of these services are running separately in its own container. AUDIENCE: [INAUDIBLE]? BRIAN YU: Good question. If I was doing development, would I want to go into the container to do it? I don't need to get into the container in order to use the web application, just to go to the 0.00:8000, because I've linked these ports together. But I might want to go into the container if I needed to modify the database, for instance. If I wanted to manually add a new airport or flight, for instance, I couldn't just say, inside of my airline4 directory now, I couldn't just say Python manage.py shell and go there because this is going to be the shell for the Django application that's running on my computer which is different from the database, for instance, that I have running inside the Docker container. So if I wanted to go into the Docker container in order to modify things inside the database, then I would need to first enter that container. And then I could modify things that are specifically happening within that container. AUDIENCE: So in this case, you have a full version of the [INAUDIBLE]?? BRIAN YU: Good question, yes. So the question is, I have a full version of Postgres contained inside the container? And yeah, inside of this Postgres container here, this one, I have an entire Postgres server running and storing the data and Docker knows where things are located and just abstracts all that for us. We don't need to worry about where exactly this is happening. But it's effectively just the same as, whereas in Heroku when we set up a database there we were able to access it because it was its own database server that was running. This is the same thing. It's just a database that just happens to be running in a container that Docker is running for me on my own computer. And because it's following these strict instructions for how to create the container based on this image, then when someone else tries to create the same container they'll be in the same environment and everything will work the same way. OK, so all of these tools of testing, and Travis, and other CI software and Docker and such, are designed to be part of this CI/CD pipeline. Of making it easy to take changes you make to your code, integrate them together frequently in order to run automated tests on them to make sure that they're working, make sure that when you are running those automated tests that they're running on the same environment. Travis also has ways of integrating with Docker, for instance, so that when you're running tests on Travis you're making sure that the application running on Travis is running in the same environment as the environment that you're developing on, or the environment that your application is eventually running on. And using Docker, that makes this process a whole lot easier by making sure we have consistent environments for when we're running tests or when we deploy an application to the internet. And these are all just good software design practices that come in handy for making sure that as we begin to work on larger, more complicated web applications, that we have the tools in place to make sure that we're avoiding bugs, that we're making sure that our application works at every stage in the process, and that the deployment process is clean and ultimately efficient. So we'll wrap up there for today. Thank you so much. And we'll have some special guests for you next week as well. Thank you.