[MUSIC PLAYING] BRIAN YU: OK. Welcome back, everyone, to web programming with Python and JavaScript. And now, at this point, we've seen a number of different techniques and tools that we can use in order to design web applications-- HTML and CSS, to describe how it is that our pages look; a programming language, like Python, using a framework like Django, in order to listen for requests, process them and provide some sort of response. And then more recently, we took a look at JavaScript, another programming language that we can use in particular on the client side, running inside of the user's web browser, in order to make our web pages even more interactive and user-friendly. 

Now what we'll transition to today is taking a look at some of software's best practices, some tools and techniques that developers actually use when they're working on web applications, especially as those web applications start to grow larger. In particular, we'll start by discussing testing, this idea of verifying that our code is correct, and then transition to CI/CD, short for Continuous Integration and Continuous Delivery, some other best practices that are used in making sure that the work that software developers are working on can be tested and deployed readily and very quickly. 

So we'll begin the conversation with testing. And testing is really about this idea of verifying and making sure that the code that software developers are writing are, in fact, correct, to make sure that the functions work the way they're supposed to, that the web pages behave the way that they're supposed to. And ideally, we'd like some way to be able to efficiently and effectively test our code over time, and as our programs grow more complicated, to allow our tests is to make sure that our program is behaving the way that we want it to. 

So we'll go ahead and start simple and consider the basic way that we might take a function, for example, written in Python and test and verify to make sure that it works the way we would expect it to. And to do so, we can start with a command in Python known as assert. And what assert does in Python is it asserts or just states that something should be true. And if that something is not true, then the assert is going to throw an exception, some sort of error, so that whoever is running the program or running the command knows that something went wrong. And this can be a very basic way that we can leverage Python's abilities to test a function and verify that that function behaves the way we would want it to. 

So let's go ahead and try a simple example of writing a Python function and then trying to test to make sure that that function works the way we would want it to. So I'll go ahead and create a new file. I'll call it assert.py. And let me define a new Python function, for example, that is going to take an integer and square it. Just want to take a number and return it square. So I'm going to define a function called square that takes as an input a number, like x. And I want to return x times x. It's a fairly straightforward function. But I would like to now verify that the function works the way I would expect it to. 

Now there are a number of ways that you could do this. One would be just like let's print out what the square of 10 is, for example, and just see what that's equal to. And then you could run a program, something like python assert.py and just say, all right, the answer is 100. And I could say to myself, OK, that's what I would expect it to be. But I now have to do the mental math of squaring the number 10, making sure that the answer comes out to be the value that I expect. It would be nice if I could automate this process. 

Well, one thing I could do is print out, does the square of 10 equal 100? I know that I want the square of 10 to equal 100, so I could just print out that value, print out, does square of 10 equal the number 100? I go ahead and run the program again. And this time, what I get is, true, for example, because those two things are equal to each other. And if, on the other hand, I had tried to check for something that wasn't true, like does square of 10 equal 101? You run the program, and, OK. Now it's going to be false. So this is nothing new, nothing we haven't seen before. 

But now what I can do is, instead of this, I can just say, let me assert that the square of 10 is equal to 100. Here I am just asserting that this expression, that the square root of 10 is equal to 100 is going to be true. And now I can run the program. And what you'll notice is nothing happens. No output, nothing at all because when an assert statement runs and the expression that it's checking turns out to be true-- that the square of 10 does equal 100-- it effectively ignores that statement altogether, just continues on to the next thing, no output, no side effect of any sort. And this is helpful because it just means that if I want to assert that something is true, I can assert it and then just continue writing my code. And it's as if I hadn't written that assert statement at all so long as the thing that I am asserting is actually true. 

But if there were a bug in my code, for example, some sort of mistake, where instead of returning x times x, imagine that I accidentally said, return x plus x to calculate the square instead, something that would be a bug in this case. Well, then when I try to run python assert.py, what I'm going to get is an exception. And the type of exception that I get is something known as an assertion error. And I can see that here. There is an assertion error. And then I see the reason why the assertion error happened. And the assertion error happened on line 4, which is this line here, where I said, I would like to assert that the square of 10 is equal to 100. 

So one way we can imagine testing our code is just by including a number of these different assert statements. If I want to verify that my code is correct, I can write various different assert statements. And for a function that's fairly simple, like the square function, probably not too many tests that I would need to write. But you can imagine for more complex functions that have multiple different conditional branches, being able to assert that no matter which conditional branch the program chooses to follow that the code will actually be correct can be a valuable thing to be able to say. 

And this can be helpful too when in working on a larger project, you want to deal with the problem of bugs that might appear inside of a project. And this gets at the idea of test-driven development, developing while keeping this notion of testing in mind. And one of the best practices would be if ever you're working on a program of your own and you encounter some bug in the program, you'll first want to fix the bug. But then you'll want to write a test that verifies that the new behavior is working as expected. 

And once you've written these tests, these tests can start to grow over time. And as you continue working on your project, you can always run those existing set of tests to make sure that nothing-- no new changes that you make to the program down the line, no future features that you add or changes you might make-- are going to break anything that was there before. And this is especially valuable, as programs start to get more complex and testing everything by hand would start to become a very tedious process, to be able to just automate the process of just run a whole bunch of tests on all of the things that I know that I would like the program to do and making sure they work as expected. That can be quite helpful. 

So assert then is one basic way of just saying that I would like for this statement to be true. And if it's not true, go ahead and throw an exception. And using Python, we know we also have the ability to catch those exceptions in order to make sure that we're able to handle those appropriately. So we can display a nice error message for example, if we wanted to do so. But now let's go ahead and try and write a more complex function, something more complex than just taking a number and squaring it, somewhere where there's more room for various different cases that I might want to test and more room where I, the programmer, might make a mistake, for example. 

So let's imagine writing a new file-- and then I'm going to call prime.py-- where here, I'll go ahead and say that I would like prime.py to implement a function called is_prime. And what the is_prime function should do is check to see if a number is prime or not. The prime number only has factors of 1 and itself. And I would like to write a function that verifies that fact. 

And so how might I go about doing that? Well, if n is less than 2, then it is definitely not prime because we say 0 and 1 are not going to be prime. And we'll only deal with numbers that are 0 or greater. And we'll deal with that for now. But let's start then with other numbers, numbers that are 2 or greater. 

Well, what do I want to do? I really want to check each of the possible factors. Like if I want to check whether or not 100 is prime or not, then I want to loop over all of the possible numbers that could be factors of 100, like 2, 3, 4, 5, 6. And when I get to a number like 2 or a number like 5 that do go into 100 cleanly, well, then I'll know that the number is not prime. 

So I could say for i in range from 2 all the way up through n, for example, let me go ahead and say, if n mod i equals 0, then return false. So what am I saying here? I'm saying go ahead and start at 2. Go up through but not including n. So if I'm checking to see if 10 is prime, for example, I'm going to check for i is 2, 3, 4, 5, 6, 7, 8, 9. And for each of those numbers, check if n, my input to this function, mod i, the factor that I would like to check, is equal to 0. This mod operator, this percent, if you don't recall, gives us the remainder when you divide one number by another. 

And so if n mod i equals 0, that means the remainder when you divide n by i equals 0-- meaning i goes into n cleanly with no remainder. And that means that it's not prime because it does have a factor. Whatever i is is going to be that factor. And if I get to the end of this for loop, then I can go ahead and just say, return true. If we weren't able to find a factor for the number other than 1 and the number itself, well, then, we can go ahead and say that true, this number is going to be prime. 

And so this, for example, could be a function that checks to see if a number is prime. But if I'm trying to optimize, I'm trying to make my function more efficient, I might realize that you really don't need to check every number from 2 all the way up to the number and itself. I could really just check up to the square root of that number, for example. That for a number like 25, I want to check 2, 3, 4, 5 because 5 squared is going to be 25. But after 5, I don't need to check any more numbers beyond that. 

That after you get to a number, after a number-- the square root of that number is multiplied by itself, there's never going to be a case where a number bigger than that could be a factor that I won't have already known about. So just thinking about things a little bit mathematically, we might be able to make some sort of optimizations where instead of going from 2 all the way up through n, I might go up to the square root of n. And I'll go ahead and import math to be able to use math dot square root. And I'll convert that number to an integer just in case the square root doesn't already happen to be an integer. 

So I think this works. I've at least talked myself into thinking that this is a function that might be able to check if a number is prime. So what could I do if I wanted to verify this? Well, I could write some assert statements. Another thing I could do is just use the Python interpreter. 

I could say, all right. Let me go ahead and type python. And I'm in the Python interpreter. And I can say, from prime, go ahead and import is_prime. prime is the name of that file. is_prime is the function in that file that I would like to test. And let's just try, all right, is_prime(5)? That's a prime number. Hopefully, it'll say true, that it's prime. All right, it does. 

Let's try is_prime(10)? See if that works. All right. is_prime(10) is false because 10 is not prime. That's good. That seems to be working as well. Let's try is_prime(99)? That's not prime because 3 is a multiple of that, for example. All right, false, so that's good. This seems to be working. And I could, in the interpreter, test this function to make sure that it works the way that I would want it to work. 

But let's now see some other ways that I might go about testing it. Well, one way is that I could write a file like tests0.py. And what tests0.py is going to do-- instead of using assert, I'm just going to do our Boolean checks like we were doing before. I'm going to import the is_prime function. And I've defined a new function called test_prime, which is going to serve the role of testing to make sure that when you square some number or when you check to see if some number n is prime, that you get some expected value, where that expected value is either true for it is prime or false for it's not prime. 

What, then, is this function doing? Well, the function is checking. We're calling the is_prime function on this number n and seeing whether or not it is equal to the expected value that we get, where we expect it to be either true or false. And if we run is_prime on n and it is not equal to what we expect, well, then, we print out, OK, there is an error. We expected some value true or false. But it turned out not to be the case. 

And so now that I have this test_prime function, well, I can say, all right. Let me go back into the Python interpreter. From tests0 import test_prime. And now I can say, all right, let me test_prime. Make sure that 5 is prime. So I'm passing in with my first input the number n, the number I would like to check. I want to check if 5 is prime. And the second input I provide is what I expect it to be-- either true or false. And here, nothing happens, which is a good thing. If there were an error, it would have printed something out. And the fact that I see nothing printed out means that everything was OK. 

If I test_prime now and say something like, all right, make sure 10 is not prime-- make sure that 10 when you pass it into is_prime is going to give us false. Again, nothing happens. Seems to be working just fine. Let me now try-- I can try more examples. Maybe I try test_prime 25. I want to make sure that 25 is not prime because 25 is not a prime number. 

All right. We get some sort of error. There's an error on is_prime(25) where I expected the output to be false. But for some reason, it looks like is_prime returns something other than false. It probably returned true. And so might indicate some sort of bug in my program, that somehow I don't think that 25 should be a prime number. But my program thinks that 25 is a prime number. And that error can be a clue to me as to how to do this. 

But ultimately, especially as programs start to grow longer, especially as I start to add more and more functions, testing each of those functions by hand is going to start to get tedious. So one thing I could do is write a script to be able to run all these tests for me automatically. 

And so here what I have a tests0.sh, .sh being like a shell script, some script that I can just run inside my terminal. And what this is doing is it's running Python 3, for Python version 3, dash c, which means I'm just going to give it a command. And it is going to run that command. 

And so I can just run these. And each of these lines does what? From tests0, it imports my test_prime function, that function that is going to test to make sure that the prime function produces the output that I would expect it to. And each time I'm testing a different number, making sure that 1 is not prime, making sure that 2 is prime, 8 is not prime, so on and so forth. And I can just write a whole bunch of these tests. And then rather than have to run each test one at a time, what I can do is I can just run tests0.sh. I can just say that I would like to run ./tests0.sh. 

And, all right. I see that I get two errors. I get an error on is_prime(8), where I expected it to not be prime. But for some reason, it seems to be prime. And then again, here, exception on is_prime(25), where I expected it to not be prime. But for some reason, my program thinks that it is prime. So a very helpful way for me to know immediately that there's some sort of error that is going on here. 

But ultimately, rather than have me have to write all this framework for how to go about testing my code on my own, there exists libraries that can help us with this. And one of the most popular in Python is a library known as unittest. And what unittest as a library designed to do is it is going to allow us to very quickly write tests that are able to check whether something is equal to something else. And then unittest is built in with an automated test runner that will run all of the tests for me and verify the output. 

And unittest gets built in to a lot of other libraries within Python. We'll see how we'll soon be able to apply this sort of idea to our Django applications as well. But let's now translate these tests that we have written ourselves just by writing a function like test whether the prime number is what we expect it to be and now translate it to using this Python unittest library instead. 

And so just to get a sense for what this looks like, I'll now go ahead and open up tests1.py, where here first thing I'm doing is I'm importing unittest, which we get for free with Python. I'm also importing the function that I would like to test. And now I'm defining a class which will contain all of my tests. This is a class that inherits from or derives from unittest.TestCase, which means that this is going to be a class that is going to define a whole bunch of functions, each of which is something that I would like to test. 

And so, for example, in this very first test, this is a test that checks to make sure that 1 is not prime. And so the way I do that is by calling self-- this testing object itself. It happens to have a method or function built into it called .assertFalse. There's an equivalent .assertTrue. But I would like to .assertFalse. And what would I like to assert that is false? is_prime(1). So whatever is_prime(1) is, that should be false. And I would like to just assert that it is false. 

Likewise, for the number 2 now, I want to check that the number 2 is prime. And the way I do that is by calling self.assertTrue. I would like to assert that when I run the is_prime function on the number 2, the output that I get is going to be a true value-- self.assertTrue. And I can translate each of the rest of my tests into one of these self.assertTrues or self.assertFalse. And then I say that if you go ahead and run the program, go ahead and call unittest.main, which will run all of these unit tests. 

So now, when I run python tests1.py, here's what I get. I get some nice output, where up at the top, I see dots every time a test succeeded and a letter F for a test that happened to fail. It says that it ran six tests. And down in the bottom, I see that there were two failures. So it's immediately going to tell me exactly what failed. And it'll give me some rationale, some reason for why it is that those tests failed as well. 

So we can see, all right. Here is one test. Here is another test. This test that failed is the test that checked that 25 is not prime. And this sentence here is what I supplied inside of what was known as a Python docstring inside of those triple quotation marks, underneath the declaration of the function. 

Those triple quotation marks, otherwise known as a docstring, serve a number of purposes. They can serve as just a comment for describing what it is the function does. But they're a special comment insofar as someone who's looking at the function can access that docstring that's usually used for documentation for what it is that the function is doing. And they can use it inside of other places as well. 

And so what unittest is doing is for every function, it uses that docstring as a description of what the test is testing for, so that if a test fails, then I can see exactly what the name is of the test that failed and what it was tested, where the description describes what was happening. 

Now in this case, where it's just one function and I'm testing a whole bunch of different numbers, it doesn't seem all that useful. But again, if you imagine projects that start to get more complex, being able to know immediately when you run your tests which parts of the program or which parts of your web application are working the way they are expected to can actually be quite helpful. 

So test 25, that was the function that triggered an assertion failure in this case. And the line that caused it was self.assertFalse(is_prime(25)). And the reason that it failed is because True, which apparently was the output of this function, is not false. And I expected it to be false instead. 

And so multiple, different ways of trying to run our tests. This happens to be one quite popular one. But this now tells me that I should go back and try and fix my is_prime function. I can go back into prime.py and say, all right. I would like to figure out why this went wrong. And if you look at this enough and maybe give it a little bit of testing, you might see that I have a slight off by 1 error, that I probably need to check one additional number than I actually am because in checking whether or not 25 is prime or not, for example, I might need to go up to and including the number 5 to know that 5 is a factor of 25. But before, I was going up to the number 5, but I wasn't including the number 5. So I also need to just check one more number. 

And now to verify that this is right, I could just manually test the function myself. Or I could just run these tests again, run python tests1.py. And this time, all these dots mean all these tests succeeded. We ran six tests, and everything was OK. No failures. And so this can be a helpful way for me to know immediately that things seem to be working OK. 

So the takeaways from here are that these tests can definitely help as you begin to write new changes to your program, especially as you begin to optimize functions. You might make a function more efficient but then run your tests to make sure that in making these improvements, you haven't broken anything. You haven't changed any behavior, that the way the program was supposed to behave and now it doesn't behave that way, you are able to verify with much more confidence that that is true. 

But of course, that only works if your tests have good coverage of all the things that you would want the function to do, and you've covered appropriately all the various different cases for how the function should behave because only if the tests are comprehensive, will they actually be useful to you in indicating that the change that you made isn't going to break anything. And only then can you actually feel confident in those changes themselves. 

So now let's take this idea of using unittest to be able to write these tests that verify that a function works and apply it to something like a web application-- like a web application written in Django that we would like to now use in order to be able to test to make sure that various, different functions inside of our Django web application work as well. So what I'm going to do is actually take a look at the airline program that we wrote back when we were first talking about Django and first talking about storing data inside of databases. 

And I'm going to open up models.py where you see that I've made one addition to our definition of a flight. And recall from before, when we first introduced this idea of defining a model inside of our application for a flight inside of an airline, we gave that model three properties. It had an origin and a destination, where both origin and destination referenced an airport object, where an airport object was an object we defined separately. But a flight has an origin airport and a destination airport. 

And in addition to that, every flight has a duration, some number of minutes long that that flight is going to last. And I might like to have some way to validate, to verify that a flight is a valid flight, that there isn't some error somewhere in how the data was entered into the database. I would like to just generally make sure that given a flight, I can check to make sure it's a valid flight. 

And what does it mean for a flight to be valid? Well, in general, given these particular fields, I'll say there are two things that need to be true for a flight to be valid. The origin and the destination need to be different. That's condition number one. It wouldn't make sense to have a flight whose origin and destination are the same airport. And condition number two, the duration of the flight needs to be greater than 0 minutes. If ever the duration is 0 or the duration is negative, that probably indicates to me that there was some sort of mistake in data entry or some problem that happened with how it is that these flights were configured. So I want to make sure that the duration is greater than 0. And those then are my two conditions for what makes a valid flight. 

And I've, in fact, written a function here called is_valid_flight that just works on this flight class that simply checks given a flight, make sure that it is, in fact, valid. And the way it's doing that is by checking for these two conditions that I've just described. It's checking to make sure that the origin is not equal to the destination. It's checking to make sure that the duration of the flight is greater than or equal to 0. And maybe I should change that to greater than to make sure it's entirely positive. But this then is my definition for what it means for something to be a valid flight. 

And what I'd like to do now is test these various, different parts of my application. I have this is_valid_flight function inside a flight that I might like to test as well. But we also have all of these other properties that I would like to test, these relationships, that a flight has an origin and a destination. We have passengers that can be associated with flights. So there's lots of relationships between my data that I would like to test and verify to make sure they work the way we would expect it to. 

So to do that, whenever we create an application in Django, like this flight's application here, we were also given this tests.py file. And we haven't yet used the tests.py file for anything. But what it's supposed to be used for is for writing these sorts of tests, testing that verifies that our application behaves the way that we want it to behave. 

So let's go ahead now and open up tests.py and see what happens to be in here. What we can do is we can define a subclass of TestCase, which behaves very similar to unittest and is based on that same idea. I'll define a new class called FlightTestCase that will just define all of the tests that I would like to run on my flight's application. 

And so things to know about this is that first, I might need to do some initial setup in order to make sure that there's some data that I can actually work with and test with. And what Django will do when I go ahead and run these unit tests is that it will create an entirely separate database for me just for testing purposes, that we have one database that contains all the information that actually pertains to the flights that are actually there on my web server. But we might also like to just test things with some dummy flights and some dummy airports just to make sure that things are working. And then once we're confident things are working, then we can deploy our web application to let actual users begin to use whatever new features we've added to the web application, for example. 

So inside of this database, I might need to do some initial setup. And I can do so by defining a setup function inside of my test case class. This is a special function. And Django knows that when it's running these tests, it should first do any of the setup steps that we need to do. 

And so how are we doing this? Well, what we're doing is inside of the setup, we're going to just add some sample data into the test database. Again, this won't touch the database that users actually see and actually interact with. This is just our test one for testing purposes. 

And we'll start by going ahead and creating some airports, so Airport.objects.create, and then specifying what the values for these fields should be. We'll just have an airport whose code is AAA for city A and an airport whose code is BBB for city B. Just a dummy airport names. They're not real airports but just used for testing purposes. And I'll save those airport objects inside of these values, a1 and a2. 

And beneath that, what I'm going to do next is go ahead and create some flights where I create using Flight.objects.create three different flights, one that goes from a1 to a2 with the duration of 100 minutes; one from a1 to a1 with a duration of 200 minutes; one from a1 to a2 with a duration of negative 100 minutes. So I have a whole bunch of these flights now that I would like to test, that I would like to make sure work in some predetermined or expected way. 

And so now if I scroll down, we can see that I have a whole bunch of these various different tests. Here is one test that just tests the departures count. So every airport has access to a field called departures, which ideally should be how many flights are departing from that airport. And I'd like to make sure that departures_count works the way I expect it to. 

So here, I go ahead and get the airport whose code is AAA. And now, using unittest-like syntax, I would like to say self.assertEqual. So assertTrue verifies if something is true. assertFalse verifies if something is false. assertEqual verifies that two numbers are equal to each other. 

And here, I'd like to verify that a.departures.count-- if I take airport a and count how many flights are departing from that airport, that that should be 3, so just verifying that works. And then after that, if this test passes, then I can be confident that elsewhere in my program, if I take an airport and call that airport.departures.count, I can feel pretty confident that that is going to work the way I would expect it to. 

I can do the same thing for arrivals, get the airport, and assert that a.arrivals.count, that that is going to be equal to the number 1 if there's only one flight that arrives at airport a1, for example. So that tests these relationships. 

And I can also now test the is_valid_flight function as well, that here I get my two airports, a1 and a2. This is the one whose code is AAA. This is the one who's code is BBB. I'll go ahead and get the flight whose origin is a1, whose destination is a2, whose duration is 100. And let me just assertTrue that this flight is going to be a valid flight because this fight is valid. The origin is different from the destination. Its duration is some positive number of minutes. And so I should feel pretty confident that this is going to be a valid flight that I can verify by calling self.assertTrue. 

I can do the same thing for testing for an invalid flight, testing for an invalid flight because the destination is bad. I can get the flight, airport a1, and get the flight whose origin and destination are both a1. And now let me self.assertFalse, say that this should not be a valid flight because the origin and the destination are the same. 

What's the other way a flight can be invalid? Well, a flight can be invalid because of its duration. So I could say something like, go ahead and get airports a1 and a2. And get me the flight whose origin is a1, destination is a2, but the duration is negative 100 minutes. That was one of the flights as well. And, well, that should not be a valid flight. So I'll say self.assertFalse is_valid_flight because when I call is_valid_flight on that flight, it shouldn't be valid because the duration makes it an invalid flight. 

So here now I've defined a whole bunch of tests. And there are more done below that we'll take a look at in a moment as well. But I've defined a whole bunch of these flights now or a bunch of these tests. And now I'd like to run them. And the way that I can run tests in Django is via a manage.py command. manage.py has a whole bunch of different commands that we can run. We've seen makemigrations and migrate and runserver. 

But one of them as well is if I go into airline0, I can say python manage.py test that's just going to run all of my tests. And we're right. It seems that we ran 10 tests. But two of them failed. So let's go ahead and see, why did those two tests fail? Well, the way to read this is that we get this heading anytime a test failed. And so we failed the test_invalid_flight_destination function. And we failed the test_invalid_flight_duration function. And docstrings could have helped me to know what it is that these tests are exactly doing. 

But it seems that true is not false. I wanted to assert that this should not be a valid flight, that it should be false. But for some reason, these appear to be valid flights. So something seems wrong with is_valid_flight where it's returning true when it should be returning false. And so this then gives me a place to start looking. 

I can say, all right. Let me go to is_valid_flight and make sure that function is correct. So I'll go back into models.py. I'll take another look at is_valid_flight. Maybe I'll think through the logic again. All right. I wanted to check that self.origin is not self.destination. I wanted to check that the duration is greater than or equal to 0. I could change this to greater than. But I don't think that's the issue because my duration was negative. And so that already should have been invalid. 

But the other thing I might realize looking at this now is that, OK, the logical connectives that I have used was not the right one. I want to check that for it to be a valid flight, it needs to satisfy both of the conditions. The origin and the destination need to be different. And the duration of the flight needs to be greater than 0, for example. And here, I've used "or" instead of "and." So I can just change it. All right. And hopefully, that will fix things. 

And to verify as much, I can rerun python manage.py test. Go ahead, and press Return. It's going to check things. I ran 10 tests. Everything is OK. And all right, it seems that now I have passed all of these tests. 

I notice here at the top, it created a test database. So it just created a test database for me in order to do all this testing work. And then it destroyed that test database at the end as well. So none of my tests-- I'm adding data, removing data. It's not going to touch any of the actual data. Inside of my database for the web application, Django will take care of the process of keeping all of that separate for me by first calling that setup function to make sure that my new test database has everything that it needs to. 

So, all right. We now have the ability, by using unittest, to be able to test various, different functions inside of our web application. We first saw that we could test a function like the is_prime function, just a Python function that we wrote. But we can also test things like functions on our model, like checking to make sure that a flight is valid, checking to make sure that we can take a flight and access all of the-- or take an airport, and access all of its arrivals and all of its departures. 

But I'd like to do more than that, especially for a web application. I'd like to check that particular web pages work the way that I want them to work. And so to do that, Django lets us simulate trying to make requests and get responses to a web application. And so let's go ahead and look at some of these other tests as well. 

Here what we have is a function called test_index. And what test_index is going to do is it's just going to test my default flights page to make sure that it works correctly. So we start by creating a client, some client that's going to be interacting request and response style. Then I'm going to call client dot get slash flights. That is the route that gets me the index page for all the flights. And I'm saving that inside of a variable called response. Whatever response I get back from trying to get that page, I would like to save inside of this variable called response. 

And now, I can have multiple assert statements inside of the same test if I would like to. Sometimes you might want to separate them. But here, I want to check that the index page works. And what that means is a couple of things. It means that response.status_code, well, that should be equal to 200. 200, again, meaning, OK. I want to make sure that whatever response I get back, that that is going to be a 200. And if there was some sort of error, like a 404 because the page wasn't found or a 500 because of some internal server error, I would like to know about that. So let me first just assert that the status code should be equal to 200. 

But then Django also lets me access the context for a response. And what is the context? Well, recall, again, in Django, when we rendered a template, for example, we called return render then provided the request and what page we were going to render. But we could also provide some context, some Python dictionary, describing all of the values that we wanted to pass in to that template. 

And Django's testing framework gives us access to that context so that we can test to make sure that it contains what we would expect it to contain. And on the index page for all my flights, I would expect that to contain a listing of all of the flights. And we created three sample flights inside of this test database. So I should be able to assert that these two things are equal. 

response.context flights, that gets me whatever was passed in as flights in the context. Dot count, well, that better be 3 because I want to make sure that there are exactly 3 results that come back when I look at the context and access whatever it happens to be inside of that flight's key. 

There are other tests I can run as well. So in this case, I've gone ahead and gotten a particular flight, the flight who, in this case, had an origin of a1 and a destination of a1, that it was not a valid flight. But we'll go ahead and get it anyway because it exists in the database. 

And now I can get slash flights slash that flight's ID because on my flight's page, I would like to be able to go to slash flights slash 1 to get at flight number 1, and go to slash flights slash 2 get at flight number 2. So if I take some valid ID, some ID of an actual flight f and go to slash flights slash dot id, well, that should work. It should have a status code of 200. 

Meanwhile, though, if I test an invalid flight page, this is a Django command that will get me the maximum value for the ID. This id__max gets me the biggest possible ID out of all of the flights that happen to exist inside of my database. If I go ahead and try and get slash flights slash max_id plus 1-- so a number that is 1 greater than any of the flights that were already inside of my database-- well, that shouldn't work. There shouldn't be a page for a flight that doesn't exist. So here then, I can assertEqual that the status code of what comes back, that that is equal to 404 because I would expect that page to return a 404. 

And finally, I can also check various different contexts for things about the passenger page. So in this case, I've added some sample passengers to the database. So inside of my test, I can manipulate the database as well, adding data into the database and checking to make sure that when you count up the number of passengers on a flight page, that that is going to be like the number 1, for example. So a number of different tests that we can then write in order to verify various different parts of our web application. 

I would like to verify not only that our database works the way we would expect the database to work in terms of how functions on our models work, in terms of relationships between those models like the relationship between a flight and an airport, but we can also simulate a GET request, simulating the request to a page and verify that the status code that comes back is what we would expect it to be; verify that the contents of the page contain the right contents; verify that the context that was passed into that template is correct as well. 

And then all of that can then be verified by saying something like python manage.py test then go ahead and running that. And we see that in this case, all of the tests happened to pass, which means that everything seems to be OK, at least for now. So this then, again, very helpful as our web programs start to get more complex, as we have multiple different models, multiple different routes, the ability to test, to make sure that if we change things in one part of the program, that it doesn't break things in another part. That can be quite helpful too. 

What we haven't yet been able to test though is any interaction that's happening exclusively in the browser. But I've been able to test a lot of things that are happening server-side. And recall that Django is all about working on writing this web server-- a Python application that acts as a web server that listens for requests that come in from users, processes those using these various different views and models, and then provides some sort of response back. 

And we can test the contents of that response, things like, does the status code match what we would expect it to? Does the context match what we would expect it to? But there are some times where we would like to really simulate like a user clicking on buttons and trying things on a page and making sure that page behaves we would like it-- how we would like it to as well even if we're not using Django, or even if we're just dealing with the front end. 

So let's create a sample JavaScript web page that we might like to test now by using these sorts of ideas-- the ability to automate the process of testing various, different parts of our application to verify that they do, in fact, work correctly. What I'm going to do now is we'll get out of the airline directory. And I'm going to create a new file that I'll call counter.html. And recall before, we created a counter application using just JavaScript where what the counter application did is it let me click a button, like an increase or a count button that just incremented a number. It went from 0 to 1 to 2 to 3 to 4 and so forth. 

I'm going to do the same thing here. We'll add a little more complexity though and give myself both an increase button to add a number and a decrease button to decrease the number by 1 as well. So I go ahead and start with our usual DOCTYPE html and our html tag. I'll give this page a title of Counter. And now, inside of the body of this page, I will go ahead and start with a big heading that just says 0. And then beneath that, I'll create two buttons. I'll have one button which will be the plus symbol and one button which will be the minus symbol. 

So now no JavaScript yet. This won't actually work. But I can go ahead and go into open counter.html. And I can see that I now have 0, and I have a plus and a minus button, although those Plus and Minus buttons don't actually do anything right now. So let's go ahead and make them do something. Let's give this button an ID called increase so that I can reference it later, and give this button an ID called decrease, again, so that I can reference it inside of my JavaScript. 

And now, in the head section of my web page, I add a script tag where I want to start running some JavaScript once the page is done loading. And to do that, you'll recall, I can say document.addEventListener c to say, go ahead and run this function once the DOM is loaded, once all the contents of this page have loaded the way I would expect it to. 

And what am I going to do? Well, I first need a variable, something like let counter equal 0. And then I can say, all right document.querySelector increase. Get me the element whose ID is increase. That's that plus button. And when you are clicked, let's go ahead and add an event handler in the form of this callback function, this function that will be called when the increase button is clicked. 

And what would I like to do? Well, I'd like to increase the counter. Go ahead and say counter++. And then I'm going to update this h1 that currently contains 0. document.querySelector h1. That gets me the H1 element. I'll go ahead and update its inner.HTML, and go ahead and set that to whatever the value of counter happens to be. 

And then I'll do the same thing for the decrease button. document.querySelector, get me the element whose ID is decrease-- that's the button that will be the minus button. And when you are clicked, go ahead and run this callback function, which will do the same thing, except we'll first do counter-- decrease the value of the counter by 1, and then get me the H1 element, set its innerHTML equal to the counter. 

I think this should work. And I could verify that by opening up counter.html. I'll refresh the page. And I can test these buttons. Test the plus button. All right. That seems to work, increases the value by 1. I can test the minus button, make sure that that decreases the value by 1. That all seems to work just fine. But, of course, this requires me having to interact with this page. I have to open up the page. I have to click on these buttons. 

And it's not the type of thing where I'm simulating like a GET request or a POST request. There is no server that I'm sending a request to and getting back a response of 1, 2, 3, 4 from. This is all happening in the browser. 

And so what I might like to have the ability to do is some sort of browser testing. And there are a number of different frameworks for doing this. One of the most popular is Selenium. And what this is going to allow me to do is I can define a test file that using unittest or a similar library can effectively simulate a web browser-- can simulate a web browser and simulate a user interacting with that web browser using something that we call a WebDriver that's going to allow me to, using code, control what it is that the browser is doing and how it is that the user is interacting with this program. 

So how is this going to work? Well, I'm going to go ahead, and just as a test, let me open up the Python interpreter. And let me import from tests import star. And that's going to give me access to a couple of different things. But the first thing you'll notice that it's doing is because I'm using this WebDriver, it's going to give me a web browser. And here I'm using Chrome. But you could use another browser. But notice it here, Chrome is telling me Chrome is being controlled by automated test software, that Chrome has the ability to allow me, using automated test software, using Python code, control what it is the web browser is doing. 

And so what I can do here is the first thing I need to do is tell Chrome to open up my web page. And it turns out that in order to do so, I need to get that page's URI or Uniform Resource Identifier, just some string that will identify that page. And I've defined a function called file_uri that gets me the URI of a particular file in this directory. 

So I'm going to say I want open up counter.html. And I need to get it's URI. But now what I can say is driver.get(uri), meaning tell this web driver, this Python-- part of the Python program that is controlling the web browser that I would like to get this web page as if the user had gone to the web page and pressed Return after typing in the URL. So I say driver.get(uri). I go ahead and press that. 

And what you'll notice on the right-hand side here is that Chrome has loaded this page. I am effectively controlling this web browser window using my Python program. I said driver.get(uri), meaning go ahead and open up the counter.html page. And then, inside of this test window, Chrome has opened that up. 

And inside my Python program now, using this web driver, I have the ability to see the same things that the user sees when they open the page. So what does a user see when they open the page? Well, they see, for example, the title of the page. So I could say driver.title and see that, all right, the title of this page is Counter. That was, in fact, the title of the page. But I could verify that inside my Python program that by checking that driver.title, by taking my WebDriver, getting the title of the current page that it's looking at, making sure that that is Counter. 

And likewise, if I looked at driver.page_source, press Return, what I see there in string format-- so it's a little bit messy-- is the content, the HTML content of this page. And you'll notice things like DomContentLoaded, here are my onclick handlers. Here's my h1 that says 0. It's very messy because it's being represented as a Python string. And these backslash n's refer to new lines where there's a line break. 

But this is the content. And this is really all the browser gets. The browser takes this information and just knows how to render it in some nice, more graphical representation that's easier for a user to look at and, therefore, be able to understand. But this is fundamentally all that my web browser is actually getting back when it tries to load a web page. 

So what can I do from here? Well, I would like to simulate a user's behavior on this page that assures I can get a page and see the title. But I want to simulate like clicking on the plus button, for example. So in order to do that, first thing I need to do is actually get the Plus button. And to do that, I could say something like driver.find_element_by_id. 

There are a number of ways that I could try and find an HTML element. But I would like to find an HTML element by its ID. And I know that the Plus button-- the Increase button-- has an ID of increase, for example. And so if I find element by ID, let me find the element whose ID is increase. And, all right. It seems here, that I've gotten some web element object back from my web driver. 

And I'll go ahead and save that inside of a variable called increase. So what I now have is a variable called increase that represents the Increase button that my WebDriver has found on the web page. It's effectively the same thing as you, the human, going through the page looking for the Increase button. The WebDriver is doing the same thing, except instead of looking for the button based on what it looks like, it looks for the button based on its ID. And so this, again, another reason why it's helpful to give your HTML elements IDs. In case you ever need to be able to find that element, it's very useful to be able to reference that element by its name. 

But now that I have a button, I can simulate user interaction with that button. I can say something like increase.click to say I would like to take the Increase button and simulate a user clicking on that button in order to see whatever it is that the user would get back when they click on that button. So increase.click, I press Return. And what you'll notice happens is that the number increases-- increases from 0 to 1. It's as if I, the user, had actually clicked on the Plus button, except all I did was say increase.click to say, go ahead and press the Increase button and let the browser do what it would normally do in response. 

And what it would do in response is get that JavaScript event handler, that onclick handler, and run that callback function that increases the value of counter and updates the h1. So I can say increase.click to simulate increasing the value of that variable. 

But this is just a function call, which means I can include it in any other Python constructs that I want, that if I want to repeat something like 25 times, for example, and press the button 25 times, I can say for i in range 25, go ahead and click the Increase button. And now, very quickly, 25 times, it's going to click the Increase button. And I'm going to see the result of all of that interaction. So I can simulate user interaction just by using the Python interpreter. 

Likewise, if instead of increasing, I wanted to decrease, well, then, I'm going to do the same thing. I'm going to say decrease equals driver.find_element_by_id. Let me get the decrease element, the element whose ID is decrease. And now say, decrease.click. And that will simulate me pressing the Decrease button. Press it again. And the more I press it, every time, it's just going to decrease by 1. 

And if I want to decrease it all the way back to 0, well, then, I'll just do it 20 times. For i in range 20, go ahead and decrease.click. And that's going to go ahead and reduce the count all the way back to 0 by simulating the user pressing a button 20 times. 

And what you'll notice is that it happened remarkably fast. Like I can simulate 100 presses of the Increase button by saying for i in range 100, increase.click. And very quickly, you'll see that number 100 times go ahead and go up to 100 faster than a human could ever have clicked that Plus button over and over again. And so these tests cannot only be automated, but they can be much faster than any human could ever be in order to test this behavior. 

So how then can we incorporate this idea into actual tests that we write, into like a unit testing framework that allows me to define all of these various different functions that test different parts of my web application's behavior? Well, to do that, let's go ahead and take another look at tests.py. Inside of tests.py, here, again, is that file_uri function, where that function has the sole purpose of taking a file and getting its URI, and we need the URI to be able to open it up. 

Then we go ahead and get the Chrome WebDriver, which is going to be what's going to allow us to run and simulate interaction with Chrome. And in order to get Chrome's WebDriver, you do have to get ChromeDriver separately. It is separate from Google Chrome itself. But Google does make it available. And other web browsers make equivalent web drivers available as well if you'd like to test how things would work in other browsers because different browsers might behave differently. And it might be useful to be able to test to make sure that not only does everything work in Google Chrome, but it's also going to work in other browsers that you might expect users to be working with as well. 

Here, then, I've defined a class that, again, inherits from unittest.TestCase that is going to define all of the tests that I would like to run on this web page, that here I have a function called test_title that's going to go ahead and first get counter.html. It's going to open up that page. And then just assertEqual, let's make sure the title of the page is actually Counter. That's what I would expect it to be. So I can write a test in order to test for that case as well. 

Here, I test the Increase button by finding the element whose ID is increase and clicking on that button to simulate a user pressing the Plus button in order to increase the value of the counter. And then, what do I want to check? Well, I want to check that when you find element by tag name, h1, and so find_element_by_tag_name, similar to find_element_by_id, except instead of finding something by its ID, it's going to look at what's the tag. And there is only one element that is in h1. 

And so here I'm saying, go ahead and get me the H1 element and access its text property, meaning whatever it is that is contained inside of those two H1 tags, I would expect that to be the number 1. And I would assert that this is equal to 1. And likewise, I can do the same thing for the Decrease button-- finding the element whose ID is decrease, clicking on that button, and then asserting equal, find me the H1 element and make sure that the contents of it are equal to the number negative 1, for instance. And this final test, just test things multiple times, that three times I'm going to press the Increase button and make sure that after I press the Increase button three times, when I check the h1, check what's inside of its text, that the answer should, in fact, be 3. 

So now I should be able to go ahead and test this code by running python tests.py. And what that is going to do is it's going to open up a web browser. And what you're going to see, very quickly flashed across my screen, were all of those tests. We tested the increase by 1. We tested decreased by 1. And then we tested like increase 3 times after we had checked to make sure the title was correct. 

And then we can see here is the output. We ran four tests in this amount of time, and everything turned out to be OK. None of the tests failed. But if one of the tests had failed, well, then, we would see a different output. So let's imagine, for example, that I had had a bug in my decrease function, for example, where the decrease function wasn't actually working. What would that bug look like? Maybe I forgot to say counter minus minus. Or maybe, perhaps more likely, what might have happened is I wanted to-- I already had written the increase function, and I decided to very quickly add the decrease function, and I thought I just-- like Copy/Paste, like copy the increase event handler. The decrease event handler is basically the same thing except I need to query for decrease instead. And maybe I just did that and forgot to change plus plus to minus minus, a bug that might happen if you're not too careful about how you copy and paste code from one place to another. 

Now when I run these tests, python tests.py, we'll see the simulation. A whole bunch get simulated. And when I go back and check the output of my tests, see what actually happened, I see that we do seem to have an assertion error here. The assertion fail was on the test decrease function. And it happened when I tried to assert that what was inside of the H1 element was negative 1 because 1 is not equal to negative 1. So this is the value of this assertion error as well. 

And this is helpful, an advantage over just assert. Assert just tells you there is an assertion error. But here, in unittest, we actually get to see if I asserted that two things are equal, it tells me what both of those things are. It tells me the actual output of h1's text. It was 1. But what I expected it to be was negative 1. So it tells me exactly what the discrepancy is. I know that for some reason, it was 1 instead of negative 1. And that can be a clue to me, a hint to me, as to how I can go about trying to solve this problem. 

And I can solve the problem by going into my decrease event handler, seeing that, all right, this was increasing the counter instead of decreasing it. Change plus plus to minus minus, and now rerun my tests and see all of the test simulated inside of my Chrome Driver. And we ran four tests. And this time, everything was OK. So all of my tests appear to have passed this time. 

So those, then, are some possibilities for being able to test our code, especially taking advantage of unittest, this library that we can use in Python in order to make various types of assertions about what we would like to be true or false about our code. And unittest contains a number of helpful methods for being able to perform these sorts of assertions. Some of them are here. So we can say things like I would like to assert that two things are equal to each other, which we've seen. There's a counterpart to that, assertNotEqual for making sure the two things are not equal to one another. assertTrue and False, we've seen as well. 

There are others as well though, things like assertIn or assertNotIn, if I would like to assert, for example, that some element is in some list, for example, or that some element is not in some list. There are other assert methods as well that we can use in order to verify that a part of our program or a part of our web application does, in fact, behave the way we would want to behave. 

And we can integrate this type of idea into a number of different types of testing. We saw integrating it into Django itself, using Django as unit testing in order to verify that our database works the way we expected it to, and that our views works the way that we expected them to and provided the right context back to the user after the user makes a request to our web application. And there are also applications of unit testing, whether using the framework or not, to browser-based testing, when I want to test inside of the user's web browser. Does it actually work when a user clicks on this button, that the JavaScript behaves the way that I would expect it to. 

And I don't need to especially use JavaScript in order to do those tests. I didn't write those tests using Python, using unittest, to be able to say, click on the button that has this ID and verify that the result that we get back is what we would expect it to be. So that then was testing. And now we'll go ahead and take a look at CI/CD-- Continuous Integration and Continuous Delivery which refer to two best practices in the software development world that has to do with how it is that code is written, especially by groups or teams of people; how it all works together, and how that code is eventually delivered and deployed to users who are using those applications. 

So CI, which refers to Continuous Integration, involves frequent merges to a main branch of some repository, like a Git repository, and then automatically running unit tests when that code is pushed. So what does that generally mean? Well, in the past, you might imagine that if multiple people are working on some project at the same time and multiple people are each working on different features or different parts of that project, then after everyone's done working on those features and we're ready to ship some new version of a web application or ship some new version of a software product, well, then, everyone's going to have to take all these various different features and combined them all together at the end and figure out how to then try and deliver that program to users. 

And this has a tendency to cause problems, especially if people have been working on different big changes all simultaneously. They might not all be compatible with one another. There might be conflicts between the various different changes that have been made. So waiting until everyone is done working on a feature to merge them all back together and then deliver it is not necessarily the best practice, which is why increasingly, many more teams are beginning to adopt a system of continuous integration, that there is one repository somewhere online that's keeping the official version of the code. Everyone works on their own version of the code, maybe on their own branch, for example. But very frequently, all of these changes are merged back together into the same branch to make sure that these incremental changes can be happening such that it's less likely that there's two really divergent paths that the program has gone under, and as a result, it's much more difficult to merge those two paths back together. 

In addition to frequently merging to its own main branch, another key idea of continuous integration is this idea of automated unit testing, where unit testing, again, refers to this idea of on our program, we run a big series of tests that verify each little part of our program to make sure that the web application behaves the way it is supposed to. And unit tests generally refer to testing particular small components of our program, making sure that each component works as expected. There are also bigger scale tests-- tests like integration tests that make sure that the entire pathway from user request and response, that everything along a certain pipeline works as well. 

But there are various different types of testing. And the important thing is making sure that anytime some new change is merged into the main branch or someone wants to merge their changes into the main branch, that these tests are run to make sure that nobody ever makes a change to one part of a program that breaks some other part of the program. 

And in a large enough code base, it's going to be impossible for any one person to know exactly what the effect of one particular change is going to be on every other part of the program. There are going to be unforeseen consequences that the one programmer may or may not know about. And so the advantage of unit testing, assuming they're comprehensive and cover all of these various different components of the program, is that any time someone makes a change and attempts to merge that change into the main branch according to the practice of continuous integration, the fact that it doesn't pass a test, we'll know about that immediately. And as a result, that programmer can go back and try to fix it as opposed to waiting until everything is done, merging everything together, and then running the tests, realizing something doesn't work, and then being unsure of where to begin. We don't know where the bug is, which change happened to cause the bug. If everything is merged more incrementally, it's easier to spot those bugs, assuming there's good coverage of tests to make sure that we're accounting for these various different possibilities. 

So continuous integration refers to that idea-- frequently and more incrementally updating the main branch and making sure that the tests are, in fact, passing. And it's closely tied to a related idea of continuous delivery, which is about the process of how it is that the software is actually released to users, how the web application actually gets deployed. 

And there are a couple of models you might go about thinking with regards to how it is that some program or web application gets deployed. You might imagine that the release cycle might be quite long, and the people spend months working on various different features on some software development team. And after they're happy with all the new changes, they've released some new version of the web application. 

But especially, with web applications that are undergoing constant change, that have lots of users, that are moving very quickly, one thing that's quite popular is this notion of continuous delivery, which refers to having shorter release schedules. Instead of immediately releasing something at the end of some long cycle, you can in shorter cycles make releases every day, every week, or so in order to say that let's just go ahead and incrementally make those changes. Whatever new changes happen to have merged to the main branch, let's go ahead and release those as opposed to waiting much longer in order to perform those releases. 

And that, again, lends itself to certain benefits, the benefit of being able to just incrementally make changes, such as something goes wrong, you know more immediately what went wrong as opposed to making a lot of changes at once, where if something goes wrong, it's not necessarily clear what went wrong. And it also allows new features to get out to users much more quickly. So especially in a competitive market where many different web applications are competing with one another, being able to take a new feature and release it very quickly can be quite helpful. So continuous delivery is all about that idea of short release cycles. Rather than wait a long time for a new version to be released, release versions incrementally as new features begin to come in. 

It's closely related to the idea of Continuous Deployment, which CD will sometimes also represent. Continuous deployment is similar in spirit to continuous delivery. But the deployments happen automatically. So rather than a human having to say, all right, we've made a couple of changes. Let's go ahead and deploy those changes. In continuous deployment, any time these changes are made, the pipeline of deploying the application to users will automatically take place as well, just removing one thing for humans to have to think about and allowing for these deployments to happen even more quickly as well. 

So the question then is, what tools can allow us to make continuous integration and continuous delivery a little bit easier? What techniques can we use in order to do so? And there are a number of different continuous integration tools. But one of them produced by GitHub more recently is known as GitHub Actions. And what GitHub Actions allows us to do is to create these workflows where we can say that anytime, for example, someone pushes to a Git repository, I would like for certain steps to take place, certain steps that might be checking to make sure that the code is styled well. 

That if a company has some style guide that it expects all of its programmers to adhere to when working on a particular product, you could have a GitHub Action such that anytime someone pushes to a repository, you have an action that automatically checks that code against the style guide to make sure that it is well-styled, well-commented, documented, and so forth. 

You might also, for instance, have a GitHub action that tests our code to make sure that anytime anyone pushes code to a GitHub repository, we automatically run whatever tests we would like to run on that particular code base. And GitHub Actions can allow us to do that as well by defining some workflow to be able to do so. And so that's what we'll take a look at in just a moment, using GitHub Actions to automate the process of running tests so that the human-- though it would be a good thing for the programmer when they're done writing their code to test their code and make sure it works-- we can enforce that by making sure that every time anyone pushes to a GitHub repository, we'll automatically run some GitHub action that is going to take care of the process of running tests on that program. And we'll know immediately as via an email that GitHub might send to you to say that this particular test failed. And you'll know every time you push to that repository. 

So how do these workflows get structured? What is the syntax of them? Well they use a particular type of syntax known as YAML, which is some language, a configuration language, that can be used in order to describe-- often described for configuration of various different tools and software. GitHub Actions happens to use it. Other technologies use it as well. 

And YAML is a file format that structures its data sort of like this, in terms of key value pairs, much in the same way that a JSON object or a Python dictionary might, where we'll have the name of a key followed by a colon followed by its value-- name of a key, followed by a colon, followed by a value. 

And the value doesn't necessarily need to be just a single value. It could be a sequence of values, like a list of values, for example. And those are generated this way, by like a hyphen indicating a list-- item 1, item 2, item 3. And in addition to just having single values and lists of items, these ideas-- these lists, these sequences, these values-- can be nested within one another, that you might have one key that leads to another set of keys that are associated with values that leads to other sets of keys associated with values as well. 

Much in the same way, that a JSON object, like a representation of keys and values, can also have nested JSON objects within a JSON object. Likewise, too, we can have nested key value pairs as the value for a particular key too. So we'll take a look at an example of what that actually looks like in the context of creating some GitHub workflow that will run some get GitHub Actions. 

So what will that look like? Let's go back into airline0, where here, I've defined inside of a .github directory a directory called workflows, inside of which I have a ci.yml file. It can be any name .yml. .yml or .yaml are the conventional file extensions for a YAML file. And here, I'll open up ci.yml. And this now is the configuration for how this workflow ought to behave. 

I give the workflow a name. It's called Testing because what I want the workflow to do is test my airline application. Then I specify an on key to mean when should this workflow run. And here, I have said on push, meaning anytime someone pushes their code to GitHub, we would like to run this workflow. 

Every workflow consists of some jobs. And so what are the jobs? What tasks should happen anytime that I try and push to this repository? Well, I've defined a job called test_project. And this is a name that I chose for myself. You can choose any name for a job that you would like. 

And now I need to specify two things for what happens on a job. One thing I need to specify is what sort of machine is it going to run on? That GitHub has its own virtual machines, otherwise known as VMs, and I would like to run this job on one of those virtual machines. And there are virtual machines for various different operating systems. Here I'm just saying, go ahead, and run on the latest version of Ubuntu, which is a later version of Linux that I would like for this test to run on. 

And then for the job, I specify what steps should happen where I can now specify what actions should happen when someone tries test a project when I try and run this job. And here I'm using a particular GitHub action. And this is a GitHub action written by GitHub called actions/checkout. 

And what this is going to do is it's going to check out my code in the Git repository and allow me to run programs that operate on that code. And you can write your own GitHub actions if you would like. But here, all we really need to do is go ahead and check out the code, as by looking at what's on the branch that I just pushed to. And then I'm going to run Django unit tests. This is just a description for me to know what's going on in this particular step. 

And here is what I would like to run. I'm going to first go ahead and install Django because I'm going to need to install Django to be able to run all of these tests. But after-- and if there are other requirements, I might need to install those requirements as well. But the airline program is fairly simple. All we really need in order to run the tests is just Django. So I'll go ahead and install Django. And then I'll run python3 manage.py test. I would like to test-- run all of the tests. And the way I can do that is just by providing this manage.py command to say that I would like to run all of the tests on this particular application. 

So this configuration file altogether now is going to specify a particular workflow, the workflow that says that every time I push to the GitHub repository, what I would like to happen is I would like to check out my code inside of the Git repository. So on some Ubuntu VM, GitHub is going to check out my code, and it's going to run these commands. It's going to install Django. And then it's going to test my code. And it will then give back to me what the response is after I do that. 

So let's go ahead and test this. And in particular, let's run it on a program where the tests are going to fail. So I might say, for example, let's go into flights and models.py. And let's go to my is_valid_flight function from before and change it back to that version that didn't work. That before it was something and something. I'll change it to something or something. That as long as the origin is not the destination or the duration is greater than 0, we'll count that as valid. But we know that that's wrong. That should not work. 

So here's what I'll do. I'll go ahead and first say git status, see, all right, what's changed? And it seems that, all right, I've changed-- I've modified models.py, which makes sense. I'll go ahead and git add. I'll add dot. We'll just add all of the files that I might have modified. I'll commit my changes. Say go ahead and use wrong valid flight function. That's what I'm going to do. 

And now I'm going to push my code to GitHub. I added it. I committed it. I pushed it. That now then pushes my code to GitHub into a repository called airline that I already have. And now, if I go ahead and go to GitHub, and I go to my airline repository, what you'll notice is that we've mostly been dealing with this Code tab. But GitHub gives us other tabs as well that are quite useful as you begin to think about working on a project in larger team. 

So in addition to looking at the code, we have issues. Issues are ways for people to just report that something is not quite right, or there is a feature request that we have for this particular code base. So the issues might maintain a list of all of the pending action items for a particular repository, things that we still need to deal with. And once those issues are dealt with, the issues can be closed. So I have no issues here as well. 

Pull requests are people that are trying to merge some part of the code from one branch into another branch. So you might imagine on a larger project, you don't want everyone merging things into master all at the same time. You might have people working on their own separate branches. And then when they feel confident and happy with their code, then they can propose a pull request to merge their code into the master branch. 

And that allows for various other features, like the ability for someone to offer a code review-- to be able to review the code, write comments, and propose suggestions for what changes should be made to a particular part of the code before it gets merged into the master branch. And that's another common practice with regards to working on a GitHub repository or any other larger project that you're controlling using source control is this idea of code reviews, that oftentimes, you don't want just one person making the changes without anyone's eyes on that code. But you want a second pair of eyes to be able to look things over, make sure the code is correct, make sure it's efficient, make sure it's in line with the practices that the application is using. And so pull requests can be quite helpful for that. 

And then this fourth tab over here represents GitHub Actions. These are the various different actions or workflows that I might want to run on this particular repository. And what we'll see here is that if I go to the Actions tab now, what I'll see is here is my most recent testing actions. So anytime I push, I get a new testing action. This one was from 29 seconds ago. I'll go ahead and click on it and see what's within it. All right. 

Here was the job that I ran, test_project. I see that on the left-hand side. You'll notice this big red X in the left-hand side of this workflow. Means something went wrong. So I'd like to know what it is that went wrong. I'll go ahead and click on test_project. And here within it, these are all of the steps, the things that happened when we actually ran this particular job. 

First the job sets up. Then the checkout action goes ahead and checks out my code because we need access to my code to be able to run it. Here was the step I defined-- run Django unit tests, which was going to install Django and run those tests. It has an X next to it, indicating something went wrong. And I see down below, annotations, 1 failure. So all over the place, GitHub's trying to tell me that something went wrong. It failed two minutes ago here. 

I'll go ahead and open this up. And what I'll see is the first thing that happened is we installed Django. And that seems to have worked OK. But down below, what you'll see is the output of running these unit tests, that we see FAILED (failures-2). And now I can see, here are the unit tests that failed. We failed the invalid flight destination test. We failed the invalid flight duration test. 

And as before, I can see in GitHub's user interface what those assertion errors are. I can see a true is not false. True is not false, those were the problems that happened when I tried to run this particular test suite. And now others who are also working on this repository can see as well what the results of these tests are and can offer suggestions, can offer ways that I might be able to fix the code in order to deal with that problem. But now I know that this particular test failed. 

And if I go back to the main code page for this GitHub repository, I'll see that next to this commit, there is a little x symbol. And that little x symbol next to the commit just tells me that the most recent time I tried to commit, something went wrong. They ran the workflow, and there was an error. And so I'll immediately see for this commit-- and I can go back and look at the history of commits and see which ones were OK and which ones had a tendency to cause some sort of problem. 

So this one, it appears caused a problem. And we know why. It caused a problem because of this condition, something or something else. So I can fix it. I'll change the or to an and. I'll go ahead and git add dot. git commit. Say I will fix valid flight check. If I do git status just to check out what's going on right now, I'm ahead of the master branch by 1 commit. That's exactly what I would expect. And now I'll go ahead and push my code to GitHub by running git push, saying, all right, let's push this update. And now, hopefully, we're going to pass the workflow now. 

Now I go back to the repository. I refresh the page. Here's my latest commit-- fix valid flight check. You notice here, there's an orange dot instead of the red x as before. This dot just means the tests are currently pending. The workflow is in progress because it takes some time for GitHub to be able to start up the VM, to be able to initialize the job, to check out my code, to run all those tests. It does take some time. 

But if I go back to the Actions tab, I'll see that, all right. This time, for testing, we get a green check mark. Everything seems to be OK. I go to test_project just to see it. And now I notice the green check mark next to Run Django unit tests means that the unit tests have passed as well. If I open those up, now I see at the bottom the same output that I saw before when I was running those unit tests on my own machine. We ran 10 tests, and everything was OK. And that tells me that these tests have passed. 

So GitHub Actions have the ability to allow for certain jobs to happen, certain work to happen anytime you push code, anytime you submit a pull request or on various different actions that might happen on a GitHub repository. And they're very helpful for being able to implement this idea of continuous integration because it means you can make sure that when you're merging code from some developer's branch into the main branch that everyone's merging their code into, you can verify that those tests can pass. And you can add rules to say that you don't want to allow anyone to merge code into the branch if the tests don't pass, to guarantee that any code that does get merged is going to pass all of those tests as well. 

And so that can definitely help the development cycle, make it easier to ensure that changes can be made quickly. But as we make those changes quickly, we're not going to lose accuracy and validity within our code, that we can make sure that our code still passes those tests by automating the process of running those tests altogether. 

So other than continuous integration then, we now talk about this idea of continuous delivery, these short application cycles where we would like to very quickly be able to deploy our application onto some sort of web server. And when we're deploying applications to a web server, there are things that we need to think about. We need to think about getting our program that was running fine on our computer working on a web server as well. 

And this can just be fraught with headaches and all sorts of configuration problems because you might imagine that the computer that you are using is not necessarily going to be the same as the computer that on the cloud, the computer in the server where your web application is actually running. It might be running a different operating system. It might have a different version of Python installed. If you have certain packages working on your own computer, those same packages might not be installed on the server. So we run into all sorts of various different configuration problems where you can be developing, deploy your code, and realize that it doesn't work on the server because of some sort of difference between what's happening on your computer and what's happening on the server. 

And this becomes even more problematic if you're working on a larger team, you and multiple other people working on a software project, but you each have different versions of various different packages or libraries installed, and those different versions have different features and might not all work and cooperate with one another. And so we need some way in order to be able to deploy applications efficiently and effectively to be able to standardize on just one version of the environment, one version of all these packages, to make sure that every developer is working on the project in the same environment. And once we deploy the application, it's going to be working in the same environment as well. 

And the solution to this comes in a number of possible options. But one option is to take advantage of a tool like Docker, which is some sort of containerization software. And by containerization software, what we're talking about is the idea that when we're running an application, instead of just running it on your computer, we're going to run it inside of a container on your computer. 

And each container is going to contain its own configuration. It's going to have certain packages installed. It's going to have certain versions of certain pieces of software. It's going to be configured in exactly the same way. And by leveraging a tool like Docker, you can make sure that so long as you provide the right instructions for how to start up and set up these containers, then if you are working on the application and someone you're working with, some colleague that's also working on the same project, so long as you're using the same instructions for how to set up a Docker container, you're going to be working in the identical environments, that if a package is installed on your computer, in your container, it's going to be installed in your colleague's container as well. 

And the advantage of this too works with this idea of continuous delivery. When you want to deliver and deploy your application to the internet, you can run your application inside of that exact same container, set up using the exact same set of instructions, so that you don't have to worry about the nightmare headaches of trying to make sure that all the right packages and all the right versions are, in fact, installed on the server. 

Docker might remind you of the idea of a virtual machine or a VM if you're familiar with that concept. GitHub uses VMs, for instance, when running its GitHub Actions. They are, in fact, different. A VM is effectively running an entire virtual computer with its own virtual operating system and libraries and application running on top of that all inside of your own computer. So a virtual machine ends up taking up a lot of memory, taking up a lot of space. 

Docker containers, meanwhile, are a bit lighter-weight. They don't have their own operating system. They're all running still on top of the host operating system. But there is this Docker layer in-between that keeps track of all of these various different containers and keeps track of for each container such that every container can have its own separate set of libraries, separate set of binaries, and an application running on top of that. 

So the advantage then of containerization is that these containers are lighter-weight than having an entire virtual machine. But they can still keep their own environment consistent such that you can feel confident that if the application is working in a Docker container, you can have that Docker container running on your computer, on someone else's computer, on the server to guarantee that the application is going to work the way that you would actually expect it to. 

And so how exactly do we configure these various different Docker containers? Well, in order to do so, we're going to write what's called a Docker file. So to do this, I'll go ahead and go into airline1. And I'll open up this Docker file. And the Docker file describes the instructions for creating a Docker image where the Docker image represents all of the libraries and other installed items that we might want to have inside of the container. And based on that image, we're able to create a whole bunch of different containers that are all based on that same image, where each container has its own files and can run the web application inside of it. 

So this Docker file, for example, describes how I might create a container that is going to run my Django web application. So first, I say FROM python:3. This happens to be another Docker image on which I'm going to base these instructions, that this is going to be a Docker image that already contains instructions for installing Python 3, installing other related packages that might be helpful. Oftentimes, when you're writing a Docker file, you'll base it on some existing Docker file that already exists. 

So here I'm saying go ahead and use Python 3. And now what do I want to do in order to set up this container? Well, I want to copy anything in dot, in my current directory, into the container. And I have to decide, where in the container am I going to store it? Well, there-- I could choose to store it anywhere. But I'll just store it in /usr/src/app, just some particular path that will take me to a directory where I am going to store the application. But you could choose something else entirely. 

So I copy all of the current files in my current directory. So that will include things like my requirements file, my manage.py file, my applications files, all my settings files. Everything inside of the directory, I would like to copy into the container. Then I'm saying WORKDIR, meaning change my working directory, effectively the same thing as something like CD on your terminal to move into some directory. I would like to set my working directory equal to that same application directory, the application directory inside of the container that now contains all of the files from my application because I copied all of those files into the container. 

Now once I'm inside of this directory, I need to install all of my requirements. So assuming I've put all my requirements like Django and any other packages that I need inside of a file called requirements.txt, I can just run the command, pip install requirements.txt. And then, finally, inside the Docker file, I specify a command. And this is the command that should run when I start up the container. Everything else is going to happen initially when we're just setting up this Docker image. But when I start up the container and actually want to run my web application, here is the command that should run. 

And I provide it-- effectively it's like a Python list where each word in the command is separated by a comma, where here I'm saying the command that I would like to run, when you start up this container is python, manage.py, runserver. And here I'm just specifying on what address and what port I would like it to run. And I'm running it on port 8000, for example. But I could choose another port that I would like to run instead. 

So what's going to happen then is that when I start up this Docker container, it's going to, if it needs to, go through these instructions and make sure that it sets up the container according to these instructions, make sure that we've installed all of the necessary requirements, make sure that we're using Python 3. And anyone using the same Docker file can generate a container that has all the same configuration on it. So we don't have to worry about configuration differences between me and someone else who might not have the exact same computer setup that I do. And the nice thing about this is that it can run on Mac and Windows and Linux. So even people running on different operating systems can still have containers that all have the same configuration, that all work in the same way just to speed up that process. 

Now so far, when we've been building Django applications, we've been using a SQLite database. SQLite database just being a file that is stored inside of our application. And this file-based database allows us to create tables, insert rows into it, delete rows from it. In most production environments, in most real web applications that are working with many, many users, SQLite is not actually the database that is used. It doesn't scale nearly as well when there are many users all trying to access it concurrently. Oftentimes, in those sorts of situations, you want your database hosted elsewhere on some separate server to be able to handle its own incoming requests and connections. 

And we talked about a couple of possible databases we could use instead of SQLite, things like MySQL, things like Postgres, or various different SQL-based databases. So imagine now I want to deploy my application. But instead of using SQLite, I would like to use Postgres, for example, as the database server that I would like to run. Well, that would seem to be pretty complicated for me to test on my own because now in addition to running my web application in one server, effectively, I also need another server that's running Postgres, for example, such that I can communicate with that Postgres database instead. 

And that's going to be even harder for other people to be able to work on as well. Potentially, it might be difficult to get the server to work in that way too. But the nice thing about Docker is that I can run each of these processes in a different container effectively. I can have one container that's running my web application using this Docker file right here. And I can have another container that's just going to run Postgres. And as long as other people also have access to that same container for running Postgres, they can be working in an identical environment to the one that I am working in as well. 

And so there's also a feature of Docker known as Docker Compose. And what Docker Compose lets us do is allow us to compose multiple different services together, that I would like to run my web application in one container, and I would like to run a Postgres database in another container. But I would like for those containers to be able to talk to each other, to be able to work together whenever I start up the application. 

So if I'd like to do that, in order to run this application on my computer and have both the web application and Postgres installed, I can create a Docker Compose file which looks like this. Here I'm specifying using version 3 of Docker Compose. Here I specify, again, using a YAML file. Much as in my GitHub workflows were formatted in YAML just as a configuration file, docker-compose.yml is a YAML file that describes all of the various different services that I want to be part of my application, where each service is going to be its own container that could be based on a different Docker image. 

Here I'm saying that I have two services, one called db for database, one called web for my web application. The database is going to be based on the Postgres Docker image, image that Postgres wrote that I don't have to worry about. Someone else has written the Docker file for how to start up a Postgres container. 

Here, though, for the web application, that's going to be built based on the Docker file in my current directory, the Docker file that I have written. And then down below, I've just specified that my current directory should correspond to the app directory. And then I've specified when I'm running this on my own computer, I would like port 8000 on the container to correspond to port 8000 on my own computer just so that I can access port 8000 in my browser and access port 8000 inside the container. It just lets my computer actually talk to the container so I can open up the web application in my web browser, for example, and actually see the results of all of this. 

So here, then, I've created two services, a database and web. So now let's actually try starting up these containers. I'm going to first go into my airline1 directory. And I'm going to say docker-compose up to mean go ahead and start up these services. I'll press Return. And what you'll see is we're going ahead and starting up two services. I'm starting up the database service. And I'm starting up the web service. 

And now as a result of all of this, I've started up the application. And I started it on port 8000. So if I go to 0.0.0.0 slash 8000 or colon 8000 slash flights, that's going to take me to the Flights page. And now this is running, not just on my own computer, but inside of a Docker container. Now, of course, right now, there are no flights inside of this page because I haven't actually added anything to the database yet. So I could do that if I wanted to. But how do I do that? 

Well, I needed to go into slash admin to say, like, let me log in and go ahead and create some sample flights. But I don't have a log in yet because I need to create a superuser account. And I can't just like inside of my airline1 directory say, python manage.py createsuperuser the way that I used to because this is running in my terminal on my computer. Whereas, what I really want to do is go into the Docker container and run this command there, inside of the container. 

So how can I do that? Well, there are various different Docker commands that I can use. docker ps will show me all of the Docker containers that are currently running. So I'll go ahead and shrink this down a little bit. I see two rows, one for each container, one for my Postgres container that's running the database, one for just my web application that's running as well. Each one has a container ID. So I want to go into my web application container in order to run some commands inside of that container. 

So I'm going to copy its container ID and say, docker exec-- meaning go ahead and execute a command on the container-- dash it will make this interactive. Here's the container ID that I would like to execute a command on. And the command I want to execute is bash, passing the dash l flag, but bash to say, I want to run a bash prompt. I want to be able to interact with a shell so that I can run commands inside of this container. 

So I press Return. And now what you'll notice is that I am inside of my container in the user source app directory, that directory that contained all of the information about this web application. I type ls. And what I'll see is here, all the files inside of this container now. And now I can say something like python manage.py createsuperuser. And now it's going to let me create a superuser. 

So I'll create a user inside of my web application called Brian. I'll give it my email address. I'll type in a password. And now we've created a superuser. And you can run other commands here. If you wanted to migrate all of your migrations, I could say python manage.py migrate. And it turns out I've already done that, so I didn't actually have to do it again. 

But you can run any commands that you can run them on your computer. But now you can run them inside of the Docker container instead. I'm going to press Control D just to log out, get out of the container and get back to my computer. But now I've created a superuser, so I could go ahead and sign in to Django's admin. And now I can begin to manipulate this database, which is a Postgres database running in a separate container. But the nice thing about it is that I can start them both up together just by running something like docker-compose up, for example. 

So Docker can be quite a powerful tool for allowing us to very quickly ensure that an application is running in the environment that we expect it to be running, to make sure that all of the right libraries are installed, make sure that all the right packages are installed as well, that the configuration between my development environment and the environment that's running on the server are the same as well. 

So those then were just some of the best practices for how you can go about developing a program now that we have the tools to do so. We have a lot of tools for being able to develop these web applications. But as our programs start to get more complex, it will be increasingly important to test them, make sure that each various different component of our web application behaves the way that it is expected to behave, and then taking advantage, especially in bigger teams, of CI/CD, Continuous Integration, Continuous Delivery to make incremental changes, and make sure each of those incremental changes, in fact, works on the web application. 

And then CD, Continuous Delivery, to say that rather than wait and then deploy everything all at once, let's deploy things incrementally as well. Let users more quickly get access to the latest features and more quickly find out if something went wrong. We can better identify what it is that went wrong if we've deployed things incrementally rather than waiting a long time in order to do so as well. 

So these are some of the best practices in modern software application development, not only for web applications but for software more generally. Next time, we'll consider other challenges that might arise as we go about trying to make web applications that are used by more and more users, in particular, taking a look at challenges that will arise in terms of scalability as the programs get bigger and also security of what security vulnerabilities open themselves up as we begin to design our web applications using Python and JavaScript. So more on that next time. And we'll see you then.