Day 7

Check out some of Colton’s music here.

In other news, the DNS for cs50.harvard.edu should be fully updated by now across HBS' servers.

Tor is an anonymization network, where people’s data is bounced between other people on the network, so tracing where the data really came from becomes much harder.

There’s this device available, called an Anonabox, that you can just plug into to access the Tor network, instead of downloading and configuring software. But there were major security flaws, like the lack of password protection for its WiFi and the ability for anyone nearby to snoop on its traffic, according to Wired. So the takeaway here is to not trust mysterious projects so easily.

Are gem installs in ruby similar to installing frameworks like Flask in Python?

What’s the difference between an object oriented language and the languages we’ve talked about so far?

Christina introduces herself as a course assistant who’s been at office hours to answer questions, and is also a full-time PM at TripAdvisor.

Are there inherent advantages in using procedural, functional, and object-oriented languages in the same way that there are advantages in compiled vs. interpreted languages?

Are all programming languages in English?

What were the key takeaways of today’s [yesterday’s] class? I followed the demos, but was not sure what the generalizable principles are. I took away:

I’m afraid that I cannot access "Discuss" so would like to mention my problem here. I could run all python programs in the folder "src6" well except froshims-3 and hello-2.

For this assignment, I changed the sentence within the hello.html file to something other than hello, world and it did not update on the website. Is this because I had to change the python code instead of the html file?

Stack Overflow, a discussion board website that many developers use to ask and answer technical questions, recently released its annual Developer Survey.

This year they had about 26,000 or respondents, and there are facts like the country they are from, the number of developers per capita, age (younger, towards the 20s and 30s), gender (predominantly male), experience (women have less, but it’s a sign that more women are beginning to enter the field), education (people tend to be self-taught), most popular technologies (JavaScript, SQL, and Java are in the lead), and more.

Though remember that this sample is of the developers who both use Stack Overflow and respond to their surveys, so it might be fairly biased.

Yesterday, all the pages we generated were thrown away. We could use caching, which saves the more popular pages so we save some processing time next time we need the page.

But we need a place to store data, not just for caching but in general, like a customer’s purchasing history.

One of these technologies is called NoSQL, named because it’s not SQL, an older query language (language that interacts with a database).

NoSQL involves storing data in a format that look like this:

{
    "_id":"02134",
    "city":"ALLSTON",
    "loc":[
        -71.13286600000001,
        42.353519
    ],
    "pop":23775,
    "state":"MA"
}

This format is called JSON, JavaScript Object Notation, and is just syntax and patterns for saving data.

Notice that this chunk of data, from an abstract level, seems to represent the city of Allston. And programmers would think of it as an object, with characteristics that other city objects would also have.

So we have the curly braces that contain everything, and inside a list of key-value pairs, as in a word in quotes, "_id" (the key), a colon, :, another word in quotes, "02134" (the value), and then a comma, ,, signalling the next key-value pair.

The value doesn’t necessarily need to be a string in quotes. For loc, it looks like the value is an array, or list, given that there are square brackets [] around two things, separated by a comma ,. pop too has a value of just 23775.

To store lots of these objects, we might separate them with commas after the curly braces. But there are no specifications, or schema, for what goes inside. In noSQL, we don’t know what types of data we might have, but we have the ability to store nested data.

For example, let’s look at this picture:

We could represent someone in an object like this:

{
    name: "Jihad",
    age: 25,
    phone: null,
    address: "123 Main Street Allson MA 02163"
}

So we can improve it like this:

{
    name: "Jihad",
    age: 25,
    phone: null,
    address: {
        street: "123 Main Street",
        city: "Allston",
        state: "MA",
        zip: "02163"
    }
}

If we had just an Excel file with columns, we might have something that looks like this:

And we have lots of people from the same city, so storing the same information about the city (city name and state and zip) seems unnecessary.

So all we need is a zip code for the person, but information somewhere else, perhaps in another file:

{
    name: "Jihad",
    age: 25,
    phone: null,
    address: {
        street: {
            number: 123,
            basename: "Main",
            suffix: "Street"
        }
        city: "Allston",
        state: "MA",
        zip: "02163"
    }
}

____

{
    zip: "02163",
    city: "Allston",
    state: "MA"
}

So in our original example, we can do this:

SQL is yet another language that allow us to query a database to manipulate data efficiently in a large data set.

A SQL database is like an Excel file with multiple sheets, where each sheet is called a table in SQL.

Queries include DELETE, INSERT, UPDATE, and SELECT.

We can do this in a command prompt:

But there also exists a graphical user interface called phpMyAdmin:

There are data types in SQL like:

Someone’s name might be VARCHAR, which means that it will be a variable length string made up of characters. Since people might have shorter or longer names, we should use VARCHAR as opposed to CHAR, which is a fixed length. But we do have to specify a maximum length, and the higher that length is, the slower the database’s performance will be, since we wouldn’t be able to randomly access elements. (Think back to arrays that had elements that were all the same size, so we could jump around easily.)

INT is a 32-bit integer, and a BIGINT is 64-bits.

A DECIMAL allows us to set the number of significant digits of the number, so we can have decimal numbers without the imprecision of floats.

A DATETIME stores a date and time in the default format of YYYY-MM-DD hh:mm:ss, though we can format it in different ways.

There are also indexes in SQL:

Finally, there are different storage engines for databases, which just means that the binary zeroes and ones are stored on the hard drive somewhat differently by each of these:

The Memory storage engine means that all of our data will be stored in RAM, making it good for performance and temporary storage, like for a cache.

Archive means the data is pre-compressed, so we’d want to use it for data we write but not read too often, since it will take some time to decompress when we do but saves space on our hard drive.

InnoDB and MyISAM also have different characteristics that give them advantages and disadvantages.

If you have a Facebook username, you can see what your user id number on Facebook is by going to a URL like this (David is showing off how low his is):

Suppose you and your roommate share a fridge, and you both like milk. So you come home one day, and your roommate is still in class, and you see that there’s no more milk in the fridge. So you leave for CVS to buy some milk. Meanwhile, your roommate gets home and realizes that there’s no milk too, and happens to go to the nearby supermarket, and by the time you both return, you have two gallons milk instead of one. This was because you and your roommate didn’t communicate with each other (texting didn’t exist back in David’s time, so this story made more sense then).

This problem also comes from the fact that there is a lag between the time you see there is no milk and the time you get back with it.

In the database world, operations that happen simultaneously with no lag are called atomic, so the milk replacement algorithm isn’t atomic.

One of you could have marked in the fridge that you were going to get milk, and analogously in the database world we "lock" a database when we want to do something to it without the state being changed by someone else.

In SQL there is syntax like this (which we won’t worry about the details of):

INSERT INTO table (id, symbol, shares) VALUES(7, "GOOG", 10)
    ON DUPLICATE KEY UPDATE shares = shares + VALUES(shares)

We could also do what’s called a transaction:

START TRANSACTION;
UPDATE account SET balance = balance - 1000 WHERE number = 2;
UPDATE account SET balance = balance + 1000 WHERE number = 1;
COMMIT;

MongoDB uses locking, and SQL supports both locking at the table level and the row level, so using transactions that only lock certain rows makes for better performance since other users can update the other rows.

What you’ll do tonight is to download a nice big database, with files that have queries that look like this:

After we import all this data into Cloud9, we’ll use a tool called phpMyAdmin. We see that our databases are on the left, and most of them already existed, except for the one called employees:

Now if we click on it, we see lots of information:

If we click around some more, we see a graphical version of what we just saw:

It’s also easy to create a table:

There are more options for our fields: COLLATION being the encoding, like ASCII or others; ATTRIBUTE with an option of UNSIGNED, meaning that numbers can’t be negative; NULL if we want any to be optionally empty, which we select for major; and INDEX, so we’ll have id be our PRIMARY index, and major be INDEX too, so we can search on it more quickly:

A_I is short for AUTO_INCREMENT, which means that the database will automatically increment the field every time a new row is added. This is useful for ID numbers, when we don’t care what it actually is, but that it’s unique for every user. And we can also select the storage engine at the bottom:

So tonight we’ll play around with databases and not get to learn every little detail, but just get a better understanding of how they work.

SQL injection attacks are also a concern. Though the familiar Harvard PIN system isn’t vulnerable to this attack, we’ll use it for the sake of discussion.

The PIN login screen looks like this:

Suppose that the code, somewhere behind this page, includes bits like this:

$username = $_POST["username"];
$password = $_POST["password"];
query("SELECT * FROM users WHERE username='{$username}' AND password='{$password}'");

First we get the variables $username and $password from some form, and substitute them into our query, which looks (and is) correct.

But a user could input something like this: (David changed the PIN field’s usual bullets to actual text to reveal what the user has typed.)

If we substitute that info into the form, the query will now look like this (note that skroob has left off single quotes on the outside of his password, assuming that the code for the query will, which it does):

$username = $_POST["username"];
$password = $_POST["password"];
query("SELECT * FROM users WHERE username='skroob' AND password='12345' OR '1' = '1'");

The takeaway here is that programmers should code defensively and anticipate users placing evil syntax that changes what they’re trying to do.

One fix is to disallow certain characters in the input, that we check for before we substitute it into our query, but this keeps us from choosing certain characters in our passwords or other input.

The fix is to escape input, or convert the characters to something that looks like this:

$username = $_POST["username"];
$password = $_POST["password"];
query("SELECT * FROM users WHERE username='skroob' AND password='12345\' OR \'1\' = \'1'");

So a SQL injection attack can happen when a programmer places strings into queries carelessly. But there are libraries with functions that escape input that we can just use pretty straightforwardly.

And now you can understand this xkcd.