In other news, you might have noticed that half of the internet
went down recently, and somehow this was Amazon.com's fault. Well,
it turns out that Amazon is not just the e-commerce site that you
might know and use.
They're also one of the world's largest cloud providers,
where cloud computing is this technique whereby
other people can run servers, and have hard drives,
and more services somewhere in the world.
And you as a customer can essentially rent those services,
so that your website your application isn't hosted by you in your own data
center, but in Amazon, or Microsoft, or Google's own data center.
Now unfortunately, something went wrong with one of Amazon's cloud services,
something called S3, simple storage service.
Such that the result, according to one popular ISP called Level Three,
was outages across the US, if not beyond,
because these websites pictured-- here is this heap map--
were relying on at least one of Amazon's services.
In fact you might notice some familiar names among the websites affected.
Codecademy, Coursera, Docker, Giphy, GitLab, GitHub, Heroku, Imgur,
Kickstarter, Medium, Quora, Slack, Travis CI, and many, many more.
In fact, perhaps best was the irony of a website called, Is It Down Right Now?
being down right now.
This is a website that typically allows you to check other websites are down,
but if you actually visited that website during Amazon's outage, would
you have seen an error like this.
Now fair is fair.
Some of CS50's own infrastructure also went down during this incident,
and that's because CS50 stores not only some of its largest video files,
but also the data related to its video player,
on Amazon S3, the cloud service in question.
So in fact, during that outage, if you tried to watch one of CS50's videos
in its own player, you probably would have seen an error screen
quite like this.
Because the video player, which is JavaScript, or client-side based,
wasn't able to pull the requisite data from Amazon servers.
Now what is Amazon S3, and what technically went wrong here?
Well at first glance, it's all pretty technical sounding.
"Amazon S3 is a simple key-based object store,"
according to Amazon's documentation.
"Keys can be any string, and can be constructed
to mimic hierarchical attributes."
But what does that mean?
Well, let's tease this apart.
It's a key-based object store.
Now an object, in this case, just refers to files, where
a file is just a whole bunch of bits.
But Amazon kind of abstracts away the notion of a file,
so there isn't really the notion of files, and folders, and all of that.
There's just objects, which are, for all intents and purposes, files.
But they are accessible via keys, which typically are strings,
much like in a hash table, if familiar.
You access some value by way of some unique key.
So for instance, in CS50, we posted this first video from fall 2016 at this URL
here.
It's an mp4, which is a video file.
Now it turns out that the video file actually lives
on a server that similarly named, but notice what's
in it, cdn.cs50.net.s3.amazonaws.com.
Which is to say that indeed, within CS50's own CDN--
content delivery network-- the data itself comes from Amazon.
Now what about the key that uniquely identifies
our objects, or videos, or other files?
Well this string here with slashes, and words, and so forth, looks like a file
inside of a bunch of folders, but--
that's fine to think about it that way-- but it really
is just a unique string that resembles a file path.
We've adopted a scheme whereby it looks like these are folders,
simply because it keeps our data nicely hierarchical.
So what went wrong, and what did users see?
Well, if you visited Amazon's status page
on the day in question-- or the days prior to the days in question--
you would have seen beautiful green check marks,
from February 27 on back, whereby all was well.
Green check means good for the S3 storage service.
Unfortunately, on February 28, did this thing rear its head.
And suffice it to say, red icon bad.
In fact, in this case it means half of the internet would appear to be down.
Now, you can read more on the details of this story,
but let's take a look at a few of the key moments.
At 2:37 PM Eastern Time on February 28 did Amazon report this.
"We can confirm high error rates for requests
made to S3 in the US EAST-1 Region.
We've identified the issue, and are working to restore normal operations."
Well what does that mean?
Well US EAST-1 Region is simply one of Amazon's data centers.
Like a lot of cloud providers, they have data centers--
buildings with lots of servers and lots of hard drives and more--
all over the world.
And US EAST-1 happens to be one of the most popular.
It's physically located in northern Virginia, in the United States.
And because CS50 isn't all that far away, in Cambridge, Massachusetts,
much of our assets live in US EAST-1 one by choice.
In fact, it's a trade-off.
We could absolutely replicate our data across multiple, multiple regions,
and have been much more tolerant against this kind of fault,
but it's a trade-off between how much storage you need,
how much money it might cost, and how much complexity you have to introduce.
So we very consciously put much of our data in US EAST-1
so that it's as close to campus as possible.
Now, Amazon explains that the reason S3 became inaccessible,
and in turn, so many of these customers--
CS50 among them-- went offline was as follows.
"The team was debugging an issue causing the S3 billing system
to progress more slowly than expected."
OK.
"An authorized S3 team member, using an established playbook,
executed a command which was intended to remove a small number of servers
for one of the three S3 subsystems that is used by the building process."
OK.
"Unfortunately, one of the inputs to the command was entered incorrectly,
and a larger set of servers was removed than intended."
In other words because of human error.
Literally a typographical error in the equivalent of a terminal window.
Mistyping a command, did Amazon take offline--
not just a few servers meant to diagnose some problem--
but a huge number of servers.
All of which then need to be rebooted, which takes some time,
and which explains the downtime.
In the real world, this might be like if Amazon were hungry
for a little bit of chocolate, and so went over
to the chocolate serving station, and picked up this here X-ACTO knife,
and just wanted to take a tiny little piece of the internet offline
so as to enjoy some chocolate.
And so you might just take off a little corner like this.
Mm-mm, that's a good server.
But that's not what Amazon in fact did.
Amazon, because of a mistyped command, for which apparently there was not
a sufficient prompting process to say, "Human are you
sure you want to take down all these servers?"
Amazon effectively took out this here saw, turned it on, and bit off
half of the internet.
Mm, that's good internet.