CS50 Live first looks today at GitLab.
GitLab is a popular source code hosting site, much like GitHub.com,
that developers can use in order to store their code centrally,
in order to version control it, share multiple copies,
as well as share it with other users.
Unfortunately, GitLab ran into a bit of an issue very recently.
The whole incident started when they saw this.
They support a feature known as Snippets,
whereby users can create snippets of code,
much like GitHub Gist, whereby users can upload small snippets of code
to share them with other people.
Unfortunately, having some 1.5 million snippets of code
created over the course of just a few days?
Not normal.
In fact, this seemed to be the result of some spamming behavior
by some adversarial folks online.
Moreover, GitLab also notice that one or more spammers seemed to be using GitLab
inappropriately, as a content delivery network, or CDN,
whereby they were serving up files in ways that they shouldn't.
Now unfortunately, these kinds of attacks
resulted in a bit of a ripple effect on their back-end databases.
Particularly, GitLab posted the following.
"We are experiencing issues with our production database
and are working to recover."
Now unfortunately, just minutes later did they
post, "We accidentally deleted production data
and might have to restore from backup."
Now what exactly happened?
Well, it's quite common for databases to be replicated from one to another
so that you have a primary and a secondary,
the latter of which is a backup of the former in real time.
As part of the diagnosis challenge for figuring out
why the databases were slowing down in terms of this replication,
one of GitLab's system administrators very deliberately
executed a command quite like this.
Now what is this command?
Well at the front of this command is "sudo,"
which says execute the following command with administrative, or root,
privileges.
What is the command to be executed?
Well, rm -rf apparently.
And rm, you might know, is to remove files or folders from a system.
-r though means recursively.
Delete the following thing recursively, so that any directories inside of that
also get deleted.
And unfortunately, the "f" -rf means forcibly,
which means don't even prompt the human to confirm or deny
that he or she wants to do this.
Now the system administrator meant to execute this command deliberately
on their secondary database, db2.cluster.gitlab.com,
so that they could resume then the replication from their primary
to their secondary database.
Unfortunately, it appears to have been late at night,
and this was a stressful situation, and darn it
if this command weren't executed on db1.cluster.gitlab.com,
the actual primary database.
Now no big deal, surely we have backups all over the place.
So we can just restore from backup, and our customers
will be perfectly happy and on their way.
Unfortunately, out of five backup or replication techniques deployed,
GitLab reported that, "None are working reliably or set up in the first place."
Indeed, if you'd like to read their whole post-mortem in which they
discussed exactly what went wrong, and how, you can check out this URL here.
But the moral of the story, for our purposes,
is please, please beware the rm -rf, especially
if you're not just deleting some directory of your own,
potentially your customers as well.