Cloud computing-- it's this term that rather
swept onto the scene in recent years.
And it sounds like it's some new and trendy technology.
But in reality, it's really just a very nice packaging
up of a whole number of technologies that have actually
been with us for some time.
In fact, cloud computing, in its simplest form,
can really be thought of as just outsourcing
the hosting of your applications and really outsourcing
the hosting of your physical servers to someone else-- put another way,
renting space and renting time on someone else's computers.
But these days, we just have so much computational capabilities-- that is,
our computers are so fast, our CPUs are so many, and we have so much RAM--
that new and fancier technologies have lent themselves
to this trend of hosting all the more software
and putting all of the more hardware off-site in the so-called cloud
so that companies, both big and small, no longer need
to host their own physical hardware or even a whole number of roles
in their own local companies.
And so what we'll do now is dive into cloud computing,
look at some of the problems it solves, look
at some of the opportunities it affords, but ultimately,
take a look from the ground up at what's underneath the hood
here so that by the end of this, we have a better
understanding of what the cloud is, why it is useful,
and what it actually is not.
So with that said, let's start with a simple scenario.
Of course, the cloud perhaps derives its origins
from how the internet, for some time, was drawn,
which was just this big, nebulous cloud, in that it doesn't really
matter what's inside that cloud.
Although at this point, you most surely appreciate that inside of this cloud
are things like routers, and running through those routers
are packets, both TCP/IP and the like.
And underneath the hood, then, of this cloud
is some transport mechanism that gets data from point A to point B.
So what might those point A's and Point B's be?
Well, if this here is my little, old laptop, connected somehow
to the internet here, and maybe down here there
is some web server on which lives a whole bunch of web pages--
maybe it's my email.
Maybe it's the day's news.
Maybe it's some social media site or the like.
I, at point A, want to somehow connect to point B down here.
Now, it turns out it's not all that hard to get a website up
and running on the internet.
You can, of course, use any number of languages.
You can use any number of databases.
And you can do it with relatively little experience,
just getting something on the internet.
In fact, it's not all that hard, relatively speaking,
to get a prototype of your application or even
your first version of your business up and running.
But things start to get hard quickly, especially if you have some success.
Indeed, a good problem to have is that you have so many customers and so
many users hitting your websites that you can't actually
handle all of the load.
Now, it's a good problem in the sense that business is booming.
But it's, of course, an actual problem in the sense
that your customers aren't going to be able to visit your web site
and buy whatever it is you're selling or read
whatever it is you're posting if your servers can't actually handle the load.
And by load, I simply mean the number of users per minute or per unit of time
that your website is actually experiencing.
And its capacity, meanwhile, would be the number
of users it can actually support.
Now, why are there these limits in the first place?
Well, you may recall that inside of a computer
is a CPU, the brains of that computer.
And inside of a computer is some memory, like RAM.
And there might be some longer-term storage, like hard disk space.
At the end of the day, all of those resources and more are finite.
You can only fit so much physical hardware in a computer.
Humans have only been able to pack so many resources
into the physical space of a computer.
And then, of course, there's cost.
You might be able to only afford so much computing capacity.
So if a computer can only do some number of things per second,
there is surely an upper bound on how many people can visit your web
site, how many people can add things to their shopping cart, how many people
can check out with their credit card.
Because you only have, at the end of the day, a finite numbers of resources.
Now, what does that mean in real terms?
Well, maybe your web server can handle 100 users per minute.
Maybe it can handle 1,000 users per minute.
Maybe it can handle 1,000 users per second, or even much more than that.
It really depends on the specifications of your hardware-- how much RAM,
how much CPU and so forth that you actually have-- and it also
depends, to some extent, on how well-written your code is and how fast
or how slow your code, your software actually runs.
So these are knobs that can ultimately be turned.
And through testing, can you figure this out in advance
by simulating traffic in order to estimate exactly how many users you
might be able to handle at a time?
Now, the relevance to today is that the cloud, so to speak,
allows us to start to solve some of these problems
and also allows us to start abstracting away the solutions to some
of these problems.
Well, let's see what this actually means.
So at some point or other--
especially when it's not just my laptop, but it's
like 1,000 laptops, or 10,000 laptops and desktops and phones and more
that are somehow trying to access my server here--
at some point, we hit that upper limit whereby no more users can
fit onto my web site per unit of time.
So what is the symptom that my users experience at that point
if I'm over capacity?
Well, they might see an error message of some sort.
They might just experience a spinning icon
because the website is super slow to respond.
And maybe it does respond, but maybe it's 10 seconds later.
So at the end of the day, they either have a bad experience or no experience
whatsoever, because my server can only handle so many requests at a time.
So what do you do to solve this problem?
If one server is not enough, maybe the most intuitive solution is, well,
if one server is not giving me enough headroom,
why don't I just have two servers?
So let's go ahead and do that.
Instead of having just one server, let's go ahead and have two.
And let me propose that on the second server, it's the exact same software.
So whatever code I've written, in whatever language it's written,
I just have copies of my web site on both the original server
and the second server.
Now I've solved the problem in the simple sense
that I've doubled my capacity.
If one server can handle 1,000 people per second,
well, then surely two servers can handle 2,000 people per second,
so I've doubled my capacity.
So that's good.
I've hopefully solved the problem.
But it's not quite as simple as that.
At least pictorially, I'm still pointing at just one of those servers,
so we're going to have to clean up this picture alone and somehow
figure out how to get users--
or more generally, traffic--
to both of these servers.
I could just naively draw an arrow like this.
But what does that actually mean?
We don't want to abstract away so much of the detail
that we're ignoring this problem.
How do we implement this notion of choosing between left arrow and right
arrow?
Well, let's consider what our solutions might be.
If a user, like me on my laptop, is trying to visit this web site--
and the web site, ideally, is going to live at something like example.com,
or facebook.com, or gmail.com, or whatever--
I don't want to have to broadcast different names for my servers.
And you might actually notice this on the internet.
You might notice, if you start noticing the URLs of websites you're visiting--
especially for certain older, stodgier companies who haven't necessarily
implemented this in the most modern way--
you might find yourself not just at www.something.com,
but if you look closely, you might find yourself occasionally
at www1.something.com, www2.something.com,
or even www13.something.com.
Which is to say that some companies appear to solve this problem by just
giving different names--
similar names, but different names-- to their two servers, three servers,
13 servers, or however many they have.
And then they somehow redirect users from their main domain
name, www.something.com, to any one of those two or three or 13 servers.
But this isn't very elegant.
The marketing folks would surely hate this,
because you're trying to build some brand recognition around your URL.
Why would you dirty it by just putting these arbitrary numbers in the URLs?
Plus if you fast forward a bit in this story,
if, for some reason down the road, you get fancier,
bigger servers that can handle more users,
and therefore you don't need 13 of them--
you can get away with just six of them--
well, what happens if some of your customers have bookmarked,
very reasonably, one of those older names, like www13.something.com?
So now when they try to visit that URL, gosh, they might hit a dead end.
So you could solve that in some other way.
But the point is it would seem to create a problem quickly,
and it's just a naming mess.
Why actually bother having your users see something
as messy as these numbered servers?
It would be nice to do this a little more transparently.
So how could we do this?
Well, let me propose that we kind of need some middleman here,
so to speak, whereby traffic comes from people like me on the internet
and then either goes to the left or goes to the right,
or no matter how many servers we have, goes
to one of those actual web servers.
So how does this middleman-- and to borrow some past terminology,
how does this black box potentially work?
Well, let's consider some of the building
blocks, some of the puzzle pieces we have technologically at our disposal
now.
You may recall that every server on the internet
has an IP address, an internet protocol address, a unique address for it.
And that's, again, a bit of a white lie, because there
are technologies by which you can have private IP
addresses that the outside world doesn't see.
But let's stipulate, for today's purposes,
that every computer on the internet certainly has an IP
address, whether public or private.
So maybe, just maybe, we could leverage an existing technology--
DNS, the Domain Name System--
so that rather than only return one IP address of a server
when you look up www.something.com, we return the IP address
of the server on the left some of the time
or the IP address of the server on the right some of the time,
effectively balancing our load, our traffic across the two servers.
And in fact, if you do this 50-50, you can
take, really, what's called a round robin approach,
and ideally uniformly distribute your traffic across multiple servers.
And what's nice in this model is that because you're using DNS,
the user doesn't really notice what's going on.
At the end of the day, none of us humans really
care what IP address we're actually going to if we visit
Facebook.com or Gmail.com or the like.
We just care that our computer can find that server or servers on the internet.
So via DNS, we could, very cleverly, via this middleman here,
which is really just going to be some third device, some separate server--
it, as a DNS device, could just respond to requests
from customers with either this IP address or this IP address,
or any number of different IP addresses.
So does this solve the problem?
Again, most everything in computer science
would seem to be a tradeoff at the end of the day.
And this seems almost too good to be true, perhaps.
It's so simple.
It leverages an existing technology.
It just works.
So what prices might we pay?
Well, DNS, it turns out, gets cached quite a bit.
And what does caching mean?
Caching something means keeping some past answer--
or more generally, piece of information-- around so
that you can access it more quickly the second and the third time and beyond.
And so computers today, Macs and PCs, as well as
servers on the internet, other DNS servers on the internet,
for performance reasons, will often remember the responses
that they get from DNS servers.
For instance, if, on my Mac, I visit Facebook.com, hypothetically
a lot of times during the day, it's kind of stupid if my laptop, again and again
and again and again, asks some DNS server
for Facebook.com's IP address if it already
asked that same question an hour ago-- or more realistically,
two minutes ago, or something like that.
It would be smarter if my operating system-- or even my browser, Chrome
or Firefox or whatever I'm using-- actually
remembers that answer for me so that my computer can just pull up that web
site faster by skipping a step, by not wasting time asking a server again
for the IP address of a server.
And after all, IP addresses, it turns out, generally don't change that often.
It's certainly possible for a company or a university or even a home user
to change their computer's IP addresses.
But the reality is it doesn't change all that often.
The common case is to have the same IP address
now as you might an hour from now, or even a day or a week or a month
from now.
But the key thing is that it can change.
And especially if you're worried about customers-- not just some personal web
site, but you might lose business.
You might lose orders if users can't visit your website.
Anything that puts your server's uptime, so to speak--
being accessible on the internet at risk--
probably is worthy of some consideration.
So let me propose, then, that just one of these servers goes offline somehow.
Maybe it's deliberate.
You need to do some service for it.
Or maybe it crashed in some way, or it got unplugged somehow,
or something went wrong such that now, one or more of your servers,
across which you've been load balancing, no longer can talk to the internet.
What might happen?
Well, if some customer's Mac, like my own,
has remembered or cached that particular server's IP address,
that is not a good situation.
Because your Mac or PC or whatever is going
to now try to revisit your web site again and again
and again at that old cached IP address that apparently can be a dead end.
And so even though you still have servers that could potentially
handle that customer's request, that customer's order,
that customer's desire to check out, he or she
really is still not going to be able to visit
the website unless that cache expires.
Maybe they reboot their computer so that the cache forcibly expires.
Maybe they just wait some amount of time so that that IP address
is forgotten by the browser or by the operating system
or by some other DNS server until the new one's available IP
addresses are picked up instead.
But there is that risk.
And I would argue that this risk is even higher especially
for companies that might be considering moving their infrastructure from one
service to another.
If you're deliberately going to move your servers from one IP address
to another, as might happen if you change cloud providers, so to speak--
more on those in a minute--
really, if you change the companies that you're using to host your servers,
your IP addresses will change.
And you certainly don't want to incur a huge amount of downtime
in a situation like that.
So there are these tradeoffs.
Easy solution, technologically pretty inexpensive to do.
It just works using existing technology.
But you open up yourselves to this risk.
So let's address that.
Putting back the old proverbial engineering hat,
let's try to solve this problem.
It seems that giving a unique IP address to this server
and to this server, and any number of other servers that are back there,
might not be the smartest idea in so far as those IPs can get cached.
So what if we use DNS as follows?
When my laptop or anyone else's requests the IP address for www.something.com,
why don't we return the IP address of this device here--
this load balancer, as we'll start calling it,
where a load balancer is usually just a physical device,
or multiple physical devices, whose purpose in life is to balance load?
Packets come in, and similar in spirit to a router,
they do route information to the left, to the right, or some other direction.
But their overarching purpose isn't just to get data from point A to point B,
but to somehow intelligently balance that traffic
over multiple possible destinations for point B, identical servers
in the case of our story here.
So what if, instead, we addressed this problem of potential downtime
by returning the IP address of the load balancer,
and then, by nature of private IP addresses
or some other mechanism that the end user does not
need to know or care about, this load balancer somehow routes the traffic
to either the first device or the second device,
LB here being our load balancer?
So we've seemed to have solved this problem.
In so far as now we have configured our DNS servers
to return the IP address of the load balancer,
there's no problem of downtime as we described a moment ago.
Because if Server 1 goes offline for whatever reason, no big deal.
The load balancer should hopefully just notice that and subsequently start
proactively routing all incoming data that reaches its IP address to Server 2
and not Server 1.
now how does the load balancer know?
Well, either a human could intervene.
Maybe someone gets a late night call or text or page saying,
uh oh, server 1 is down, you better do something.
And then he or she can manually configure the load balancer
to no longer send any traffic to Server 1.
That seems kind of stupid in an age of automation and smart software.
Maybe we can do better.
And indeed, we can.
A technique that's often used by servers is
something modeled from the human world to use
what you might describe as heartbeats to actually configure
the load balancer and Servers 1 and 2 to operate as follows.
Maybe every second, every half a second, maybe every five seconds
you configure Server 1 and Server 2 to send some kind of heartbeat message
to the load balancer.
This is just a TCP/IP packet, some kind of network packet
that's the equivalent of saying I'm alive.
I'm alive.
Or more goofily, like boom, boom, boom, boom, ergo the heartbeat metaphor.
But the point is that 1 and 2, and any number of other servers,
should be configured to just constantly reassure
the load balancer that they are alive.
They are accessible.
They are ready to receive traffic.
And the load balancer, similarly--
and you might see where this is going-- can very simply
be configured to listen for that heartbeat.
And if it ever doesn't hear a heartbeat from Server 1 or Server 2,
it should just assume that something is wrong.
The server has died.
It's gone offline.
Something bad has happened.
So the load balancer subsequently should simply not
route any traffic to that particular server
until some human or some automated process
brings the server back alive, so to speak, and the heartbeat resumes.
Now, of course, this problem doesn't go away permanently.
If servers 1 and 2 stop emitting a heartbeat,
we really have no capacity for users.
But that would be an extreme scenario.
Hopefully it's just one or a few of our servers go offline in that way.
So we can configure our servers for these heartbeats, which is--
think about it--
a very simple physiologically-inspired solution to a problem.
And even if it's not obvious how you implemented it in code,
it really is just an algorithm, a simple set of instructions
with which we can solve this problem.
And yet, damnit, we've introduced a new problem.
And so this really is the old leaky hose,
where just as we've plugged one leak or solved one problem,
another one has sprung up somewhere else along the line.
So what's the problem now?
What's the problem now?
The whole motivation of introducing Server Number 2, in addition
to Server Number 1, was to make sure that we have enough capacity,
and better yet, to make sure that if Server 1 or Server 2 goes offline,
the other one can hopefully pick up the load unless it's
a super busy time with lots and lots of users visiting all at once.
So in fact, the general idea at play here
is high availability ensuring that if one server goes down,
you have other servers that can pick up the load.
Being highly available means you can be tolerant to issues like that.
And then load balancing, of course, is just
the mere process of splitting the load across those two endpoints.
But we have introduced another problem.
This might be abbreviated SPOF, or more explicitly, Single Point Of Failure.
Just as I've solved one problem by introducing this load balancer,
so have I introduced a new problem, which is this.
There is now, as you might infer from the name
alone, a single point of failure.
It's fine that I can now tolerate Server 1 or Server 2 going down,
but what can I not tolerate, clearly?
What if the load balancer goes down?
So this is a very real concern.
Maybe the load balancer itself gets overloaded.
Maybe the load balancer itself has some kind of issue.
And if the load balancer goes down, it doesn't
matter how many web servers I have down here,
or how much money I've spent down here to ensure my high availability.
My server is offline if this single point of failure indeed fails.
Now, you'd like to think that the load balancer--
especially since it only has one job in life--
can at least handle more traffic than any individual server.
Indeed, clearly, it must be the case that the load balancer
is fast enough and capable enough to handle twice as
much traffic as any individual server.
But that's generally accepted as feasible insofar as your website.
Your real intellectual property is probably
doing a lot of work-- talking to a database,
writing out files, downloading things, or any number of other features
that just take more effort than just routing data from one server
to another as a load balancer does.
But it doesn't matter how performant it is.
If the load balancer breaks, goes offline for some reason,
your entire infrastructure is inaccessible.
So how do we solve this?
How do we go about and architect a solution to this?
Well, how did we address this issue earlier?
We addressed the issue of insufficient capacity or potential downtime
by just throwing hardware at the problem.
And so maybe we could do that same thing here.
Maybe we could just introduce a second load balancer.
I'll call this LB as well.
And now we somehow have to--
I feel like we're just endlessly going to be adding more and more rectangles
to the picture.
But somehow, we need to be able to load balance across now two servers and two
load balancers.
So how do we do this?
Well, let me clean this up so that we have a bit more room to play with here
and consider how a pair of load balancers might actually work.
So if my first server is here and my second server is here,
and I'm proposing now to have two load balancers-- one here and one here--
surely, both of these have to be able to talk to both servers.
So we already have this necessity.
And somehow, traffic has to come from the internet
into this set of load balancers, but probably only to one,
because we don't want to solve this with DNS
and just have two IP addresses out there.
Because if one breaks, we can recreate the same problem
as before if we're not careful.
So what if we do this?
What if we use this building block of heartbeats in another way as well?
What if we ensure that our load balancers--
plural-- have just one IP address, which a moment ago seemed
to create a single point of failure?
But what if we do this?
What if we also allow the load balancers to talk to,
to communicate over a network with each other so that one of the load balancers
is constantly saying to the other, I'm alive.
I'm alive.
And so what the load balancers could be configured to do
is that only one of them operates at any given point in time.
But if the other server, the other load balancer,
no longer hears from that primary load balancer because of the heartbeats
that are ideally both being emitted in both directions
so that they can both be assured of the other's up time--
if the secondary load balancer stops hearing the primary load balancer,
the secondary load balancer can just presumptuously
reconfigure itself to take on that one and only IP address,
effectively assuming that the first load balancer is not going
to be responding to any traffic anyway.
And the second load balancer can simply take on the entire load itself.
But the key difference now in this particular solution
is that there's only one IP address that describes this whole architecture, only
one IP address between the two load balancers
so we don't risk those potential dead ends that we had a little bit ago
with our back end servers.
So now it's starting to get more robust, more highly available.
So that's pretty good.
We've solved most of these problems.
We've generously, though, swept one problem underneath the rug, whereby
every time I draw another rectangle--
not just the first time, but now the second time--
and add some interconnectivity, somehow, among them someone
somewhere is spending some money.
And indeed, I am solving these problems thus far
by throwing money at the problem, and frankly introducing complexity.
Already look at how many arrows or edges there
are now, which might simply refer to physical wires, which is fine.
But there's also a logical configuration that's now necessary.
And God forbid we have a third load balancer for extra high availability
or any number of servers here--
13 or 20 or 100 or 1,000 servers.
It's a lot of cross-connections-- not just physically,
but logically in terms of the requisite configuration.
So this complexity does add up.
And the cost certainly adds up.
And now, once upon a time-- and not all that
long ago-- if a company wanted to architect this kind of solution,
you would literally buy two load balancers,
and you would buy two or more web servers,
and you would buy the requisite physical ethernet
cables to interconnect the two.
And you'd probably buy a whole bunch of other hardware
that we've not even talked about, like firewalls and switches and more.
But you would physically buy all of this hardware.
You would physically connect all of this hardware
and configure it to implement these several kinds of features.
But the catch is that the more and more hardware
you buy, just probabilistically, the more and more you
invite some kind of failure.
Maybe it's some stupid human error.
But more realistically, one of your hard drives is going to fail.
And hard drives are typically rated for the enterprise in terms of Mean Time
Between Failure, MTBF, which generally means
how long should you expect a hard drive to work on average before it fails.
It breaks.
It just stops working.
So if you have a whole bunch of servers, each of which
has a whole bunch of hard drives, at some point,
combinatorially, one or more of those drives is just going to fail,
which is to say you're going to have a problem,
and you're going to have to fix it yourself.
At some point, too, you're going to run out of physical space.
In fact, perhaps one of the most constraining resources,
especially for startups, is the physical space itself.
You probably don't want to start housing your servers in your physical office,
because you need a special room for it, typically, with enough cooling,
with enough access, with enough electricity, and enough humans
to actually maintain it.
Or you graduate from your own office space and go to a data center,
a co-location facility, whereby you maybe
rent space in a physical cage with a locking door,
inside of which you put racks of servers,
just racked up on big metal poles, and you pack as many servers in there
as you can.
But at some point, you're going to be bumping up
against other constrained resources-- physical space, actual power capacity,
cooling, as well as the humans to actually run this.
And so very quickly does operations, ops,
so to speak, become an increasing cost and an increasing challenge.
And one of the most alluring features of the cloud, so to speak,
is that you can move all of these details off-site.
And you can abstract many of these, let's say, implementation details
away whereby you yourself don't have to worry about the physical wires.
You don't have to worry about the make and model of servers
that you're buying.
You don't have to worry about things actually breaking,
because someone else will deal with that for you.
But you have to still understand the topology and the architecture
and the features that you want to implement so that you can actually
configure them in the cloud.
So what do you actually get from cloud providers?
There's any number of them out there these days.
But perhaps three of the biggest are Amazon, Google, and Microsoft,
all of whom offer, these days, of very similar palettes of options.
And it's outright overwhelming, if you visit
each of their web sites, just how many cloud products they offer.
But they would generally offer a number of standard products
in the cloud-- for instance, a virtualized server.
So you don't have to physically buy a server these days
and plug it into your own ethernet connection, your own internet
connection in your own office.
You can instead essentially rent a server
in the cloud, which is to say that Amazon, Google,
Microsoft, or any number of other companies
will host that server physically for you,
and they will take care of the issues of power and cooling.
And if a hard drive fails, they will go remove the old one
and plug in the new one.
And ideally, they will provide you with backup services.
But more sophisticated than that, they can also
help us recreate, in software, this kind of topology.
In other words, even without having a human physically wire together
this kind of graph, so to speak, that we've been building up
here logically, thanks to software these days,
you can implement this whole paradigm--
not with physical cables, not with physical devices,
but with software virtually.
What does that mean?
It means that humans, over the past several years,
have been writing software that mimics the behavior of physical servers.
Humans have been writing software that mimics the behavior of a router.
Humans have been writing software that mimics the behavior of a load balancer.
And implementing mimics the behavior of-- really, we're just building,
in software, what historically might have been implemented entirely
in hardware.
And even that's a bit of an oversimplification.
Because even when something is bought as hardware,
there is, of course, software running on that hardware that actually makes
it do something.
But they're no longer dedicated devices.
You can use generic commodity PC server hardware, really,
and transform that hardware into a certain role, a back end web
server, a back end database, a load balancer, a router, a switch,
any number of other things.
And so what you were getting from companies like Amazon and Google
and Microsoft and more is the ability to build up
your infrastructure in software.
In fact, the buzzword here, the acronym, is IaaS, Infrastructure as a Service.
So you sign up for an account on any of those companies' cloud services web
sites, and you put in your credit card information or your invoicing
information, and you literally, via a command line tool-- so a keyboard,
or via a nice, web-based graphical user interface, GUI--
do you point and click and say, give me two servers and one load balancer.
Or if you have enough money in the bank, you
say give me two servers and two load balancers
configured for high availability.
Or better yet, you don't say any of that.
You just tell the provider, give me a web server and give me a load balancer,
and you deal with the process of scaling those things as needed.
In fact, a buzzword de jeur is auto scaling, which refers to a feature,
implemented in software, whereby if a cloud
provider notices that your servers are getting a lot of traffic--
business is good, or it's the holiday season,
and you are bumping up against just how many users your one or two
or three or more servers can handle--
auto-scaling is a feature that will enable the cloud provider to just turn
on, virtually, more servers for you so that you go from two to three
automatically.
You can be happily asleep in the middle of the night,
and even though your traffic is peaking, it doesn't matter.
Your architecture is going to auto scale.
And better yet-- especially financially--
if the cloud provider notices, maybe 12 hours later-- oh, all of your customers
have gone to sleep, we don't really need all of this excess capacity.
Or maybe the holidays are now in the past.
You really don't need this excess capacity.
Auto scaling also dictates that those servers can be virtually turned off.
So you're no longer using them.
You're no longer load bouncing to them.
And most importantly, you're no longer paying for them.
So this is a really, really nice value add at this point.
There's no human crawling around on the floor rewiring things
and plugging in new servers.
There's no finance person having to approve the PO to actually order more
servers just to increase your capacity.
And most importantly, there is no latency between the time when
you notice, oh, my god, we're getting really successful
and can't handle our load-- uh oh.
It's going to be a two, three-week lead time
before we can even get in the more servers.
Thanks to cloud computing, you can literally log in to Amazon's, Google's,
Microsoft's web site and, click, click, click,
have more server capacity within seconds, within minutes,
far faster than the physical world traditionally allowed.
So those are just some of the features now
that we gain from outsourcing to the so-called cloud.
So where does some of this capability come from?
Well, it turns out that over the past many years,
humans have been getting better and better and better
at packing more physical hardware into the same form factor,
into the same physical space.
So at the level of CPUs, the brains of a computer,
we humans have gotten much better at packing more and more transistors,
for instance, onto a CPU.
And transistors are the little switches that can turn things on and off--
0 and 1, 1 and 0.
So you can store more information and you
can do more with that information more quickly.
CPUs today also have more cores, which you
can think of as mini CPUs inside of the main CPU,
so that a computer with multiple cores can literally
do multiple things at a time.
But the funny thing is that we humans, over the past decade or two,
really haven't been getting fundamentally faster at life.
At the end of the day, I can only check my email so quickly.
I can only post on Facebook so quickly.
I can only check out from Amazon so quickly.
Because we humans have, of course, a finite speed to ourselves.
We're not just getting-- we're not doubling in speed
a la Moore's law every year or two.
So we have, it would seem, a lot of excess computing capacity these days.
Computers are getting so darn fast, we don't necessarily
know what to do with all of these CPU cycles and with all of the RAM
that we can fit into the same physical box at half the price
that it cost us last year.
And so manufacturers and companies realize
that we could actually build a business on this increased capacity.
We can implement the computer equivalent of timesharing, so to speak,
which has long been with us in the history of computing.
But we can do this on a much more massive scale
now by taking one physical server that has maybe two CPUs, or 16 CPUs,
or 64 CPUs, and maybe gigabytes--
tens of gigabytes or hundreds of gigabytes of RAM--
all inside of the same physical device, plug it in to an internet connection,
and then run special software on that one server that creates the illusion
that there's multiple servers living inside of that box.
And this virtualization software is implemented
by way of software called a virtual machine, or virtual machine monitor,
or another word might be hypervisor.
There's different ways to describe essentially the same thing.
But a virtual machine is a piece of software
running on a computer inside of which is running some other operating system,
typically.
So you might have one server running Windows.
But inside of that server are multiple virtual machines, each of which
itself is running Windows.
So you might be able to chop up one computer into 10, or even into 100.
Or perhaps more commonly, you might have a server
running Linux or some Unix-based operating system,
also with virtual machines on it.
But those virtual machines might be running
Linux themselves, or Unix, or Windows, or any number of versions of Windows.
And so this is the beauty.
When you have so much excess capacity and so many
available CPU cycles and so much RAM, you
can slice that up and then sell portions of the server's capacity to customers.
And if you're really clever, you might look at your customers' usage patterns
and realize that, you know what, it's not necessarily
as simple as just taking my server and dividing it up into n different slices,
where n is a generic variable for number,
and then selling it or renting that space, really, to end customers.
Because you know what?
Some of those customers might have some booming businesses, which is great.
But some of those customers might not have many users.
Maybe it's a few dozen.
Maybe it's a few hundred.
But it's really a drop in the bucket.
So instead of selling my computing resources to just end customers,
maybe I'll sell it to twice as many customers or three times
as many customers, and essentially over-sell my server's capacity,
but expect that on average, this is just going
to work out because some customers will be using a lot of those cycles
because business is good, and some won't be,
because it's just they don't have many customers,
or really, it's a personal website that doesn't get much usage anyway.
And so for some time, there has, of course,
been this risk, when you sign up for a web hosting company or a cloud
provider, that your web site actually might get really slow for reasons
outside of your control.
If you are co-located on a server that some other booming business is on,
your users might actually suffer if your web host has oversold itself.
And so in fact, this is one of those situations
where you get what you pay for.
If you're googling around and finding various cloud providers,
or web hosting companies more specifically,
you might be able to find a deal, like $10 per month or $50 per month,
as opposed to $100 or $200 or more per month.
And you do get what you pay for, because those fly-by-night operations that
are selling you space and capacity super cheaply probably
are overselling and over-committing.
So these are the trade-offs, too--
how much money do you want to save versus how much risk
do you actually want to take on?
Generally, it's safer to go with some of the bigger fish these days, certainly
when building a business, as you might on a company like Amazon or Google
or Microsoft or derivatives thereof.
So just to paint a more concrete technical picture
of what virtualization is, here's a picture, as you might think of it.
So you have your physical infrastructure here.
So that's the actual server from Dell or IBM or whoever.
Then you have the host operating system, which might be Windows,
but is often Linux or some variant of Unix instead.
And then you have the hypervisor.
This is the piece of software that you install
on your server that allows you to run multiple virtual machines on top of it.
And those virtual machines can each run any number
of different operating systems themselves, or even different versions
of operating systems.
And so depicted here up top are the disparate guest OS operating
systems that might be on there.
Maybe this is Linux and Solaris, and this is Windows itself,
or any number of other combinations.
Whatever your customers want or whatever you want to provide or essentially
rent to customers, you can install.
But you do pay a price.
So as beautiful as this situation is, and as clever
as it is that we're leveraging these excess resources by slicing up
one server into the illusion of, in this case, three, or more generally more,
there is some overhead.
Because this hypervisor has to be a middleman between your guest operating
systems and your host operating system, the one actually
physically installed on the server.
And any layers of indirection like this, so to speak,
have got to cost you some amount of time.
If there's some work being done here and you only
have a finite number of resources, the hypervisor
itself is surely consuming some of your resources.
And gosh, this just seems really inefficient,
especially if all of your customers are using the same operating system.
My god, why do you have to have copies of the same OS multiply installed?
This just doesn't feel like it's leveraging much economy of scale.
And so it turns out there's a newer technology that's gaining steam,
and this is known not as virtualization, per se, but containerization,
the most popular instance of which is perhaps a company called Docker.
And the world of Docker is a little shorter.
It's a little smarter about how resources are shared.
You still have your infrastructure, your physical server,
and you still have your host operating system,
whether it's Linux or Unix or something like that.
But then instead of a hypervisor, you have the Docker engine,
which is really just an equivalent of that base layer of software.
But notice what's different.
In this case here, we've collapsed the previous picture.
In fact, thanks to our friends at Docker who put this together here,
the guest OS has disappeared.
And you instead have your different applications
and your different binaries and libraries,
as this abbreviation means, all running on the Docker engine.
Now, what does this mean?
This means when running Docker, you typically
choose your operating system--
for instance, Ubuntu Linux or Debian Linux or something else altogether--
and then you essentially share that one operating system
across multiple containers.
Instead of virtual machines, we now have containers.
So in other words, you ensure that your different slices all
share some common software-- the kernel, so to speak,
the base core of the operating system.
But then you uniquely layer on top of that base system,
that base set of default files, whatever customizations your customers or you
yourself want, but you share some of the resources.
And long story short, what this means is that containers
tend to be a little lighter weight.
There's less waste of resources because there's less overhead of running them,
which is to say that you can generally start them even more quickly.
And better yet, you can still isolate your different products
and your different services-- database and web server and email
server and any number of other features--
all within the illusion of their own installation, their own operating
system, even though there are some shared resources here.
So this, too, has been made possible by the capabilities of modern hardware
and the cleverness, frankly, of humans in actually
finding solutions or creative uses for those available resources.
But what other features or topics come into play
in this world of cloud computing?
We've talked about availability and caching and costing, really
figuring out where we're going to actually spend
our money by throwing hardware at problems and scaling more generally.
But there's also issues of replication, which
actually do relate to high availability, so to speak.
But replication refers to duplication of data,
and really backups more generally as a topic.
And then there's also some other funky acronyms that are very much in vogue
these days.
Besides Infrastructure as a Service, there's
also Platform as a Service, PaaS, or Software as a Service, SaaS.
Now, SaaS, even if you've not used it under this name,
odds are you have been using it.
If you do use Gmail or Outlook.com or any web-based email service,
you are using software as a service.
You don't really know, or need to care, where in the world
your emails physically live, or how many servers they're spread across,
or how your data is backed up, or for that matter, when
you click Send, how the email even gets from point A to point B.
You are treating Gmail and Outlook as a software
as a service with all of the underlying implementation details abstracted away.
You just don't know or care how it's implemented-- well,
at least if everything is working.
You probably do care if something goes down.
But there's this intermediate step between this extreme form
of abstraction where all you see is just the top-level service.
And the lowest level implementation that we've
discussed, which is infrastructure as a service,
whereby when using something like Amazon,
you literally click the button that says give me a load balancer.
You literally click a button that says give me two servers.
You literally click a button that says give me
a firewall or any number of other features.
So Amazon and Microsoft and Google, to some extent,
have all implemented these low-level services
that still require that you understand the technology,
and you understand networking, and you understand scaling and availability.
But you so much more easily and inexpensively and efficiently--
literally with just a laptop or desktop, without any data center of your own--
stitch together the topology or the architecture that you actually
want, albeit in the cloud.
Platform as a service, though, has arisen as a middle ground
here, whereby you might have services like Herouku,
which you might have heard of, which themselves actually
run on infrastructures like Amazon or Google or Microsoft or the like.
But they provide themselves a layer of abstraction
that isn't quite as high level, so to speak, as what
you get from software as a service.
In fact, these platforms as a service don't provide you with applications.
They just make it easier for you to run your applications in the cloud.
Now, what does that mean?
Well, it's all fun and exciting to understand load balancing
and understand networking and understand the need
for multiple servers and the entire conversation that we've had thus far.
But at the end of the day, if I'm a software developer
or I'm trying to build a business, all I care
about is making my internet application available to real users.
I really don't care about how many servers I have,
how many databases I have, how the load balancers talk to one another.
That's all fine and intellectually interesting.
But I just want to get real work done.
So I'm willing to pay a bit more for this.
I'm willing to pay some middleman, like a Herouku, or any number
of other services, a platform as a service,
to abstract away those kinds of details.
So I have the wherewithal, and I have the willingness
to actually say host this as a web server.
So give me a web server.
I will pay you some number of dollars per month to give me a web server.
But I want you, Herouku, to deal with the auto scaling of it.
I don't care how many servers it is.
I don't care how they are connected.
I don't care anything about these heartbeats.
I just want to have the illusion, for my own sake,
of just one server that somehow grows or shrinks
dynamically to handle my customer base.
Meanwhile, things like load balancing, I just want my customers
to be able to reach my server.
I don't care how it's implemented.
I don't care how it's made to be highly available.
I just want that to work.
And so companies like Herouku provide these platforms
as a service that just make your life a little bit easier.
And you don't have to think about or know about or worry about as many
of these details.
Now, to be fair, if something breaks, you
might not understand exactly what's going wrong,
and you yourself might not be able to solve it.
Indeed, you might be entirely at the mercy of the cloud provider, or the PAS
provider, to solve the problem for you.
But you're saving time.
You're saving energy elsewhere by not having
to worry about those lower-level implementation details, at least
in the common case.
But odds are you're paying a little more to Herouku than you would to an Amazon
directly because they're providing you with this value-added service.
So as cryptic as these acronyms really mean,
they're really just referring to disparate levels
of abstraction, all of which somehow relate to the cloud.
But infrastructure as a service is a virtualization
of these hardware ideas, the physical cabling
that we drew here on the screen.
Software as a service really is just that application
that the user interacts with.
And platform as a service is an intermediate step,
whereby you, in building your software in the cloud,
can worry a little bit about how to actually make it available to users.
But let's consider one other challenge now--
that of database replication since, of course, thus far,
we've been talking about a web server as though it's the entire picture.
But the reality is most any business that
has a web-based presence or a mobile presence
is going to be storing information.
When users register, when users check something out,
add something to their shopping cart, so to speak, all of that data
needs to somehow be stored.
So let's consider now what the world really likely looks like.
So here is my laptop again.
And here is the cloud that's between me and some service
that I'm interested in.
We'll assume for now that there is some kind of load balancing.
And I'm just going to draw it a little bigger
this time to suggest that-- let's just think of it now as a black box.
And maybe it's one server.
Maybe it's two.
Maybe it's more.
But somehow or other, load balancing is implemented.
Then I'm going to have all of my servers here,
which we'll abstract away as maybe three or more at this point-- one, two,
and then we'll call this n.
But a web server typically does not do everything these days.
In fact, it's been trending for some time
to actually have different servers or different virtual machines,
or even more recently, different containers.
Each provide individual services.
Sometimes people call these micro services
if a container only does one, and one very narrowly defined thing,
like send emails, or save information to a database,
or respond to HTTP requests.
So these back end web servers are not the only types of servers we have.
Odds are we at least have one database.
So let's consider now the implication of all
of these architectural decisions we've made thus far
on how we actually store our data.
So in simplest form, our database might look like this.
And for historical reasons, it's generally drawn as a cylinder.
And this is our database.
Now, it's immediately obvious that if all servers--
1, 2, dot, dot, dot, n-- need to save information or read information
from a database, they've all got to somehow communicate with that database
so they all have some kind of connectivity, physically or otherwise.
So this seems fine so long as the software that's running on servers 1,
2, dot, dot, dot, and no matter what language we're using,
whether it's Java or Python or PHP or C# or something else--
so long as those servers can talk to, via the network, this database,
that's great.
They can all save their data to the same place,
and they can all read their data from the same place.
So everything stays nicely in sync.
But what's the first problem that motivated the entirety
of this discussion from the outset?
Well, what if one database isn't really enough?
Well, we could take the approach of vertically scaling
our architecture, which is another piece of jargon in this space.
So vertical scaling means if your one database isn't quite up to snuff,
and you're running low on disk space or capacity because of numbers
of requests per second are, of course, limited, you know what you can do?
You can go ahead and disconnect this one and go ahead and put in a bigger one,
and therefore increase your capacity.
And vertical scaling means to really pay more money
or get something higher end, a higher, more premium
model, a more expensive model that's got more disk space and more RAM
and a faster CPU or more CPUs.
So you just throw hardware at the problem--
not in the sense of multiple servers, but just one bigger and better server.
But what are the challenges here?
Well, if you've ever bought a home computer,
odds are whether it's been on Dell's site or Microsoft's or Apple's
or the like, you often have this good, better, best thing
where, for the top of the line laptop or desktop,
you're going to be paying through the roof--
through the nose, so to speak.
You're going to be paying a premium for that top of the line model.
But you might actually be able to save a decent number of dollars
by going for the second best or the third best,
because the marginal gains of each additional dollar
really aren't all that much.
Because for marketing reasons, they know that there
might be some people out there that will always
pay top dollar for the fastest one.
But just because you're paying twice as much
doesn't mean the laptops is going to be twice as good, for instance.
So this is to say to vertically scale your database, you might end up
paying, through the nose, some very expensive hardware just
to eke out some more performance.
But that's not even the biggest problem.
The most fundamental problem is at the end of the day,
there is a top-of-the-line server for your database that only can support
a finite number of database connections at a time,
or a finite number of reads or writes, so to speak,
saving and reading from the database.
So at some point or other, it doesn't matter how much money you have
or how willing you are to throw hardware at the problem.
There exists no server that can handle more users than you currently have.
So at some point, you actually have to put away your wallet
and put back on the engineering hat alone and figure out
how to not vertically scale, but horizontally scale your architecture.
And by this, I mean actually introducing not just one big, fancy server,
but two or more maybe smaller, cheaper servers.
In fact, one of the things that companies like Google
were especially good at early on was using
off-the-shelf, inexpensive hardware and building supercomputers out of them,
but much more economically than they might
have had they gone top of the line everywhere,
even though that would mean fewer servers.
Better to get more cheaper servers and somehow
figure out how to interconnect them and write the software that
lets them all be useful simultaneously so that we can instead
have a picture that looks a bit more like this, with maybe
a pair of databases in the picture now.
Of course, we've now created that same problem
that we had earlier about where does the data go.
Where does the traffic or the users flow,
especially now where we have one on the left and one on the right?
So there's a couple of solutions here, but there are some different problems
that arise with databases.
If we very simply put a load balancer in here, LB, and route traffic uniformly--
say, to the left or to the right--
that's probably not the best thing.
Because then you're going to end up with a world where you're
saving some data for a user here and some data for a user
here just by chance, because you're using round robin, so to speak,
or just some probabilistic heuristic where some of the traffic
goes this way, some of the traffic goes that way.
And that's not so good.
OK.
But we could solve that by somehow making sure that if this user, User A,
visits my web site, I should always send him or her to the same database.
And you can do this in a couple of ways.
You can enforce some notion of stickiness,
so to speak, whereby you somehow notice that, oh, this is
User A. We've seen him or her before.
Let's make sure we send him to this database on the left
and not the one on the right.
Or you can more formally use a process known as sharding.
In fact, this is very common early on in databases,
and even in websites like Facebook, where you have so many users
that you need to start splitting them across multiple databases.
But gosh, how to do that?
Back in the earliest days of Facebook, what they might have done
was put all Harvard users on one database, all MIT users on another,
all BU users on another, and so forth.
Because Facebook, as you may recall, started scaling out
initially to disparate schools.
That was a wonderful opportunity to shard
their data by putting similar users in their respective databases.
And at the time, I think you couldn't even
be friends with people in other schools, at least very early
on, because those databases, presumably, were independent,
or certainly could have been topologicaly.
Or you might do something more simple that doesn't create
some problems like isolation there.
Maybe all of your users whose last name start with A go on one server,
and all of your users whose names start with B
go on another server, and so forth.
So you can almost hash your users, to borrow a terminology from hash tables,
and decide where to put that data.
Of course, that does not help with backups or redundancy.
Because if you're putting all of your A names here and all of your B names
here, what happens, god forbid, if one of the servers goes down?
You've lost half of your customers.
So it would seem that no matter how you balance the load,
you really want to maintain duplicates of data.
And so there's a few different ways people solve this.
In fact, let me go ahead and temporarily go
back to that first model, where we had a really fancy, bigger database
that I'll deliberately draw as pretty big.
And this is big in the sense that it can respond to requests quickly
and it can store a lot of data.
This might be generally called our primary or our master database.
And it's where our data goes to live long term.
It's where data is written to, so to speak, and could also be read from.
But if we're going to bump up against some limit of how much work
this database can do at once, it would be
nice to have some secondary servers or tertiary servers.
So a very common paradigm would be to use this primary database for writes--
we'll abbreviate it w--
and then also have maybe a couple of smaller databases, or even
the same size databases, that are meant for reads, abbreviated R.
And so long as these databases are somehow talking to one another,
this topology will just work.
This is a feature known as replication.
So long as the databases are configured in such a way
that any time data is written to the primary database or the master
database, that data gets replicated to any replicas, as they're called.
Meanwhile, servers 1, 2, and n should also be able to talk to these replicas.
And if your code is smart enough-- and you would have to think about this
and design this into your codebase-- you could ensure that any time you
read data from a database, it comes from one, or really any, of your replicas,
replicas in the sense that they are meant to have duplicate data.
But anytime you write data-- a SQL INSERT or UPDATE or DELETE,
as opposed to a SQL SELECT--
you only send your write operations to the primary or master database
and leave it to it to then replicate it to the read replicas.
Now, of course, there are some problems here.
There's some latency, potentially.
Maybe it takes a split second.
Maybe it takes a couple seconds for that data to replicate.
So things might not appear to be updated instantaneously.
But you have now a very scalable model in that if you have the money to spend,
you can even have more read replicas and have even more and more read capacity.
Of course, you're going to eventually bump up against a limit on your rights,
at which point we need to introduce another solution.
But again, this is a very incremental approach.
And we can throw a little bit of money at the problem each time
and a little bit of engineering wherewithal
in order to at least get us over that next ledge, which
is super important, certainly, when you're first building your business.
If You don't necessarily have the resources to go all in on things,
you at least want to get over this hurdle
or at least build in some capacity for the next load of users.
So what if we run out of capacity, though,
with that that writable server, the master database, so to speak?
We need to be a little more clever.
And it turns out we can borrow this idea of these horizontal arrows
here to replicate our data, but for a slightly different purpose.
We could still have a pretty souped up writable database.
But we could have another one, maybe identical in its specs, writable.
But somehow, these things need to be able to synchronize with themselves.
And maybe there's still some read replicas over here--
R for read, and another one over here, R for read.
And these are all somehow interconnected as well.
But you can have what's called master master replication, whereby
your server's code writes to one of these servers.
And maybe it's either of them now.
Maybe the load balancer actually does send some of the writes
this way, some of the writes this way.
But the master database, the writable ones now,
are configured, in software, to replicate horizontally, so to speak.
So here too, you might have a little bit of latency.
It might take a few milliseconds or seconds
for the data to actually replicate.
But at least now we've doubled the capacity for our writes
so as to handle twice as many writable operations.
And we can continue to hang more and more read replicas off of these
if you want in order to handle more and more users.
And so this is the challenge and, dare say, the fun of engineering
architecturally-- understanding some of these basic building blocks.
And even if you might not know the particular manufacturers
or how you physically configure the servers,
or how in software you configure these servers, at the end of the day,
these really are just puzzle pieces that can somehow be interlocked.
And these puzzle pieces can be used to solve more and more interesting
problems.
But to our discussion PaaS and Software as a Service and Infrastructure
as a Service, there's also these different layers of abstraction.
And so thematic throughout this in all of our discussions
has been this layering.
Indeed, we started, really, down here with those zeros and ones and bits,
and very quickly went to Ascii, and very quickly went
to colors and images and videos and so forth.
Because once you understand some of those ingredients or puzzle pieces,
can you build something more interesting?
And then can you slap a name on it--
sometimes cryptic, like IaaS, or PaaS, or SaaS?
But at the end of the day, those are just
labels that describe, really, black boxes, inside of which
is a decent amount of complexity, a clever amount of engineering,
but ultimately, a solution to a problem.
And so in cloud computing, do we really have this catch-all phrase that's
referring to a whole class of solutions to problems that ultimately are all
about getting one's business or getting one's personal website
out on the internet for users to access, whether via laptops or desktops
or mobile devices and more?
So at the end of the day, what is the cloud?
It's this evolving definition.
It's this evolving class of services that just continues to grow.
But each of those services is solving a problem.
Each of those problems derives from plugging one hole in a leaky hose,
seeing another one spring up, and then addressing that one,
and then layering on top of those solutions these are abstractions,
and ultimately some marketing speak, like cloud computing itself,
so that you can build, out of these more sophisticated puzzle pieces,
bigger and better solutions to actual problems
you have when you're trying to build your own site.