JAMES CUFF: Hi, good afternoon, everyone. My name is James Cuff. I'm the Assistant Dean for Research Computing here at Harvard University. And today I'm going to talk to you about why scale-out computing is essential. 

So I guess, first up, who is this guy? Why am I here? Why am I talking to you? I have a background in scientific computing and research computing, stretching back to the United Kingdom-- The Wellcome Trust Sanger Institute for the human genome-- and then more recently in the United States working at the Broad and other esteemed places of learning, such as Harvard. 

I guess what that really means is that I'm a recovering molecular bio physicist. So what right have I got to tell you about scale-out computing? There's a however. 18 years or so I've just seen the most dramatic increases in scale complexity and overall efficiency of computing systems. 

When I was doing my PhD at Oxford, I was pretty excited with a 200 megahertz Silicon Graphics machine with 18 gigabytes of storage and a single CPU. Times have changed. If you fast forward now, we're spinning over 60,000 CPUs here at Harvard. Many other organizations are spinning many more. 

The important takeaway from this is that scale is now not only inevitable, it's happened and it's going to continue to happen. So let's, for a moment, kind of rewind and talk very quickly about science, my favorite subject, the scientific method. 

If you are to be a scientist, you have to do a few key things. If you don't do these things you can not consider yourself a scientist and you will struggle being able to understand your area of discipline. 

So first of all, you would formulate your question, you generate hypotheses, but more importantly, you predict your results-- you have a guess as to what the results will be. And then finally, you test your hypothesis and analyze your results. 

So this scientific method is extremely important in computing. Computing of both the prediction and being able to test your results are a key part of what we need to do in the scientific method. These predictions and testings are the real two cornerstones of the scientific method, and each require the most significant advances in modern computation. 

The two pillars of science are that of theory and that of experimentation. And more recently, computing is often mentioned as being the third pillar of science. So if you students are watching this, you have absolutely no pressure. Third pillar of science-- no big deal-- computing, kind of important. So glad this is the computing part of computer science course 50. 

So enough of the background. I want to tell you the plan of what we're going to talk about today. I'm going to go over some history. I'm going to explain why we got here. I'm going to talk about some of the history of the computing here at Harvard, some activities around social media, green things-- very passionate about all things green-- storage-- computer storage-- how chaos affects scale-out out systems, and distributive systems in particular. 

And then I'm going to touch on some of the scale-out hardware that's required to be able to do computing at scale. And then finally, we're going to wrap up with some awesome science. 

So, let's take a minute to look at our actual history. Computing has evolved. So since the '60s, all the away through to today, we've seen basically a change of scope from centralized computing to decentralize computing, to collaborative and then independent computing and right back again. 

And let me annotate that a little bit. When we first started off with computers, we had mainframes. They were inordinately expensive devices. Everything had to be shared. The computing was complex. You can see, it filled rooms and there were operators and tapes and all sorts of whirry, clicky, spinny devices. 

Around the '70s early '80s, you started to see an impact of the fax machines. So you're starting to see computing start to appear back in laboratories and become closer to you. The rise of the personal computer, certainly in the '80s, early part of the decade, really changed computing. 

And there's a clue in the title, because it was called the personal computer, which meant it belonged to you. So as the evolution of computing continued, people realized that their personal computer wasn't really big enough to be able to do anything of any merit, or significant merit, in science. 

And so folks started to develop network device drivers to be able to connect PCs together to be able to build clusters. And so this begat the era of the Beowulf cluster. Linux exploded as a response to proprietary operating system, both cost and complexity. 

And then, here we are today, where, yet again, we're faced with rooms full of computer equipment and the ability to swipe one's credit card and get access to these computing facilities, remotely. 

And so you can then see, in terms of history impacting how we do computing today, it's definitely evolved from machine rooms full of computers through some personal computing all the way right back again to machine rooms full of computers. 

So this is my first cluster. So 2000, we built a computer system in Europe to effectively annotate the human genome. There's a lot of technology listed on the right hand side there that, unfortunately, is no longer with us. It's passed off to the great technology in the sky. 

The machine itself is probably equivalent of a few decent laptops today, and that just kind of shows you. However, we did carefully annotate the human genome and both protected it with this particular paper in Nature from the concerns the data being public or private. 

So this is awesome, right? So we've got a human genome. We've done computing. I'm feeling very pleased myself. I rolled up to Harvard in 2006, feeling a lot less pleased with myself. 

This is what I inherited. This is a departmental mail and file server. You can see here there's a little bit of tape that's used to hold the system together. This is our license and print server. I'm pretty sure there maybe passwords on some of these Post-it Notes. 

Not awesome. Pretty far from awesome. And so, I realize this little chart that I showed you at the beginning from sharing to ownership back to sharing, that we needed to change the game. And so we changed the game by providing incentives. And so human beings, as this little Wikipedia article says here, our purposeful creatures. And the study of incentive structures is essential to the study of economic activity. 

So we started to incentivize our faculty and our researchers. And so we incentivized them with a really big computer system. So in 2008, we built a 4,096 processor machine-- 10 racks, couple hundred kilowatts of power. 

What I think is interesting is it doesn't matter where you are in the cycle. This same amount of power and compute, the power is the constant. It was 200 kilowatts when we were building systems in Europe. It's two hundred kilowatts in 2008, and that seems to be the [? quanter ?] of small university-based computing systems. 

So Harvard today-- fast forward, I'm no longer sad panda, quite a happy panda. We have 60-odd thousand load balanced CPUs, and their climbing dramatically. We have 15 petabytes of storage, also climbing. Again, this 200 kilowatt increment, we seem to be adding that every six or so months. Lots and lots of virtual machines. And more importantly, about 1.8 megawatts of research computing equipment. 

And I'm going to come back to this later on, as to why I now no longer necessarily count how much CPU we have, but how big is the electricity bill. 20 other so dedicated research computing staff. And more importantly, we're starting to grow our GPGPUs. I was staggered at how much of this is being added on a day-to-day basis. So, history lesson over, right? 

So how do we get there from here? Let's look at some modern scale-out compute examples. I'm a little bit obsessed with the size and scale of social media. There are a number of extremely successful large scale computing organizations now on the planet, providing support and services to us all. So that's the disclaimer. 

And I want to start with a number of ounces in an Instagram. It's not actually a lead-in to a joke, it's not even that funny, actually, come to think of it. But anyway, we're going to look at ounces in Instagram. And we're going to start with "My bee and a flower." I was at [INAUDIBLE] Village and I took a little picture of a bee sitting on a flower. And then I started to think about what does this actually mean. And I took this picture off my phone and counted how many bytes are in it, and it's about 256 kilobytes. Which when I started, would basically fill a 5 and 1/4 inch floppy. And started to think, well, that's cool. 

And I started to look and do some research on the network. And I found out that Instagram has 200 million MAUs. I wasn't actually that sure what a MAU was. And a MAU, down here, is a monthly active user. 

So, 200 million MAUs-- pretty cool. 20 billion photographs-- so quite a lot of photographs. 60 million new photographs each and every day coming out at about .002 gig per photo. That's about five petabytes of disk just right there. And that's really not the central part of what we're going to talk about. That is small potatoes. Or as we say in England, tiny spuds. 

So let's look at the real elephant in the room-- unique faces. Again, let's measure in this new quanta call a MAU. Facebook itself has 1.3 billion MAUs. WhatsApp, which I hadn't even heard of until recently, it's some sort messaging service, is 500 million MAUs. Instagram, which we just talked about, 200 million MAUs. And Messenger, which is another messaging service, is also 200 million MAUs. 

So total that up, it's about 2.2 billion total users. Clearly there's some overlap, but that's equivalent to a third of the planet. And they send something in the region of 12 billion messages a day. And again, there's only 7 billion people on the planet. Not everyone has a smartphone. So this is insane numbers. 

And I'm going to argue that it's not even about the storage or the compute. And to quote the song, it's all about that graph. Here's our lovely Meghan Trainor down here, singing about all the bass. Note, she also has quite a bit of bass herself-- 207, well 218 million people have seen this young lady singing her song. 

So my argument is it it's all about the graph. So we took some open source software and started to look at a graph. And this is LinkedIn, so this is a Facebook for old people. And so, this is my LinkedIn graph. I have 1,200 or so nodes, so-called "Friends." And here's me at the top. And here's all of the interconnections. 

Now, think back to the Instagram story. Each one of these is not just the photo, it has a whole plethora of connections between this particular individual and many others. This is central piece is either a bug in the graph drawing algorithm, or this maybe David Malan, I'm not sure yet. So you can redraw the graphs in all sorts of ways-- gephi.gihub.io is where you can pull that software from. It's really cool for being able to organize communities. You can see here, this is Harvard and various other places that I've worked, because this is my work-related data. 

So just think about the complexity of the graph and all of the data that you pull along with. So meanwhile, back at FriendFace, right? We looked at the Instagram data that was of the order of five petabytes. No big deal. Still quite a lot of data, but no big deal in the greater scheme of things. 

From this article on the old internet, "Scaling the Facebook data warehouse to 300 petabytes." That's a whole different game changer now, when you're starting to think of data and the graph and what you bring along with. And their high data is growing of the order of 600 terrabytes a day. 

Now, you know, well, then-- I mean, 600 terrabytes a day, 300 petabytes-- they're also now starting to get very concerned about how to keep this stuff and to make sure this data stays around. And this gentleman here, Jay Parikh, is looking at how to store an exabyte of data. 

Just for those of you who are watching along at home, an exabyte-- 10 to the 18. It's got its own Wikipedia page, it's that big of a number. That is the size and scale of what we're looking at, to be able to store data. And these guys aren't mucking around, they're storing that amount of data. So one of the clues that they're looking at here is data centers for so-called cold storage. 

Which brings me to being green. And here is Kermit. He and I agree-- it's extremely difficult to be green, but we give it our best try. Kermit can't help it, he has to be green all the time, can't take his green-ness off at all. 

So, being concepts-- a few kind of core concepts of greenness, when it relates to computing. The one that is the most important is the longevity of the product. If your product has a short lifetime, you cannot, by definition, be green. The energy taken to manufacture a disk drive, a motherboard, a computer system, a tablet, whatever it may be, the longevity of your systems are a key part of how green you can be. 

The important part, as all of you are building software algorithms-- algorithm's a partial word for software, right? So, your algorithm design is absolutely critical in terms of how you are going to be able to make quick and accurate computations to use the least amount of energy possible. And I'll get to this in a little bit. 

Data center design-- you've seen that we already have thousands upon thousands of machines, sitting quietly in small, dark corners of the world, computing. Resource allocation-- how to get to the compute, to the storage, through the network. Operating systems are a key part of this, and a lot of virtualization to be able to pack more and more compute into a small space. 

I'll give you a small example from research computing. We needed more ping, more power, and more pipe. We needed more bigger, better, faster computers, and needed to use less juice. And we couldn't work out how to do this. I don't know if the hashtag gowest as probably been used by the Kardashian, but anyway, gowest. And we did. 

We picked up our operation and we moved it out to Western Massachusetts in a small mill town called Holyoke, just north of Chikopee and Springfield. We did this for a couple of reasons. The main one was that we had a very, very large dam. And this very large dam is able to put out 30 plus megawatts of energy, and it was underutilized at the time. 

More importantly, we also had a very complicated network that was already in place. If you look at where the network goes in the United States, it follows all the train tracks. This particular piece of network was owned by our colleagues and friends at Massachusetts Institute of Technology, and it was basically built all the way out to Route 90. 

So we had a large river tick, Route 90 tick, we had a short path of 100 miles, and a long path of about 1,000 miles. We did have to do a very large network splice, as you can see here, to basically put a link in, to be able to connect to Holyoke, but we had all of the requisite infrastructure-- ping, power, pipe. Life was good. And again, big dam. 

So we built basically the Massachusetts Green High Performance Computing Center. This was a labor of love through five universities-- MIT, Harvard, UMass, Northeastern, and BU. Five megawatt day one connected load. We did all sorts of cleverness with airside economizers to keep things green. And we built out 640-odd racks, dedicated for research computing. 

It was an old brownfield site, so we had some reclamation and some tidy-up and some clean-up of the site. And then we started to build the facility and, boom-- lovely facility with the ability to run sandbox computing, to have conferences and seminars, and also a massive data center floor. 

Here is my good self. I'm obviously wearing the same jacket. I maybe only have one jacket, but there's me and John Goodhue-- he's the executive director of the Center-- standing in the machine room floor, which, as you can see, is pretty dramatic, and it goes back a long, long way. 

I often play games driving from Boston out to Holyoke, pretending that I'm a TCP/IP packet. And I do worry about my latency driving around in my car. So that's the green piece. So let's just take a minute and think about stacks. So we're trying very carefully to build data centers efficiently, computing efficiently, make good selection for the computing equipment and deliver, more importantly, our application, be it a messaging service or a scientific application. 

So here are the stacks. So physical layer, all the way up through application-- hoping that this is going to be a good part of your course. OSI seven layer model is basically, you will live, eat, and breathe this throughout your computing careers. This whole concept of physical infrastructure-- wires, cables, data centers, links. And this is just describing the network. 

Up here is, well, obviously, this is an old slide, because this should say HTTP, because nobody cares about simple mail transport protocols, anymore. It's all happening in the HTTP space. So that's one level of stack. 

Here's another set of stacks, where you have a server, a host, a hypervisor, a guest, binary library, and then your application. Or, in this case, the device driver, a Linux kernel, native c, Java virtual machine, Java API, then Java applications, and so on and so forth. This is a description of a virtual machine. 

Holy stacks, Batman! Think about this in terms of how much compute you need to get from what's happening here, all the way up to the top of this stack, to then be able to do your actual delivery of the application. 

And if you kind of rewind and start to think about what it takes to provide a floating point operation, your floating point operation is a sum of the sockets, the number of cores in the socket, a clock, which is how fast can the clock turnover-- four gigahertz, two gigahertz-- and then the number of operations you can do in a given hertz. 

So those microprocessors today do between four and 6 FLOPs per clock cycle. And so a single-core 2.5 gig clock has a theoretical performance of about a mega FLOP, give or take. 

But, as with everything, we have choices. So and Intel Core 2, Nehalem Sandy Bridge, Haswell, AMD, take your choices-- Intel Atom . All of these processor architectures all have a slightly different way of being able to add two numbers together, which is basically their purpose in life. Must be tough. There's millions of them sitting in data centers, now though. 

Sor, flops per watt-- this is the big thing. So if I want to get more of this to get through this stack, faster, I've got to work on how many floating point operations a second, I can do, and then give them watt. And fortunately, folks have thought about this. 

So there's a large contest every year to see who can build the fastest computer that can diagonalize a matrix. It's called the Top 500. They pick the top from the best 500 computers on the planet that can diagonalize matrices. And you get some amazing results. 

A lot of those machines are between 10 and 20 megawatts. They can diagonalize matrices inordinately quickly. They don't necessarily diagonalized them as efficiently per watt, so there was this big push to look at what a green 500 list would look like. And here is the list from June. There should be a new one very shortly. 

And it calls out-- I'll take the top of this particular list. There's two specific machines-- one from the Tokyo Institute of Technology and one from Cambridge University in the United Kingdom. And these have pretty staggering mega flops per watt ratios. This one's 4,389, and the next one down is 3,631. 

I'll explain the difference between these two, in the next slide. But these are these are moderately sized test clusters. These are just 34 kilowatts or 52 kilowatts. There are some larger ones here-- this particular one at the Swiss National Supercomputing Centre. The take home message for this is that we're trying to find computers that can operate efficiently. 

And so, let's look at this top one, cutely called, the KFC. And a little bit of advertising here. This particular food company has nothing to do with this. It's the fact that this particular system is soaked in a very clever oil-based compound. And so they got their chicken fryer moniker when they first started to build these types of systems. 

But basically what they've taken here is a number of blades, put them in this sophisticated mineral oil, and then worked out how to get all the networking in and out of it. Then, not only that, they've put it outside so that it can exploit outside air cooling. It was pretty impressive. So you have to do all of this shenanigans to be able to get this amount of compute delivered for small wattage. 

And you can see this is the shape of where things are heading. The challenge is that regular air cooling is the economy of scale and is driving a lot of the development of both regular computing, and high performance computing. So, this is pretty disruptive. I think this is fascinating. It's a bit messy when you try to swap the disk drives, but it's a really cool idea. 

So not only that, there's a whole bunch of work being built around what we're calling the Open Compute Project. And so, more about that a little bit later. But the industry's starting to realize that the FLOPs per watt is becoming important. And you, as folks here, as you design your algorithms and you design your code, you should be aware that your code can have a knock-on effect. 

When Mark was sitting here in his dorm room writing Facebook 1.0, I'm pretty sure he had a view that it was going to be huge. But how huge it would be on the environment is a big dealio. And so all of ya'll could come up with algorithms that could be the next challenging thing for folks like me, trying to run systems. 

So let's just think about real world power limits. This paper by Landauer-- is not a new thing. 1961 this was published in the IBM Journal. This is the canonical "Irreversibility and Heat Generation in the Computing Process." And so he argued that machines inevitably perform logistic functions that don't have single-valued inverse. 

So that the whole part of this is that back in the '60s, folks knew that this was going to be a problem. And so the law of limits said 25 degrees C, a sort of canonical room temperature, the limit represents 0.1 electron volts. But theoretically, this is the theory, computer memory, operating at this limit could be changed at one billion bits a second. 

I don't know about you, but not come across many one billion bits a second data rate exchanges. The argument there was that only 2.8 trillions of a watt of power ought to ever be expanded. 

All right, real world example-- this is my electric bill. I'm 65% percent of that lovely data center I showed you, in this particular time. This is back in June last year. I've taken an older version so that we can and sort of anonymize a little bit. I was spending $45,000 a month for energy there. 

So the reason being there is that we have over 50,000 processes in room. So could you imagine your own residential electricity bill being that high? But it was for a 199 million watt hours over a month. 

So the question I pose is, can you imagine Mr. Zuckerberg's electric bill? Mine is pretty big, and I struggle. And I'm not alone in this is. There's a lot of people with big data centers. And so, I guess, full disclosure-- my Facebook friends a little bit odd. 

So my Facebook friend is the Prineville data center, which is one of Facebook's largest, newest, lowest energy data center. And they post to me, things like power utilization effectiveness, as in how effective is the data center versus how much energy you're putting into it, how much water are they using, what's the humidity and temperature. 

And they have these lovely, lovely plots. I think this is an awesome Facebook page, but I guess I'm a little bit weird. 

So one more power thing, research computing that I do is significantly different to what Facebook and Yahoo and Google and other on-demand, fully, always available services. And so I have the advantage that when ISO New England-- and ISO New England helps set the energy rates for the region. 

And it says it's extending a request to consumers to voluntarily conserve high energy, because of the high heat and humidity. And this was back on the 18th of July. And so I happily Tweet back, Hey, ISO New England, Green Harvard. We're doing our part over here in research computing. And this is because we're doing science. 

And as much as people say science never sleeps, science can wait. So we are able to quiesce our systems, take advantage of grade rates on our energy bill, and help the entire New England region by shedding many megawatts of load. So that's the unique thing that differs about scientific computing data centers and those that are in full production 24/7. 

So let's just take another gear here. So, I want to discuss chaos a little bit. And I want to put it in the auspices of storage. So for those that kind of were struggling getting their head around what petabytes of storage look like, this an example. And this is the sort of stuff I deal with all the time. 

Each one of these little fellas is a four terabyte hard drive, so you can kind of count them up. We're getting now between one to 1 and 1/2 petabytes in a standard industry rack. And we have rooms and rooms, as you saw in that earlier picture with John and I, full of these racks of equipment. So it's becoming very, very easy to build massive storage arrays 

It's mostly easy inside of Unix to kind of count up how things are going. So this is counting how many MAU points have I got there. So that's 423 intercept points. And then if I run some sketchy awk, I can add up, in this particular system, there was 7.3 petabytes of available storage. 

So that's a lot of stuff. And storage is really hard. And yet, for some reason, this is an industry trend. Whenever I talk to our researchers and our faculty and say, hey, I can run storage for you. Unfortunately, I have to recover the cost of the storage. I get this business. And people reference Newegg or they reference Staples or how much they can buy a single terabyte disk drive for. 

So this, you'll note here, that there's a clue. There's one disk drive here. And if we go back, I have many. Not only do I have many, I have sophisticated interconnects to be able to stitch these things together. So the risk associated with these large storage arrays is not insignificant. 

In fact, we took to the internet and we wrote a little story about a well-meaning, mild-mannered director of research computing-- happens to have a strange English accent-- trying to explain to a researcher what the no underscore backup folder actually meant. It was quite a long, little story, a good four minutes of discovery. 

And note, I have an awful lot less space than the lady that sings about all the bass. We're quite a few accounts lower. But anyway, this is an important thing to think about, in terms of what could go wrong. So if I get a disk drive, and I throw it in a Unix machine, and I start writing things to it, there's a magnet, there's a drive head, there's ostensibly, a one or a zero being written down on to that device. 

Motors-- spinny, twirly things always break. Think about things that break. It's always been spinny, twirly things. Printers, disk drives, motor vehicles, etc. Anything that moves is likely to break. 

So you need motors, you need drive firmware, you need SAS/SATA controllers, wires, firmware on the SAS/SATA controllers, low level blocks. Pick your storage controller file system code, whichever one it may be, how you stitch things together. And your virtual memory manager pages, DRAM fetch and stores. Then, you get another stack, which is kind of down the list on this one, algorithms, users. 

And if you multiply this up, I don't know how many, there's a lot of places where stuff can go sideways. I mean, that's an example about math. But it's kind of fun to think of how many ways things could go wrong, just for a disk drive. We're already at 300 petabytes, so imagine the number of disk drives you need at 300 petabytes that can go wrong. Not only that-- so that's storage. And that alludes to the person I'd like to see enter stage left, which is the Chaos Monkey. So at a certain point, it gets even bigger than just the disk drive problem. 

And so, these fine ladies and gentleman that run a streaming video service realized that their computers were also huge and also very complicated and also providing service to an awful a lot of people. They've got 37 million members-- and this slide's maybe a year or so old-- thousands of devices. There are billions of hours of video. They log billions of events a day. 

And you can see, most people watch the telly later on in the evening, and it far outweighs everything. And so, they wanted to be able to make sure that the service was up and reliable and working for them. So they came up with this thing called Chaos Monkey. It's piece of software which, when you think about talking about the title of this whole presentation, scale-out means you should test this stuff. It's no good just having a million machines. So the nice thing about this is, Chaos Monkey is a service which identifies groups of systems and randomly terminates one of the systems in a group. Awesome. So I don't know about you, but if I've ever built a system that relies on other systems talking to each other, you take one of them out, the likelihood of the entire thing working, diminishes rapidly. 

And so this piece of software runs around Netflix's infrastructure. Luckily, it says it runs only in business hours with the intent that engineers will be alert and able to respond. So these are the types of things we're now having to do to perturb our computing environments, to introduce chaos and to introduce complexity. 

So who, in their right mind, would willingly choose to work with a Chaos Monkey? Hang on, he seems to be pointing me. Well, I guess I should-- cute. But the problem is you don't get the choice. The Chaos Monkey, as you can see, chooses you. 

And this is the problem with computing at scale is that you can't avoid this. It's an inevitability of complexity and of scale and of our evolution, in some ways, of computing expertise. And remember, this is one thing to remember, Chaos Monkeys love snowflakes-- love snowflakes. A snowflake-- we've explained the Chaos Monkey-- but a snowflake is a server that is unique and special and delicate and individual and will never be reproduced. 

We often find snowflake service in our environment. And we always try and melt snowflake service. But if you find a server in your environment that is critical to the longevity of your organization and it melts, you can't put it back together again. So Chaos Monkey's job was to go and terminate instances. If the Chaos Monkey melts the snowflake, you're over, you're done. I want to talk about some hardware that we're seeing in terms of sort of scale-out activities too. And some unique things that are in and around the science activity. We are now starting to see, remember this unit of issue, this rack? So this is a rack of GPGPUs-- so general purpose graphics processing units. 

We have these located in our data center, 100 or so miles away. This particular rack is about 96 tera FLOPS of single-precision math able to deliver out the back of it. And we have order 130-odd cards in an instance that we-- multiple racks of this instance. 

So this is interesting in the sense that the general purpose graphics processes are able to do mathematics incredibly quickly for very low amounts of energy. So there's a large uptick in the scientific computing areas, looking at graphics processing units in a big way. 

So I ran some Mcollective through our puppet infrastructure yesterday, very excited about this. just short of a petaflop of single-precision. Just to be clear here , this little multiplier is 3.95. Double-precision math would be about 1.2, but my Twitter feed looked way better if I said we had almost a petaflop of single-precision GPGPUs. 

But it's getting there. It's getting to be very, very impressive. And why are we doing this? Because quantum chemistry, among other things, but we're starting to design some new photovoltaics. 

And so Alan Aspuru-Guzik, who's a professor in chemistry-- my partner in crime-- for the last few years. We've been pushing the envelope on computing. And the GPGPU is ideal technology to be able to do an awful lot of complicated math, very, very quickly. 

So with scale, comes new challenges. So huge scale-- you have to be careful how you wire this stuff. And we have certain levels of obsessive compulsive disorder. These pictures probably drive a lot of people nuts. And cabinets that aren't wired particularly well drive our network and facilities engineers nuts. Plus there's also airflow issues that you have to contain. 

So these are things that I would never have thought of. With scale, comes more complexity. This is a new type of file system. It's awesome. It's a petabyte. It can store 1.1 billion files. It can read and write to 13 gigabytes and 20 gigabytes a second-- gigabytes a second. So it can unload terabytes in no time at all. 

And it's highly available. And it's got amazing lookup rates-- 220,000 lookups a second. And there are many different people building these kind of systems. And you can see it here graphically. This is one of our file systems that's under load, quite happily reading at just short of 22 gigabytes a second. So that's cool-- so complexity. 

So with complexity and scale, comes more complexity, right? This is one of our many, many network diagrams, where you have many different chassis all supporting up into a main core switch, connected to storage, connecting to low latency interconnects. And then all of this side of the house, is just all of the management that you need to be able to address these systems from a remote location. So scale has a lot of complexity with it. 

Change gear again, let's go back and have a little spot of science. So, remember, research computing and this little shim-- little pink shim between the faculty and all of their algorithms and all of the cool science and all of this power and cooling and data center floor and networking and big computers and service desks and help desks and so forth-- and so, we're just this little shim between them. 

What we've started to see is that the world's been able to build these large data centers and be able to build these large computers. We've gotten pretty good at it. What we're not very good at is this little shim between the research and the bare metal and the technology. And it's hard. 

And so we've been able to hire folks that live in this world. And more recently, we spoke to the National Science Foundation and said, this scale-out stuff is great, but we can't get our scientists on to these big complicated machines. And so, there have been a number of different programs where we really were mostly concerned about trying to see if we could transform the campus infrastructure. 

There are a lot of programs around national centers. And so, ourselves, our friends at Clemson, University of Wisconsin Madison, Southern California, Utah, and Hawaii kind of got together to look at this problem. And this little graph here is the long tail of science. 

So this is-- it doesn't matter what's on this axis, but this axis is actually number of jobs going through the cluster. So there's 350,000 over whatever time period. These are our usual suspects along the bottom here. In fact, there's Alan Aspuru-Guzik, who we were just talking about-- tons and tons of compute, really effective, knows what he's doing. 

Here's another lab that I'll talk about in a moment-- John Kovac's lab . They've got it. They're good. They're happy. They're computing. Great science is getting done . And then, as you kind of come down here, there are other groups that aren't running many jobs. 

And why is that? Is it because the computing is too hard? Is it because they don't know how to? We don't know, because we've gone and looked. And so that's what this project is all about, is locally, within each of these regions, to look to avenues where we can engage with the faculty and researchers actually in the bottom end of the tail, and understand what they're doing. 

So that's something that we're actually passionate about. And that's something that science won't continue to move forward until we solve some of these edge cases. Other bits of science that's going up-- everyone seen the Large Hadron Collider. Awesome, right? This stuff all ran out at Holyoke. We built-- the very first science that happened in Holyoke was the collaboration between ourselves and Boston University. So it's really, really cool. 

This is a fun piece of science for scale. This is a digital access to a sky century at Harvard. Basically, it's a plate archive. If you go down Oxford-- Garden Street, sorry, you'll find one of the observatory buildings is basically full of about half a million plates. 

And these are pictures of the sky at night, over 100 years. So there's a whole rig set up here to digitize those plates, take pictures of them, register them, put them on a computer. And that's a petabyte and a half, just right there-- one small project. 

These are other projects. This Pan-STARRS project is doing a full wide panoramic survey, looking for near Earth asteroids and transient celestial events. As a molecular biophysicist, I love the word transient celestial event. I'm not quite sure what it is, but anyway, we're looking for them. 

And we're generating 30 terabytes a night out of those telescopes. And that's not really a bandwidth problem, that's like a FedEx problem. So you put the storage on the van and you send it whatever it is. 

BICEP is really interesting-- so background imaging of cosmic extra galactic polarization. When I first started working at Harvard seven or so, eight years ago, I remember working on this project and it didn't really sink home as to why polarized light from the cosmic microwave background would be important, until this happened. 

And this was John Kovac, who I talked to before, using millions upon millions of CPU hours, in our facility and others, to basically stare into the inside of the universe's first moments after the Big Bang, and trying to understand Einstein's general theory of relativity. It's mind blowing that our computers are helping us unravel and stare into the very origins of why we're here. 

So when you talk about scale, this is some serious scale. The other thing of scale is, that particular project hit these guys. And this is the response curve for BICEP [INAUDIBLE] This was our little survey. 

And you can see here, life was good until about here, which was when the announcement came out. And you have got literally seconds to respond to the scaling event which corresponds to this little dot here, which ended up shifting four or so terabytes of data through the web server that day-- pretty hairy. 

And so, these are the types of things that can happen to you in your infrastructure if you do not design for scale. We had a bit of a scramble that day, to be able to span out enough web service to keep the site up and running. And we were successful. This is a little email that's kind of cute. This is a mail to Mark Vogelsberger, and Lars Hernquist, who's a faculty member here at Harvard. More about Mark later. But I think this is one sort of sums up kind of where the computing is in research computing. Hey, team, since last Tuesday, you guys racked up over 28% of the new cluster, which combined is over 78 years of CPU in just three days. And I said, it's still only just Friday morning. This is pretty awesome! Happy Friday! 

Then I give them the data points. And so that was kind of interesting. So remember about Mark, he'll come back into the picture in a little bit. So scale-out computing is everywhere. 

We're even helping folks look at how the NBA functions, and where people are throwing balls from. I don't really understand this game too well, but seemingly, it's a big deal. There's hoops and bowls and money. 

And so, our database, we built a little 500 [INAUDIBLE] parallel processor cluster, a couple of terabytes of RAM, to be able to build this for Kirk and his team. And they're doing computing in a whole other way. Now this is project we're involved with that's absolutely fascinating, around neural plasticity connectomics and genomic imprinting-- three very heavy hitting areas of research that we fight with on a day-to-day basis. The idea that our brains are under plastic stress when we are young. And much of our adult behavior is sculpted by experience in infancy. So this is a big dealio. 

And so this is work that's funded by the National Institutes of Mental Health. And we are trying to basically, through a lot of large data and big data analysis, kind of peer into our human brain through a variety of different techniques. 

So I wanted to stop and kind of just pause for a little moment. The challenge with remote data centers is it's far away. It can't possibly work. I need my data close by. I need to do my research in my lab. 

And so I kind of took an example of a functional magnetic resonance imaging data set from our data center in Western Mass. and connected it to my desktop in Cambridge. And I'll play this little video. Hopefully it will kind of work. 

So this is me going through checking my GPUs are working. And I'm checking that VNC's up. And this is a clever VNC. This is a VNC with 3D pieces. And so, as you can see shortly, this is me spinning this brain around. I'm trying to kind of get it oriented. And then I can move through many different slices of MRI data. And the only thing that's different about this is, it's coming over the wire from Western Mass. to my desktop. And its rendering faster than my desktop, because I don't have a $4,000 graphics card in my desktop, which we have out Western Mass. Of course, I'm trying to be clever. I'm running GLX gears in the background, whilst doing all this, to make sure that I can stress the graphics card, and that it all kind of works and all the rest of it. But the important thing is, is this is 100 miles away. And you can see from this that there's no obvious latency. Things holding together fairly well. 

And so that, in and of itself, is an example and some insight into how computing and scale-out computing is going to happen. We're all working on thinner and thinner devices. Our use of tablets is increasing. 

So therefore, my carbon footprint is basically moving from what used to do that would've been a huge machine under my desk, to what is now a facility-- could be anywhere. It could be anywhere at all. And yet, it's still able to bring back high performance graphics to my desktop. 

So, getting near the end-- remember Mark? Well, smart lad is Mark. He decided that he was going to build a realistic virtual universe. That's quite a project, when you think you've got to pitch this. I'm going to use a computer, and I'm going to model the 12 million years after the Big Bang to represent a day. And then I'm going to do 13.8 billion years of cosmic evolution. All right. 

This actually uses a computer the was bigger than our computer, and it spilled over onto the national resources to our friends down in Texas. And to the national facilities, this was a lot of compute. But we did a lot of the simulation locally to make sure that the software worked and the systems worked. 

And it's days like this when you realize that you're supporting science at this level of scale, that people can now say things like, I'm going to a model a universe. And this is his first model. And this is his team's first model. There are many other folks that are going to come behind Mark, who are going to want to model with high resolution, with more specificity, with more accuracy. 

And so, in the last couple of minutes, I just want to show you this video of Mark and Lars's that to me, again, as a life scientist , is kind of cute. So this, at the bottom here, to orient you, this is telling you the time since the Big Bang. So we're at about 0.7 billion years. And this is showing the current update. So you're seeing at the moment, dark matter and the evolution of the fine structure and early structures in our known universe. 

And the point with this is that this is all done inside the computer. This is a set of parameters and a set of physics and a set of mathematics and a set of models that are carefully selected, and then carefully connected to each other to be able to model the interactions. 

So you can see some starts of some gaseous explosions here. And gas temperature is changing. And you can start to see the structure of the visible universe change. And the important part with this is, each little tiny, tiny, tiny dot is a piece of physics and has a set of mathematics around, informing its friend and its neighbor. 

So from a scaling perspective, these computers have to all work in concert and talk to each other efficiently. So they can't be too chatty. They have to store their results. And they have to continue to inform all of their friends. Indeed, you'll see now, this model's getting more and more complicated. There's more and more stuff going on. There's more and more material flying around. 

And this is what the early cosmos would've looked like. It was a pretty hairy place. There's explosions all over the place, powerful collisions. And formation of heavy metals and elements. And these big clouds smashing into each other with the extreme force. 

And so now we're 9.6 billion years from this initial explosion. You're starting to see things are kind of calmed down a little bit, just a little bit, because the energy is now starting to relax. And so the mathematical models have got that in place. And you're starting to see coalescence of different elements. And starting to see this thing kind of come together and slowly cool. 

And it's starting to look a little bit more like the night sky, a little bit. And it's [? QSing. ?] We're now 30.2 billion years and we're kind of done. And then what they did was that they took this model, and then looked at the visible universe. And basically then, were able to take that and overlay it with what you can see. And the fidelity is staggering, as to how accurate the computer models are. 

Of course, the astrophysicists and the research groups need even better fidelity and even higher resolution. But if you think about what I've been talking to you today through this little voyage through both storage and structure and networking and stacks, the important thing is, is scale-out computing essential? That was my original hypothesis-- back to our scientific method. 

I hope that at the early part of this I would predict that I would be able to explain to you about scale-out computing. And we kind of tested some of those hypotheses. We went through this conversation. And I'm just going to say scale-out computing is essential-- oh, yes, very much yes. 

So when you're thinking about your codes, when you're doing the CS50 final projects, when you're thinking about your legacy to humanity and the resources that we need to be able to run these computer systems, think very carefully about the FLOPS per watt, and think about the Chaos Monkey. 

Think about your snowflakes, don't do one-offs, reuse libraries, build reusable codes-- all of the things that the tutors have been teaching you in this class. These are fundamental aspects. They're not just lip service. These are real things. 

And if any of you want to follow me, I am obsessive with the Twitter thing. I've got to somehow give that up. But a lot of the background information is on our research computing website at rc.fas.harvard.edu. 

I try and keep a blog up to date with modern technologies and how we do distributive computing and so forth. And then our staff are always available through odybot.org. And odybot is our little helper. He often has little contests on his website too, where you can try and spot him around campus. He's the friendly little face of research computing. 

And I'll kind of wrap up there and thank you all for your time. And I hope you remember that scale-out computing is a real thing. And there are a lot of people who've got a lot of prior art who will be able to help you. And all of the best of luck with your future endeavors in making sure that our computing both scales, is high performing, and helps humanity more than anything else. So, thank you for your time.