1 00:00:00,000 --> 00:00:00,392 2 00:00:00,392 --> 00:00:02,100 GREG MITTLEIDER: Good evening, everybody. 3 00:00:02,100 --> 00:00:03,920 My name is Greg Mittleider, and on behalf of CS50 4 00:00:03,920 --> 00:00:05,628 I would like to thank you for joining us, 5 00:00:05,628 --> 00:00:11,210 both in person and online for the final in our series of the 2017 tech talks. 6 00:00:11,210 --> 00:00:16,850 Tonight we are very fortunate to have the founder and CEO of Sentry, Mr. 7 00:00:16,850 --> 00:00:17,910 David Cramer. 8 00:00:17,910 --> 00:00:20,240 Tonight he's going to talk to us about the importance 9 00:00:20,240 --> 00:00:25,610 of shipping continuously and getting product to market as soon as possible. 10 00:00:25,610 --> 00:00:28,845 Without further ado, David Cramer. 11 00:00:28,845 --> 00:00:29,720 DAVID CRAMER: Thanks. 12 00:00:29,720 --> 00:00:33,150 13 00:00:33,150 --> 00:00:40,110 So, disclaimer, I prepared this presentation as for a mixed background. 14 00:00:40,110 --> 00:00:43,417 So it doesn't go super high level, but it doesn't go super low level. 15 00:00:43,417 --> 00:00:46,500 Hopefully a lot of this applies when you're thinking about any principles, 16 00:00:46,500 --> 00:00:50,820 but it really applies to software engineering. 17 00:00:50,820 --> 00:00:52,920 As mentioned, I'm the founder and CEO of Sentry. 18 00:00:52,920 --> 00:00:55,200 I'm a software engineer by trade. 19 00:00:55,200 --> 00:01:00,010 Previously, I worked on infrastructure teams at Dropbox and Discuss. 20 00:01:00,010 --> 00:01:02,010 I'm sure you're all familiar with the companies. 21 00:01:02,010 --> 00:01:04,440 But they're fairly large technology companies, at least 22 00:01:04,440 --> 00:01:07,260 in terms of their footprint on the internet. 23 00:01:07,260 --> 00:01:10,020 And Dropbox, at the very least, is a very large company 24 00:01:10,020 --> 00:01:12,609 in general at this point. 25 00:01:12,609 --> 00:01:15,150 But suffice to say there's a lot of hard technology problems. 26 00:01:15,150 --> 00:01:17,066 And at both of these companies, I really honed 27 00:01:17,066 --> 00:01:20,910 in on things that revolve around infrastructure scale and developer 28 00:01:20,910 --> 00:01:22,390 productivity. 29 00:01:22,390 --> 00:01:25,339 So a lot of that is how do we make it easier to write software? 30 00:01:25,339 --> 00:01:27,630 How do we make it so we can ship software more quickly, 31 00:01:27,630 --> 00:01:28,645 and minimize mistakes? 32 00:01:28,645 --> 00:01:31,430 33 00:01:31,430 --> 00:01:34,150 And so today we're going to talk a little bit about that. 34 00:01:34,150 --> 00:01:36,108 This is not really a talk about Sentry, though. 35 00:01:36,108 --> 00:01:37,740 This sort of aligns with what we do. 36 00:01:37,740 --> 00:01:40,570 It's talk about what we call continuous iteration. 37 00:01:40,570 --> 00:01:45,340 And I'll get into definitions of that here in a little bit. 38 00:01:45,340 --> 00:01:47,590 But let's start about software development flow. 39 00:01:47,590 --> 00:01:50,930 So, if you've been using computers for any period of time, 40 00:01:50,930 --> 00:01:52,960 you have an operating system. 41 00:01:52,960 --> 00:01:57,260 And operating systems are probably the classic version of-- 42 00:01:57,260 --> 00:02:00,130 I don't want to say archaic, they saw benefits-- 43 00:02:00,130 --> 00:02:04,142 but an old release cycle, an old workflow that isn't necessarily 44 00:02:04,142 --> 00:02:05,100 the right idea anymore. 45 00:02:05,100 --> 00:02:08,860 So, often what you have is something on the order of a-- 46 00:02:08,860 --> 00:02:11,940 these days it's probably a 6 to 12 month release cycle. 47 00:02:11,940 --> 00:02:16,710 And what that means is you have, so say it's a 12 month release cycle. 48 00:02:16,710 --> 00:02:19,720 You spend 12 months probably on development 49 00:02:19,720 --> 00:02:25,916 and then another N months on QA, and a bunch of manual human processes. 50 00:02:25,916 --> 00:02:28,540 If you look at Microsoft, they have dedicated teams that go in, 51 00:02:28,540 --> 00:02:30,420 and they actually test everything. 52 00:02:30,420 --> 00:02:32,170 And there's a lot of reasons they do this. 53 00:02:32,170 --> 00:02:35,290 One is for stability, but the reason they have to do this 54 00:02:35,290 --> 00:02:37,600 is because of that process. 55 00:02:37,600 --> 00:02:41,420 And if we look at where technology's gone, especially with the internet, 56 00:02:41,420 --> 00:02:43,600 things don't really work that way anymore. 57 00:02:43,600 --> 00:02:45,850 Even these big companies, like Microsoft and Apple, 58 00:02:45,850 --> 00:02:48,912 are trying to do more rapid, smaller iteration cycles. 59 00:02:48,912 --> 00:02:50,620 And we'll go into a lot of those reasons, 60 00:02:50,620 --> 00:02:54,340 and why that matters, and sort of how we address that today. 61 00:02:54,340 --> 00:02:59,200 But a quick high level, the biggest benefits we look at those old processes 62 00:02:59,200 --> 00:03:01,370 are the big stability increase. 63 00:03:01,370 --> 00:03:03,525 So if you look at a lot of web properties today-- 64 00:03:03,525 --> 00:03:07,180 I'm trying to think of a recent problem-- 65 00:03:07,180 --> 00:03:11,530 but I was just talking to David earlier about Twitter, 66 00:03:11,530 --> 00:03:13,300 in the day when the fail whale was famous, 67 00:03:13,300 --> 00:03:16,270 which is when you would load Twitter and it was always down, 68 00:03:16,270 --> 00:03:18,670 because they didn't have enough capacity. 69 00:03:18,670 --> 00:03:23,290 That's kind of one of these issues that happens when you change rapidly. 70 00:03:23,290 --> 00:03:26,470 Another might be just like introduction of bugs that are not well tested 71 00:03:26,470 --> 00:03:27,309 or something. 72 00:03:27,309 --> 00:03:30,100 But when you have these long processes, you can spend a lot of time 73 00:03:30,100 --> 00:03:32,470 on the QA phase, and you actually need to. 74 00:03:32,470 --> 00:03:34,990 And you primarily need to, because if there is an issue, 75 00:03:34,990 --> 00:03:38,530 it's going to take you quite a while to come out with a patch for that. 76 00:03:38,530 --> 00:03:41,260 We see this often in software these days with security concerns. 77 00:03:41,260 --> 00:03:47,280 There was a pretty important Wi-Fi security vulnerability, 78 00:03:47,280 --> 00:03:51,490 it was a few weeks back at this point, and the day it was announced, 79 00:03:51,490 --> 00:03:53,354 vendors still did not have a patch ready. 80 00:03:53,354 --> 00:03:56,020 And that's because they have to go through a pretty long process 81 00:03:56,020 --> 00:03:57,460 to vet these patches to make sure they're not 82 00:03:57,460 --> 00:03:58,910 going to cause other problems. 83 00:03:58,910 --> 00:04:03,080 And this is kind of where we see the flaw of this old paradigm. 84 00:04:03,080 --> 00:04:06,310 So today the focus is really going to be on how do we fix that, 85 00:04:06,310 --> 00:04:09,670 and why and how do we want to ship continuously? 86 00:04:09,670 --> 00:04:13,410 I think the why is kind of self-explanatory. 87 00:04:13,410 --> 00:04:17,410 But let's start with the obvious case in software development. 88 00:04:17,410 --> 00:04:22,110 So this is how we described Sentry early on when we did our first venture 89 00:04:22,110 --> 00:04:24,060 capitalists pitches. 90 00:04:24,060 --> 00:04:28,590 You have a product, Twitter is not a good example here, 91 00:04:28,590 --> 00:04:29,840 but we'll actually use Sentry. 92 00:04:29,840 --> 00:04:33,680 And you've launched a new version or you've released a change, 93 00:04:33,680 --> 00:04:36,797 and somebody on Twitter is going to start yelling at you. 94 00:04:36,797 --> 00:04:39,380 You may not pay attention to Twitter, but you probably should. 95 00:04:39,380 --> 00:04:42,463 And they're going to say it's broken, and that's all they're going to say. 96 00:04:42,463 --> 00:04:44,280 So what happens is, in a larger company-- 97 00:04:44,280 --> 00:04:46,490 and this was even true at Dropbox at the time-- 98 00:04:46,490 --> 00:04:47,780 you have separate departments. 99 00:04:47,780 --> 00:04:49,160 So, say you have a support department. 100 00:04:49,160 --> 00:04:51,050 And they're going to address that customer's concern first, 101 00:04:51,050 --> 00:04:53,630 but they don't really know anything the customer customer's talking about, 102 00:04:53,630 --> 00:04:54,830 unless it's a known issue. 103 00:04:54,830 --> 00:04:58,480 So the customer is going to be like, I can't load dropbox.com. 104 00:04:58,480 --> 00:05:01,130 And the support team is going to go through some steps, that's 105 00:05:01,130 --> 00:05:03,970 kind of standard, they're going go back and forth with the customer collecting 106 00:05:03,970 --> 00:05:04,470 information. 107 00:05:04,470 --> 00:05:07,664 And then, if they deem it important, so say if there's enough customers, 108 00:05:07,664 --> 00:05:09,330 they're going to circle that internally. 109 00:05:09,330 --> 00:05:11,121 So they might have to go to multiple teams. 110 00:05:11,121 --> 00:05:13,507 So, say Dropbox is owned by a web team. 111 00:05:13,507 --> 00:05:15,590 And then there's an API team that works with that. 112 00:05:15,590 --> 00:05:18,320 And does it affect the mobile team. 113 00:05:18,320 --> 00:05:20,990 But they're going to go in this loop of finding out 114 00:05:20,990 --> 00:05:25,160 who to talk to, what the problem is, and eventually addressing that problem. 115 00:05:25,160 --> 00:05:29,120 Somebody is probably going to ship that update to production to the customers, 116 00:05:29,120 --> 00:05:30,472 and hopefully it's fixed. 117 00:05:30,472 --> 00:05:32,930 But then there's sort of like this need to communicate that 118 00:05:32,930 --> 00:05:34,730 back to the customer as well. 119 00:05:34,730 --> 00:05:38,060 And this is sort of a more concrete illustration of the problem 120 00:05:38,060 --> 00:05:40,250 with that archaic lifecycle process. 121 00:05:40,250 --> 00:05:43,170 It's a very long, inefficient process. 122 00:05:43,170 --> 00:05:47,240 And really what you want to do is slim up this machine in a lot of ways. 123 00:05:47,240 --> 00:05:51,530 And, fortunately we're here at a CS class, and a lot of this 124 00:05:51,530 --> 00:05:53,900 comes down to technology and automation, and how 125 00:05:53,900 --> 00:06:00,230 we cut out people as much as possible, or at least repurpose those people. 126 00:06:00,230 --> 00:06:04,939 So a few things to know about, these terms change every few years, 127 00:06:04,939 --> 00:06:06,980 but there's three things that we think about when 128 00:06:06,980 --> 00:06:07,880 we talk about these subjects. 129 00:06:07,880 --> 00:06:09,200 One is continuous integration. 130 00:06:09,200 --> 00:06:11,600 This idea has been around for a long time. 131 00:06:11,600 --> 00:06:13,600 I know a lot of people are familiar with GitHub. 132 00:06:13,600 --> 00:06:17,120 It's a lot of this, you take a poll request, 133 00:06:17,120 --> 00:06:21,920 you say is it going to actually be able to be shipped, like merged in your core 134 00:06:21,920 --> 00:06:22,460 codebase. 135 00:06:22,460 --> 00:06:26,810 But, also part of that is can you verify that it's going to be good? 136 00:06:26,810 --> 00:06:29,060 Can you have automated tests, or even human tests, 137 00:06:29,060 --> 00:06:30,810 it actually doesn't matter, but really you 138 00:06:30,810 --> 00:06:32,930 want automation that says this change is actually 139 00:06:32,930 --> 00:06:36,380 good enough to move to the next stage. 140 00:06:36,380 --> 00:06:39,560 And this has been a big trend for probably the past decade, 141 00:06:39,560 --> 00:06:41,237 but now it's sort of getting required. 142 00:06:41,237 --> 00:06:43,070 And this is the biggest thing that everybody 143 00:06:43,070 --> 00:06:45,252 needs to be able to move more quickly. 144 00:06:45,252 --> 00:06:47,210 And even the big companies, this is why they're 145 00:06:47,210 --> 00:06:50,924 able to do larger updates much more quickly. 146 00:06:50,924 --> 00:06:53,090 It's just because this huge investment in technology 147 00:06:53,090 --> 00:06:54,860 around automating this process. 148 00:06:54,860 --> 00:06:58,430 Continuous deployment is sort of one of these newer concepts. 149 00:06:58,430 --> 00:07:00,770 The simplest version of this is anybody at Sentry 150 00:07:00,770 --> 00:07:04,160 can go into a tool we have internally and they can click deploy. 151 00:07:04,160 --> 00:07:08,330 And that will take whatever is past the continuous integration phase, 152 00:07:08,330 --> 00:07:11,430 and it will deploy it to our customers in a matter of minutes. 153 00:07:11,430 --> 00:07:14,480 So it's not about always deploying everything, 154 00:07:14,480 --> 00:07:18,800 but it's about deploying anything when you deem it's ready and it's safe. 155 00:07:18,800 --> 00:07:23,600 So, it's giving ownership of that, and it's really automating that process 156 00:07:23,600 --> 00:07:26,210 and making it quick and painless. 157 00:07:26,210 --> 00:07:28,010 And then the last one-- 158 00:07:28,010 --> 00:07:31,910 observability, which really, at the end of today just means monitoring. 159 00:07:31,910 --> 00:07:35,630 This is kind of a newer industry trend where you build something that you're 160 00:07:35,630 --> 00:07:39,011 going to have some kind of user of. 161 00:07:39,011 --> 00:07:41,510 You don't want to launch it to your users or your customers, 162 00:07:41,510 --> 00:07:44,709 and then you have that Twitter scenario where it's broken, and then be like, 163 00:07:44,709 --> 00:07:45,500 well, I don't know. 164 00:07:45,500 --> 00:07:46,900 I didn't actually instrument anything. 165 00:07:46,900 --> 00:07:48,440 And I didn't have any monitoring set up. 166 00:07:48,440 --> 00:07:50,900 So observability is the idea that you do that from day one. 167 00:07:50,900 --> 00:07:54,110 So before you ever ship anything to your customers or have any users, 168 00:07:54,110 --> 00:07:56,489 you build in this instrumentation or this monitoring. 169 00:07:56,489 --> 00:07:59,030 But otherwise it's just the same thing at the end of the day. 170 00:07:59,030 --> 00:08:03,889 171 00:08:03,889 --> 00:08:05,930 When we're talking about doing this continuously, 172 00:08:05,930 --> 00:08:09,876 it's this feedback lifecycle that we want to shorten. 173 00:08:09,876 --> 00:08:11,000 We go back to that picture. 174 00:08:11,000 --> 00:08:14,240 And it's like, the customer has an issue, and it gets fixed. 175 00:08:14,240 --> 00:08:15,800 And the customer's aware of that fix. 176 00:08:15,800 --> 00:08:17,300 How do we make that as quick as possible? 177 00:08:17,300 --> 00:08:19,300 And that's, really at the end of the day, that's 178 00:08:19,300 --> 00:08:21,010 why we want this fast life cycle. 179 00:08:21,010 --> 00:08:23,850 180 00:08:23,850 --> 00:08:28,070 So there's some things if we're looking to implement this in a system. 181 00:08:28,070 --> 00:08:32,360 The first and foremost is that it has to be very fast to deliver these changes, 182 00:08:32,360 --> 00:08:34,370 and it has to be incremental. 183 00:08:34,370 --> 00:08:38,330 The best example I can give you-- and I don't know how much you all do peer 184 00:08:38,330 --> 00:08:39,419 review-- 185 00:08:39,419 --> 00:08:44,000 but if you've ever tried reviewing a 5,000 line change set, 186 00:08:44,000 --> 00:08:45,210 it's really, really hard. 187 00:08:45,210 --> 00:08:47,210 And it's hard because it's a lot of information, 188 00:08:47,210 --> 00:08:48,622 and you just don't have context. 189 00:08:48,622 --> 00:08:51,330 And, honestly, you're never going to be able to safely review it. 190 00:08:51,330 --> 00:08:56,120 So, part of this is minimizing what the component is that's changing. 191 00:08:56,120 --> 00:09:00,816 And this is valuable in so many things. 192 00:09:00,816 --> 00:09:03,440 And the way we sort of approach this, a measure for that, is we 193 00:09:03,440 --> 00:09:06,190 say, "Well, we need to be able to release changes at least daily." 194 00:09:06,190 --> 00:09:08,902 And that's sort of a gold standard for any major company. 195 00:09:08,902 --> 00:09:11,360 I think even Facebook these days still does daily releases. 196 00:09:11,360 --> 00:09:14,900 But they do it sort of on a weekly life cycle, where 197 00:09:14,900 --> 00:09:18,200 they are shipping things every day, but they rotate that to customers 198 00:09:18,200 --> 00:09:19,610 seven days later. 199 00:09:19,610 --> 00:09:22,790 And Dropbox also had a daily life cycle, with a similar paradigm. 200 00:09:22,790 --> 00:09:26,600 Sentry, we probably ship changes to production 20, 30 times a day, 201 00:09:26,600 --> 00:09:27,590 but it's a lot easier. 202 00:09:27,590 --> 00:09:29,500 We're only like 60 people. 203 00:09:29,500 --> 00:09:32,780 Things scale a little bit better at that size. 204 00:09:32,780 --> 00:09:35,480 The other thing is, when we're doing this, 205 00:09:35,480 --> 00:09:39,500 we want to keep the changes small, which helps because it minimizes the impact. 206 00:09:39,500 --> 00:09:42,260 But we still want to maintain that high quality by moving fast. 207 00:09:42,260 --> 00:09:45,680 Because, as an engineer, the thing you really want to do is just 208 00:09:45,680 --> 00:09:48,409 write more code, and sort of move forward with the project, 209 00:09:48,409 --> 00:09:51,200 but you don't want to constantly be shipping bugs to your customer. 210 00:09:51,200 --> 00:09:54,650 So, how do you ship this new update, but minimize the impact? 211 00:09:54,650 --> 00:09:58,730 And that comes down to automated testing and systems around that. 212 00:09:58,730 --> 00:10:00,980 To make this scalable, again automation. 213 00:10:00,980 --> 00:10:03,320 We need to minimize the humans involved. 214 00:10:03,320 --> 00:10:05,195 We don't actually do any manual QA at Sentry. 215 00:10:05,195 --> 00:10:07,570 I don't think we're nearly big enough for that to matter. 216 00:10:07,570 --> 00:10:09,630 But we've also spent a lot of time on automation, 217 00:10:09,630 --> 00:10:12,560 so we don't really need those kinds of processes. 218 00:10:12,560 --> 00:10:18,490 And then the last one, which is sort of a challenge, is you often-- 219 00:10:18,490 --> 00:10:21,050 if you look at Mac OS today, whenever it updates, 220 00:10:21,050 --> 00:10:23,180 there's nothing really exciting about it. 221 00:10:23,180 --> 00:10:27,680 But if you look back at Windows, when it was like Windows 98 to 2000 222 00:10:27,680 --> 00:10:29,494 was the next one, or 2000 to-- 223 00:10:29,494 --> 00:10:31,910 I think there was something in between that wasn't great-- 224 00:10:31,910 --> 00:10:35,940 but, Windows 2000, Windows 7, was a massive leap in technology, 225 00:10:35,940 --> 00:10:37,440 and product, and everything. 226 00:10:37,440 --> 00:10:39,410 And it was this really big splash. 227 00:10:39,410 --> 00:10:40,910 And you don't really have that anymore, because you're 228 00:10:40,910 --> 00:10:42,150 doing these smaller changes. 229 00:10:42,150 --> 00:10:45,330 You can still do this, but it's a little bit trickier 230 00:10:45,330 --> 00:10:46,872 and you have to invest in technology. 231 00:10:46,872 --> 00:10:49,288 I'm not actually going to cover that in this presentation. 232 00:10:49,288 --> 00:10:51,810 If you're curious, I can talk through how you go about it. 233 00:10:51,810 --> 00:10:56,820 But it often isn't a very important thing anymore. 234 00:10:56,820 --> 00:10:58,400 So this is our objective. 235 00:10:58,400 --> 00:11:03,770 These are the soft level requirements, if you will. 236 00:11:03,770 --> 00:11:06,630 And the biggest thing here is this is going 237 00:11:06,630 --> 00:11:10,080 to be riskier than doing that large release cycle, 238 00:11:10,080 --> 00:11:12,150 but we can minimize all those risks. 239 00:11:12,150 --> 00:11:16,110 To give you the best example of this, Sentry's core technology 240 00:11:16,110 --> 00:11:17,850 basically captures errors. 241 00:11:17,850 --> 00:11:21,990 And it doesn't really stop you from creating bugs, but what it does 242 00:11:21,990 --> 00:11:23,670 is tell you about them within seconds. 243 00:11:23,670 --> 00:11:26,520 And if you have a good process like this, you can actually go fix the issue 244 00:11:26,520 --> 00:11:27,353 and ship the update. 245 00:11:27,353 --> 00:11:29,969 So it's limited the scope of the impact on your customers. 246 00:11:29,969 --> 00:11:32,760 And there's a lot of other ways you can minimize that risk as well. 247 00:11:32,760 --> 00:11:35,640 But you just have to accept that it's going to be there. 248 00:11:35,640 --> 00:11:38,084 And this is a principle we think a lot about. 249 00:11:38,084 --> 00:11:40,500 I was trying to find this quote, and I couldn't dig it up. 250 00:11:40,500 --> 00:11:46,080 But I read something from, I think it's Paul Buchheit from Y Combinator, that 251 00:11:46,080 --> 00:11:50,100 was, it's something about this like 90% of your customers-- 252 00:11:50,100 --> 00:11:53,730 focusing on what the 90% needs and ignoring the 10%. 253 00:11:53,730 --> 00:11:55,690 You can sort of treat software in the same way. 254 00:11:55,690 --> 00:11:57,450 And we can take the risk in the same way. 255 00:11:57,450 --> 00:11:59,639 What's going to be 90% safe? 256 00:11:59,639 --> 00:12:02,430 And maybe there's this 10% risk that it's going to cause a problem, 257 00:12:02,430 --> 00:12:04,120 but we haven't fully vetted that. 258 00:12:04,120 --> 00:12:07,050 If we can minimize that, and be able to react quickly, 259 00:12:07,050 --> 00:12:09,195 that's actually a totally acceptable risk. 260 00:12:09,195 --> 00:12:11,070 And we sort of apply that rule on everything. 261 00:12:11,070 --> 00:12:13,050 What's the 90% good enough feature? 262 00:12:13,050 --> 00:12:15,700 What's the 90% good enough technical solution to this problem? 263 00:12:15,700 --> 00:12:18,360 264 00:12:18,360 --> 00:12:21,550 OK, so the four key components of this process-- 265 00:12:21,550 --> 00:12:24,130 and you can probably distill this down in different ways-- 266 00:12:24,130 --> 00:12:26,740 but these are what we see as the major principles behind this. 267 00:12:26,740 --> 00:12:29,530 So, integration we talked about. 268 00:12:29,530 --> 00:12:33,420 This often takes the form of manual QA or automated QA. 269 00:12:33,420 --> 00:12:36,160 And it can happen at different stages, but in this stage 270 00:12:36,160 --> 00:12:39,790 we're talking about before we've shipped it to any kind of customer. 271 00:12:39,790 --> 00:12:40,687 Deployment. 272 00:12:40,687 --> 00:12:42,520 It's very important, and this is repeatable. 273 00:12:42,520 --> 00:12:45,394 274 00:12:45,394 --> 00:12:47,935 If you've played with Heroku-- which, I was talking to David, 275 00:12:47,935 --> 00:12:50,230 and he mentioned you might in this class-- 276 00:12:50,230 --> 00:12:52,947 things like Heroku make it so you have a change set 277 00:12:52,947 --> 00:12:55,030 and your build, what you're shipping to customers, 278 00:12:55,030 --> 00:12:56,654 is always the same for that change set. 279 00:12:56,654 --> 00:13:00,640 So if you go back, and you ship an older version of your product, 280 00:13:00,640 --> 00:13:03,100 it's still the same old version that it was, 281 00:13:03,100 --> 00:13:06,430 even though you're shipping it six months later. 282 00:13:06,430 --> 00:13:07,240 Monitoring. 283 00:13:07,240 --> 00:13:09,430 There's a lot of stuff in here. 284 00:13:09,430 --> 00:13:13,050 We like to focus on what matters to application developers and not systems. 285 00:13:13,050 --> 00:13:16,852 So, in Sentry we have a small team of operations engineers. 286 00:13:16,852 --> 00:13:19,060 And they're responsible for just maintaining servers, 287 00:13:19,060 --> 00:13:20,080 at the end of the day. 288 00:13:20,080 --> 00:13:22,930 And they monitor things like, do we have enough disk space? 289 00:13:22,930 --> 00:13:25,360 Is everything online? 290 00:13:25,360 --> 00:13:27,429 Is a machine overloaded on CPU? 291 00:13:27,429 --> 00:13:30,220 But when we're developing product, we don't really care about that, 292 00:13:30,220 --> 00:13:32,030 and we shouldn't have to. 293 00:13:32,030 --> 00:13:33,700 I'll talk a lot about tools. 294 00:13:33,700 --> 00:13:37,690 And there's a lot of systems that are trying to take that off of our hands, 295 00:13:37,690 --> 00:13:41,910 which is great, because it allows us to focus on more relevant things. 296 00:13:41,910 --> 00:13:44,920 So, whatever our goals are, versus just reimplementing everything 297 00:13:44,920 --> 00:13:46,240 under the sun. 298 00:13:46,240 --> 00:13:48,100 And lastly is the feedback phase. 299 00:13:48,100 --> 00:13:51,481 And the analogy I was using before, Twitter, is a feedback mechanism. 300 00:13:51,481 --> 00:13:54,730 It's not a great one, but it's one where your customers will give you feedback 301 00:13:54,730 --> 00:13:58,780 whether it's, we love your product, or this is totally broken. 302 00:13:58,780 --> 00:14:02,320 The big thing is-- and this is the gray area in technology these days. 303 00:14:02,320 --> 00:14:04,300 Sentry fits in this world, but there's not 304 00:14:04,300 --> 00:14:08,580 a lot that is active feedback where it's automated. 305 00:14:08,580 --> 00:14:11,070 It's automated user feedback in a way. 306 00:14:11,070 --> 00:14:12,840 So we'll talk about some of those, too. 307 00:14:12,840 --> 00:14:16,010 But it's the last key important piece of this. 308 00:14:16,010 --> 00:14:17,331 And, OK good. 309 00:14:17,331 --> 00:14:19,580 I don't actually remember what order my slides are in. 310 00:14:19,580 --> 00:14:22,760 But this is how we think about the workflow, in what it 311 00:14:22,760 --> 00:14:25,710 looks like in any major technology company today. 312 00:14:25,710 --> 00:14:28,590 So, you're familiar with a lot of this stuff. 313 00:14:28,590 --> 00:14:31,250 But you're going to make some changes to the code. 314 00:14:31,250 --> 00:14:33,090 It's going to go through a testing phase. 315 00:14:33,090 --> 00:14:34,260 You're going to deploy it. 316 00:14:34,260 --> 00:14:35,840 And this phase actually might repeat. 317 00:14:35,840 --> 00:14:39,485 So you might go through a testing phase, deploy it to 10% of your audience. 318 00:14:39,485 --> 00:14:41,360 And then there's another automated phase that 319 00:14:41,360 --> 00:14:44,345 checks if it can be deployed to a larger set of your audience. 320 00:14:44,345 --> 00:14:46,970 Those are often very expensive processes, so you only find them 321 00:14:46,970 --> 00:14:48,590 at like Facebook and Google, and companies 322 00:14:48,590 --> 00:14:50,880 that have a lot of engineering time to invest in it. 323 00:14:50,880 --> 00:14:54,260 But then we go through what we consider sort of this automated monitoring QA 324 00:14:54,260 --> 00:14:56,300 phase, which is where things like Sentry, 325 00:14:56,300 --> 00:15:00,020 or any other monitoring, uptime monitoring might come into play. 326 00:15:00,020 --> 00:15:02,070 What you do with that information is up to you. 327 00:15:02,070 --> 00:15:04,850 But in the ideal world, you create a triage resolution phase, 328 00:15:04,850 --> 00:15:08,300 which actually might encompass this whole thing, where you're going back, 329 00:15:08,300 --> 00:15:09,209 fixing issues. 330 00:15:09,209 --> 00:15:11,000 And eventually we get to the success phase. 331 00:15:11,000 --> 00:15:13,500 And the success phase is also where we think about feedback. 332 00:15:13,500 --> 00:15:18,260 So, a good example of this is if you sign up for Sentry, 333 00:15:18,260 --> 00:15:19,400 and you hit an error. 334 00:15:19,400 --> 00:15:23,000 We track this in the system, and we have an internal SLA 335 00:15:23,000 --> 00:15:25,501 where we say we try to deal with that error in the same day. 336 00:15:25,501 --> 00:15:27,250 And as soon as we deal with that error, we 337 00:15:27,250 --> 00:15:29,570 go through this phase of pushing out a fix for it. 338 00:15:29,570 --> 00:15:31,490 And then, when we hit the success phase, what we actually do 339 00:15:31,490 --> 00:15:34,614 is reach out to any customer who was affected, because our systems actually 340 00:15:34,614 --> 00:15:35,300 track them all. 341 00:15:35,300 --> 00:15:38,591 And we're like, hey, you know this error you hit that you didn't tell us about? 342 00:15:38,591 --> 00:15:39,950 We fixed it. 343 00:15:39,950 --> 00:15:41,930 And this is kind of the newer train of thought 344 00:15:41,930 --> 00:15:43,880 that doesn't exist a lot in software. 345 00:15:43,880 --> 00:15:46,010 Often the only time you're going to talk to anybody from a company, 346 00:15:46,010 --> 00:15:47,470 is if you proactively reach out. 347 00:15:47,470 --> 00:15:50,170 348 00:15:50,170 --> 00:15:53,440 OK, so, we're going to dive into each of those phases a little bit. 349 00:15:53,440 --> 00:15:56,470 My goal with each of these is not so much implementation, 350 00:15:56,470 --> 00:15:58,700 but I want to cover high-level what we look for. 351 00:15:58,700 --> 00:16:01,630 And I want to talk about some of the tools that you can use. 352 00:16:01,630 --> 00:16:03,910 A big part of what we think of in the industry 353 00:16:03,910 --> 00:16:06,880 now is just how do we do less work. 354 00:16:06,880 --> 00:16:08,551 So, we don't want to run servers. 355 00:16:08,551 --> 00:16:10,300 We don't want to build our own monitoring. 356 00:16:10,300 --> 00:16:14,360 We want to use the best solutions for each thing and piece them together. 357 00:16:14,360 --> 00:16:16,310 So that's what we're going to go over here. 358 00:16:16,310 --> 00:16:17,518 The first one is integration. 359 00:16:17,518 --> 00:16:20,050 And this is arguably the most important thing. 360 00:16:20,050 --> 00:16:22,480 It's very hard, it's a very rigorous thing 361 00:16:22,480 --> 00:16:24,280 to get into play, doing automated testing, 362 00:16:24,280 --> 00:16:27,040 and writing very high quality tests. 363 00:16:27,040 --> 00:16:31,286 I can tell you, I've been doing software engineering for 15 years, 364 00:16:31,286 --> 00:16:33,160 and I don't think I've ever been in a company 365 00:16:33,160 --> 00:16:35,680 where this is a really solved problem. 366 00:16:35,680 --> 00:16:39,550 Even at Sentry, we do a lot of testing, and we still ship bugs 367 00:16:39,550 --> 00:16:42,670 every single day to production. 368 00:16:42,670 --> 00:16:45,100 But what this comes down to is a change control process. 369 00:16:45,100 --> 00:16:48,410 And, if you do an internship at a big company, 370 00:16:48,410 --> 00:16:50,500 you're going to be forced to deal with this. 371 00:16:50,500 --> 00:16:52,417 Startups, they often don't have a lot of this. 372 00:16:52,417 --> 00:16:55,166 But it comes down to something like, you're going to go on GitHub, 373 00:16:55,166 --> 00:16:58,106 you're going to propose a change through a pull request. 374 00:16:58,106 --> 00:16:59,980 You'll often have peer review of that change. 375 00:16:59,980 --> 00:17:01,270 That review kind of varies. 376 00:17:01,270 --> 00:17:03,190 Sometimes it's technical feedback. 377 00:17:03,190 --> 00:17:06,010 It's like, no, you should actually write the code this way. 378 00:17:06,010 --> 00:17:07,750 But oftentimes, it's design feedback. 379 00:17:07,750 --> 00:17:09,880 Or it's like, hey, did you think about this case? 380 00:17:09,880 --> 00:17:13,510 And that's really actually what we care about, because in parallel, 381 00:17:13,510 --> 00:17:16,020 often you have this automated verification phase. 382 00:17:16,020 --> 00:17:18,520 And that's actually going to test, is the code good? 383 00:17:18,520 --> 00:17:19,079 Is it valid? 384 00:17:19,079 --> 00:17:20,700 Is it correct? 385 00:17:20,700 --> 00:17:22,810 And this is honestly where we spend the most 386 00:17:22,810 --> 00:17:25,062 investment from engineering resources. 387 00:17:25,062 --> 00:17:28,270 And then the last one, which is sort of a culmination of all this automation, 388 00:17:28,270 --> 00:17:29,320 is can we ship this? 389 00:17:29,320 --> 00:17:31,060 And that's what they call integration. 390 00:17:31,060 --> 00:17:33,070 But we just call it "can we merge it?" 391 00:17:33,070 --> 00:17:36,190 which comes from a Git term, but it's like, 392 00:17:36,190 --> 00:17:39,446 can this go to production at this point? 393 00:17:39,446 --> 00:17:40,820 So a little bit in each of those. 394 00:17:40,820 --> 00:17:44,727 So, pretty much everybody I think is probably familiar with GitHub 395 00:17:44,727 --> 00:17:45,310 at this point. 396 00:17:45,310 --> 00:17:51,600 It's sort of the canonical solution for just doing any kind of code management. 397 00:17:51,600 --> 00:17:57,610 But when you are at Sentry, or many companies, what you are required to do 398 00:17:57,610 --> 00:18:00,340 is create a change request just like this. 399 00:18:00,340 --> 00:18:02,020 Outline what's going on. 400 00:18:02,020 --> 00:18:03,520 This is actually a change to Sentry. 401 00:18:03,520 --> 00:18:05,645 Our whole thing is open source, so you can actually 402 00:18:05,645 --> 00:18:07,300 dig this up if you want to. 403 00:18:07,300 --> 00:18:10,577 But it's going to go through this review phase, where often we're 404 00:18:10,577 --> 00:18:12,160 outlining what the design is for this. 405 00:18:12,160 --> 00:18:15,770 Sometimes it's as short as just a line, but oftentimes it's more complex 406 00:18:15,770 --> 00:18:17,620 and we go into details. 407 00:18:17,620 --> 00:18:20,890 And then we actually mandate that somebody reviews your code. 408 00:18:20,890 --> 00:18:23,440 The review may not provide any value at all, 409 00:18:23,440 --> 00:18:26,470 but it's one of those checks and balances that's 410 00:18:26,470 --> 00:18:29,247 kind of a lightweight cost that continue to iterate quickly. 411 00:18:29,247 --> 00:18:32,080 From there, there's a lot of automation that's built on top of this. 412 00:18:32,080 --> 00:18:33,310 And the great thing about GitHub is it works 413 00:18:33,310 --> 00:18:36,490 with a lot of these other technologies, these open source tools. 414 00:18:36,490 --> 00:18:38,920 I'll talk about some of those. 415 00:18:38,920 --> 00:18:41,830 I mentioned peer review already. 416 00:18:41,830 --> 00:18:43,765 But we sort of lump these in verification. 417 00:18:43,765 --> 00:18:45,640 And Sentry actually relies on a lot of these. 418 00:18:45,640 --> 00:18:49,570 So, I don't talk about each of these specifically in here. 419 00:18:49,570 --> 00:18:50,860 I'm happy to dig into them. 420 00:18:50,860 --> 00:18:54,100 But we do something that does visual testing. 421 00:18:54,100 --> 00:18:55,990 So it actually renders pages of our website, 422 00:18:55,990 --> 00:18:58,630 and we'll know if there's a difference. 423 00:18:58,630 --> 00:19:01,780 We do things that say, "Are there enough tests covering this code?" 424 00:19:01,780 --> 00:19:04,360 For example, if you were to change our billing code, 425 00:19:04,360 --> 00:19:08,290 we actually require that every line of code is run by tests. 426 00:19:08,290 --> 00:19:12,370 We have our standard just, did the tests pass? 427 00:19:12,370 --> 00:19:15,162 We have another set of visual testing in here. 428 00:19:15,162 --> 00:19:16,870 And then we actually have a security test 429 00:19:16,870 --> 00:19:20,740 as well, which just says, vulnerabilities introduced, or are 430 00:19:20,740 --> 00:19:23,770 there dependencies being added here that are a concern? 431 00:19:23,770 --> 00:19:27,310 And every single one of these is provided by a third party, 432 00:19:27,310 --> 00:19:30,760 and they just all integrate with GitHub very seamlessly. 433 00:19:30,760 --> 00:19:34,895 Most of these you can also use for free or dirt cheap, 434 00:19:34,895 --> 00:19:38,020 especially if you're just working on a side product that's very low volume. 435 00:19:38,020 --> 00:19:40,560 436 00:19:40,560 --> 00:19:42,940 Sort of what we think about in the tooling chain, 437 00:19:42,940 --> 00:19:45,840 so the big thing here is, sit infrastructure that's 438 00:19:45,840 --> 00:19:47,280 going to run whatever you tell it. 439 00:19:47,280 --> 00:19:49,530 So there's a lot of services that I showed back there, 440 00:19:49,530 --> 00:19:52,390 but those are just going around on top of these systems. 441 00:19:52,390 --> 00:19:55,470 So, what we strongly suggest is exploring GitHub and Travis CI. 442 00:19:55,470 --> 00:19:58,350 They pair together really, really nicely. 443 00:19:58,350 --> 00:20:00,010 The learning curve is not huge. 444 00:20:00,010 --> 00:20:03,090 But it's important to know there are a lot of tools in this industry. 445 00:20:03,090 --> 00:20:05,160 And this is, honestly, this is just the four 446 00:20:05,160 --> 00:20:06,900 I can remember off the top of my head. 447 00:20:06,900 --> 00:20:10,887 There's probably like 40 of them that just run say, your tests. 448 00:20:10,887 --> 00:20:13,470 Some of these are going to be things you have to run yourself. 449 00:20:13,470 --> 00:20:17,520 Jenkins is an older Java technology that I definitely 450 00:20:17,520 --> 00:20:19,770 do not recommend you try to set up. 451 00:20:19,770 --> 00:20:20,820 But it's tried and true. 452 00:20:20,820 --> 00:20:22,114 And a lot of companies use it. 453 00:20:22,114 --> 00:20:24,780 Whereas things like Codeship, and Circle, and Travis, and GitLab 454 00:20:24,780 --> 00:20:28,252 are newer, often SaaS services that try to be drop in, 455 00:20:28,252 --> 00:20:30,960 where you don't have to learn a lot and you can just wire this up 456 00:20:30,960 --> 00:20:33,150 to your projects. 457 00:20:33,150 --> 00:20:34,290 But yeah. 458 00:20:34,290 --> 00:20:35,900 Generally, explore this area. 459 00:20:35,900 --> 00:20:40,824 I truly think this is the most important thing you can commit yourself to 460 00:20:40,824 --> 00:20:42,990 in software engineering, is getting over this hurdle 461 00:20:42,990 --> 00:20:47,790 of we have to spend time on this integration phase. 462 00:20:47,790 --> 00:20:53,730 And actually, a quick aside, I joined Discuss about, I don't know, seven 463 00:20:53,730 --> 00:20:56,800 or eight years ago, and there were like 10 people on the team. 464 00:20:56,800 --> 00:21:01,200 And at the time I think we had something around the order of like 100 465 00:21:01,200 --> 00:21:04,840 million page views, we call it. 466 00:21:04,840 --> 00:21:07,539 So when somebody is loading something through our network-- 467 00:21:07,539 --> 00:21:10,080 I forget if that was a day-- but it was a significant amount. 468 00:21:10,080 --> 00:21:12,288 And the first week I'm there, I took down everything. 469 00:21:12,288 --> 00:21:13,650 I shipped a bug. 470 00:21:13,650 --> 00:21:16,157 And I shipped a bug because they had none of this in place. 471 00:21:16,157 --> 00:21:18,240 They had some automated tests that just never ran, 472 00:21:18,240 --> 00:21:20,880 so you had to run them manually. 473 00:21:20,880 --> 00:21:24,720 They didn't really have a lot of monitoring. 474 00:21:24,720 --> 00:21:27,570 I knew about these concepts, but I never really grasped 475 00:21:27,570 --> 00:21:28,860 how important they could be. 476 00:21:28,860 --> 00:21:31,320 Because, you can go in and be like, "Oh yeah, I tested my code." 477 00:21:31,320 --> 00:21:33,210 And that's easy if you have a very small app, 478 00:21:33,210 --> 00:21:37,102 but as soon as your change is affecting a large complex application, 479 00:21:37,102 --> 00:21:39,060 you're just not going to know what's happening. 480 00:21:39,060 --> 00:21:41,670 So a lot of what we build here is future-proof tests. 481 00:21:41,670 --> 00:21:45,719 I'm often not writing tests to say, the code I'm building right now is correct. 482 00:21:45,719 --> 00:21:48,510 What I'm doing is writing tests so when somebody changes that code, 483 00:21:48,510 --> 00:21:51,780 or changes how it interacts, those tests will fail and prevent somebody 484 00:21:51,780 --> 00:21:53,880 from causing some other significant issue. 485 00:21:53,880 --> 00:21:55,730 So this is a very, very important thing. 486 00:21:55,730 --> 00:21:57,870 There's a lot of technology out there, thankfully. 487 00:21:57,870 --> 00:21:59,220 But it is still very manual. 488 00:21:59,220 --> 00:22:03,840 And, the unfortunate part about at least this area of technology, 489 00:22:03,840 --> 00:22:05,490 is it changes a lot. 490 00:22:05,490 --> 00:22:08,310 I think even Sentry, we've gone through like four iterations 491 00:22:08,310 --> 00:22:14,620 in three years of what is a good system for these kinds of things. 492 00:22:14,620 --> 00:22:17,716 This is just an example of how you might configure Travis. 493 00:22:17,716 --> 00:22:19,590 There's a bunch of links, and I think you all 494 00:22:19,590 --> 00:22:21,131 have access to the slides afterwards. 495 00:22:21,131 --> 00:22:23,790 But this is a truncated version of one of our configurations. 496 00:22:23,790 --> 00:22:25,498 But you basically just say, OK, I'm going 497 00:22:25,498 --> 00:22:28,030 to run this language, which is Python in this case, 498 00:22:28,030 --> 00:22:30,114 I need a couple database servers running. 499 00:22:30,114 --> 00:22:32,280 And here's how I'm going to install my dependencies. 500 00:22:32,280 --> 00:22:33,750 And here's how I'm going to run tests. 501 00:22:33,750 --> 00:22:36,150 And you don't have to run tests, you can do whatever you want. 502 00:22:36,150 --> 00:22:38,940 But in this case we're just like using a standard testing framework. 503 00:22:38,940 --> 00:22:41,130 The way we can solve dependencies is just a standard way. 504 00:22:41,130 --> 00:22:42,920 So there's nothing really out of the norm. 505 00:22:42,920 --> 00:22:44,590 So it's often very easy to get started. 506 00:22:44,590 --> 00:22:47,120 507 00:22:47,120 --> 00:22:48,770 The next big thing is deployment. 508 00:22:48,770 --> 00:22:51,020 And this is, unfortunately, the area that there's not 509 00:22:51,020 --> 00:22:54,420 a lot of technology around. 510 00:22:54,420 --> 00:22:56,870 But let's talk about what we look for here. 511 00:22:56,870 --> 00:23:00,790 So number one is we want to offload control. 512 00:23:00,790 --> 00:23:05,040 So, if you go back, even five years ago, and even today at a lot of companies, 513 00:23:05,040 --> 00:23:06,990 there is what's called a release manager. 514 00:23:06,990 --> 00:23:10,170 And this person is often designated as the person that presses 515 00:23:10,170 --> 00:23:12,690 the deploy button, or ships code. 516 00:23:12,690 --> 00:23:16,840 And I remember at, I think even at Dropbox, there was a daily check in. 517 00:23:16,840 --> 00:23:19,265 And so these changes are all going to go live today. 518 00:23:19,265 --> 00:23:21,390 Are you here to sign off on your change going live? 519 00:23:21,390 --> 00:23:22,806 And then somebody hits the button. 520 00:23:22,806 --> 00:23:26,670 And the person is often a system admin, or an operations engineer. 521 00:23:26,670 --> 00:23:30,420 And the idea of continuous deployment is that goes away. 522 00:23:30,420 --> 00:23:34,020 We want you, the author of the change, to be able to ship your change, 523 00:23:34,020 --> 00:23:36,720 and be fully responsible and accountable for your change. 524 00:23:36,720 --> 00:23:38,550 That's like the distillation of this idea. 525 00:23:38,550 --> 00:23:40,675 But really, what it actually looks like in practice 526 00:23:40,675 --> 00:23:42,960 is, oh you're the team that's responsible for the API. 527 00:23:42,960 --> 00:23:44,910 You should be able to run the API yourselves. 528 00:23:44,910 --> 00:23:47,340 We shouldn't have to manage that as a larger company. 529 00:23:47,340 --> 00:23:48,990 So when you look at big companies now, what they're doing 530 00:23:48,990 --> 00:23:51,991 is they're building teams that all they do is build this platform layer. 531 00:23:51,991 --> 00:23:54,656 And the platform layer is intended to let you run your own test. 532 00:23:54,656 --> 00:23:57,570 It's intended to let you deploy your own code, monitor your own code. 533 00:23:57,570 --> 00:24:02,620 And they no longer have to act as the source of truth for our thing's good. 534 00:24:02,620 --> 00:24:04,591 And the good thing about this is that was it 535 00:24:04,591 --> 00:24:07,340 was always sort of a thing at the biggest companies in the world-- 536 00:24:07,340 --> 00:24:10,440 Google has probably had this for 15 years-- 537 00:24:10,440 --> 00:24:14,980 but now like even the tiniest companies have this kind of capability. 538 00:24:14,980 --> 00:24:16,501 So giving ownership is important. 539 00:24:16,501 --> 00:24:18,250 The repeatable builds is really important. 540 00:24:18,250 --> 00:24:20,500 And this is just sort of a thing to think about, 541 00:24:20,500 --> 00:24:23,190 if you ever go down this world because you'll often 542 00:24:23,190 --> 00:24:25,380 find that when you're building software now, 543 00:24:25,380 --> 00:24:27,150 you're using a whole mess of dependencies. 544 00:24:27,150 --> 00:24:31,062 And those dependencies are not very well controlled. 545 00:24:31,062 --> 00:24:33,270 And so you'll have something like, a version changes, 546 00:24:33,270 --> 00:24:35,850 or a dependency's dependency changes. 547 00:24:35,850 --> 00:24:39,090 And if you were to say, rebuild the same version of your app 548 00:24:39,090 --> 00:24:44,010 that uses those dependencies, and something changed under the hood, 549 00:24:44,010 --> 00:24:46,380 all of a sudden your app just might not function. 550 00:24:46,380 --> 00:24:49,679 And there was actually a very famous version of this-- 551 00:24:49,679 --> 00:24:50,970 I don't remember when this was. 552 00:24:50,970 --> 00:24:53,180 Like a year, maybe two ago. 553 00:24:53,180 --> 00:24:54,930 It's called left-pad if you Google for it. 554 00:24:54,930 --> 00:24:58,260 But somebody unpublished a dependency that was used by everything 555 00:24:58,260 --> 00:25:00,690 on the internet, and everybody's builds broke. 556 00:25:00,690 --> 00:25:04,620 It actually prevented a lot of companies from being able to deploy software 557 00:25:04,620 --> 00:25:06,750 for a day or something. 558 00:25:06,750 --> 00:25:09,390 So it's a very important thing, that until it bites you, 559 00:25:09,390 --> 00:25:11,970 you don't really care about. 560 00:25:11,970 --> 00:25:14,400 The next one is very hard I would say. 561 00:25:14,400 --> 00:25:18,160 Rollout strategies often don't exist at small companies. 562 00:25:18,160 --> 00:25:19,920 We don't really have one. 563 00:25:19,920 --> 00:25:22,170 The simplest version of a rollout strategy is it says, 564 00:25:22,170 --> 00:25:24,660 I'm going to deploy this to a staging environment first, 565 00:25:24,660 --> 00:25:28,270 and I'm going to verify it's working there, whether I manually visit, 566 00:25:28,270 --> 00:25:30,215 or I automatically check it. 567 00:25:30,215 --> 00:25:32,840 That's kind of an OK version of it, but it doesn't really scale 568 00:25:32,840 --> 00:25:33,940 and it's not all that useful. 569 00:25:33,940 --> 00:25:35,440 So we have that but we don't use it. 570 00:25:35,440 --> 00:25:36,940 We don't find value in it. 571 00:25:36,940 --> 00:25:40,020 But going back to Dropbox or Facebook, you're going to go in 572 00:25:40,020 --> 00:25:42,780 and you're going to say, OK, ha my change has been released today. 573 00:25:42,780 --> 00:25:44,760 But actually what's happened is Facebook employees 574 00:25:44,760 --> 00:25:46,384 are the only people that see my change. 575 00:25:46,384 --> 00:25:49,650 And they're using my change for a week before any customers see my change. 576 00:25:49,650 --> 00:25:52,608 And then maybe in a week it only goes in 1% of customers, or something. 577 00:25:52,608 --> 00:25:54,430 And in another week maybe it goes to 10%. 578 00:25:54,430 --> 00:25:56,370 So that's what I mean by a rollout strategy. 579 00:25:56,370 --> 00:25:58,281 And what that does, again it minimizes risk, 580 00:25:58,281 --> 00:26:00,030 and then you can often back out from that. 581 00:26:00,030 --> 00:26:02,320 So especially if you have a big enough company, 582 00:26:02,320 --> 00:26:04,320 if it's your employees that are hitting the bug, 583 00:26:04,320 --> 00:26:05,820 it's actually not really a big deal. 584 00:26:05,820 --> 00:26:07,500 They're invested in this. 585 00:26:07,500 --> 00:26:10,020 But you can't do that at a small organization, either. 586 00:26:10,020 --> 00:26:13,186 So there is a lot of technology that's being built to make this a little bit 587 00:26:13,186 --> 00:26:14,910 more approachable. 588 00:26:14,910 --> 00:26:17,369 But I think it's much too complex of a subject, 589 00:26:17,369 --> 00:26:20,160 and it's not something you're ever going to want to build yourself, 590 00:26:20,160 --> 00:26:22,727 so I'm only going to lightly touch on it. 591 00:26:22,727 --> 00:26:25,810 And then the last thing-- and this sort circles back to our change control 592 00:26:25,810 --> 00:26:26,320 process-- 593 00:26:26,320 --> 00:26:30,790 is not allowing anybody to circumvent integration. 594 00:26:30,790 --> 00:26:34,360 So, for a lot of companies, including ourselves, 595 00:26:34,360 --> 00:26:38,680 we have compliance regulation that says you cannot deploy any changes that 596 00:26:38,680 --> 00:26:40,310 circumvent the change control process. 597 00:26:40,310 --> 00:26:42,400 And basically all that says is you can not 598 00:26:42,400 --> 00:26:46,870 deploy a change to production that has not passed all the tests, 599 00:26:46,870 --> 00:26:50,050 and has not passed whatever those verification techniques you had were. 600 00:26:50,050 --> 00:26:52,383 And that's actually a pretty easy thing to get in place, 601 00:26:52,383 --> 00:26:54,310 but it does require rigor because you're like, 602 00:26:54,310 --> 00:26:56,950 the test is failing because, I don't know, 603 00:26:56,950 --> 00:26:58,840 it's a leap year, or something like that. 604 00:26:58,840 --> 00:27:02,599 Or the test is failing because it's just flaky. 605 00:27:02,599 --> 00:27:04,890 But in our world, we actually just have to accept that, 606 00:27:04,890 --> 00:27:06,400 and we say it doesn't matter. 607 00:27:06,400 --> 00:27:09,870 No matter what, we have to say, this blocks unless it's green. 608 00:27:09,870 --> 00:27:12,520 And so it ends up causing more time consumption, 609 00:27:12,520 --> 00:27:16,510 but overall it helps with general happiness. 610 00:27:16,510 --> 00:27:18,530 So this is kind of our agenda. 611 00:27:18,530 --> 00:27:20,530 The first thing I want to talk about is control. 612 00:27:20,530 --> 00:27:22,821 So, I was actually talking to David about this earlier. 613 00:27:22,821 --> 00:27:26,024 He mentioned that he talks a lot about Heroku 614 00:27:26,024 --> 00:27:28,690 and stuff, which I think is a great platform that gives control. 615 00:27:28,690 --> 00:27:32,187 You don't have to learn a lot, you can just deploy projects on it. 616 00:27:32,187 --> 00:27:34,270 This week was Hack Week at Sentry, and we actually 617 00:27:34,270 --> 00:27:37,754 used Firebase, which is owned by Google, to actually do a lot of prototypes. 618 00:27:37,754 --> 00:27:39,670 And it's another one of those kind of systems, 619 00:27:39,670 --> 00:27:41,470 it just provides even more for you. 620 00:27:41,470 --> 00:27:44,410 So in this case, I'm just using the Firebase command line 621 00:27:44,410 --> 00:27:48,460 to deploy this Hack Week project, and it's no different than Heroku. 622 00:27:48,460 --> 00:27:50,360 I'm able to create a project, spin it up. 623 00:27:50,360 --> 00:27:53,276 And in this case, I'm actually just deploying a JavaScript application 624 00:27:53,276 --> 00:27:54,717 with nothing else. 625 00:27:54,717 --> 00:27:57,550 But it's important that it's just giving it to me, that I can do it. 626 00:27:57,550 --> 00:27:58,510 I don't have to ask somebody else. 627 00:27:58,510 --> 00:28:01,450 Somebody else isn't getting a phone call in the middle of the night 628 00:28:01,450 --> 00:28:04,000 if I can't deploy because I need to ship a bug fix. 629 00:28:04,000 --> 00:28:06,340 It's really all on me at the end of the day. 630 00:28:06,340 --> 00:28:08,479 And now this works for a side project. 631 00:28:08,479 --> 00:28:10,270 But as soon as you are inside of a company, 632 00:28:10,270 --> 00:28:12,190 you don't really want anybody doing this. 633 00:28:12,190 --> 00:28:14,890 You want to have this controlled through some mechanism that 634 00:28:14,890 --> 00:28:16,900 ensures they can deploy, because-- 635 00:28:16,900 --> 00:28:20,140 One thing you'll note here, I'm actually not doing 636 00:28:20,140 --> 00:28:21,735 any kind of proper change control. 637 00:28:21,735 --> 00:28:24,610 This is me just saying, whatever's in master, I'm going to deploy it. 638 00:28:24,610 --> 00:28:26,740 I'm not saying, this build is green. 639 00:28:26,740 --> 00:28:28,360 It's passed tests. 640 00:28:28,360 --> 00:28:30,780 I don't know even know if I have tests on this. 641 00:28:30,780 --> 00:28:34,580 And so, it doesn't really fit all of our requirements. 642 00:28:34,580 --> 00:28:38,271 So, what you'll often find, and unfortunately this 643 00:28:38,271 --> 00:28:41,020 is one of those areas that there's not a lot of technology around, 644 00:28:41,020 --> 00:28:43,390 is a company will have their own system for controlling 645 00:28:43,390 --> 00:28:44,710 how these deploys go out. 646 00:28:44,710 --> 00:28:48,710 The old school way is it's owned by a person and they press the button. 647 00:28:48,710 --> 00:28:50,980 But in Sentry's case, we actually built something 648 00:28:50,980 --> 00:28:52,900 which anybody in the company-- 649 00:28:52,900 --> 00:28:57,070 I think even like if you're an accountant, or an office 650 00:28:57,070 --> 00:28:58,070 manager of the company-- 651 00:28:58,070 --> 00:29:00,960 I think you could technically go in here and deploy Sentry. 652 00:29:00,960 --> 00:29:03,009 But you just have any service in here. 653 00:29:03,009 --> 00:29:05,800 You can deploy it, even if it's not your service, which isn't great 654 00:29:05,800 --> 00:29:07,130 but it's good enough. 655 00:29:07,130 --> 00:29:09,820 This is actually open source but I wouldn't worry about it. 656 00:29:09,820 --> 00:29:12,100 I think it Heroku has a little bit of this built in. 657 00:29:12,100 --> 00:29:14,475 But again, at the end of the day, this is often a company 658 00:29:14,475 --> 00:29:16,870 specific, because the way you ship code to production 659 00:29:16,870 --> 00:29:19,280 is often different between each company. 660 00:29:19,280 --> 00:29:22,030 Just because it's often very coupled to how your servers function. 661 00:29:22,030 --> 00:29:25,090 662 00:29:25,090 --> 00:29:26,090 The rollouts component. 663 00:29:26,090 --> 00:29:28,840 So again, I mentioned, there's not a lot of great technology here. 664 00:29:28,840 --> 00:29:32,530 I do want to highlight one piece. 665 00:29:32,530 --> 00:29:34,990 I only recommend exploring this if you're very, 666 00:29:34,990 --> 00:29:36,430 very into operations and systems. 667 00:29:36,430 --> 00:29:38,896 But there's a technology called Kubernetes. 668 00:29:38,896 --> 00:29:41,020 It's very interesting, very compelling around this. 669 00:29:41,020 --> 00:29:42,686 It makes it very easy to stage rollouts. 670 00:29:42,686 --> 00:29:45,430 So you could say, "I'm going to quickly spin up 671 00:29:45,430 --> 00:29:48,820 10 copies of the new version of the app, and as soon as those are working well 672 00:29:48,820 --> 00:29:50,350 I'm going to tear down the 10 old copies, 673 00:29:50,350 --> 00:29:52,141 and I'm just going to cut everything over." 674 00:29:52,141 --> 00:29:54,160 We call that a blue-green build. 675 00:29:54,160 --> 00:29:56,920 But Kubernetes is a nice technology that enables that. 676 00:29:56,920 --> 00:30:02,530 And it's very accessible on platforms like Google, their cloud platform. 677 00:30:02,530 --> 00:30:04,350 But it is a little bit tricky. 678 00:30:04,350 --> 00:30:07,330 As somebody who's been doing this for a very long time, 679 00:30:07,330 --> 00:30:10,740 I've never been a classic systems admin, but I sort of know my way around. 680 00:30:10,740 --> 00:30:15,679 And I sort have this side project, and it took me far too long, 681 00:30:15,679 --> 00:30:17,470 like embarrassingly long, to figure out how 682 00:30:17,470 --> 00:30:19,390 to make anything work on this platform. 683 00:30:19,390 --> 00:30:20,307 But it is very good. 684 00:30:20,307 --> 00:30:22,390 AUDIENCE: It's the same thing with Docker Compose. 685 00:30:22,390 --> 00:30:23,223 DAVID CRAMER: It is. 686 00:30:23,223 --> 00:30:26,140 It's very complex, but it's got a lot of good ideas. 687 00:30:26,140 --> 00:30:28,210 But it's often overwhelming. 688 00:30:28,210 --> 00:30:30,780 And Kubernetes is all built on top of Docker, 689 00:30:30,780 --> 00:30:35,260 so it's a good, like, connecting the dots together. 690 00:30:35,260 --> 00:30:37,010 You know, again, going into the ecosystem, 691 00:30:37,010 --> 00:30:38,210 there's kind of two ways we look at this. 692 00:30:38,210 --> 00:30:40,462 There's one, there's an abstraction of machines. 693 00:30:40,462 --> 00:30:42,170 We call that infrastructure as a service. 694 00:30:42,170 --> 00:30:44,280 It's AWS, it's Google, it's Azure. 695 00:30:44,280 --> 00:30:45,600 They're all good. 696 00:30:45,600 --> 00:30:47,180 It's, like, pick your poison here. 697 00:30:47,180 --> 00:30:48,763 We actually use Google for everything. 698 00:30:48,763 --> 00:30:50,900 We just transitioned all of our hardware there. 699 00:30:50,900 --> 00:30:52,460 I will say I'm very, very happy. 700 00:30:52,460 --> 00:30:55,794 Without picking sides, I think Google is the best. 701 00:30:55,794 --> 00:30:57,710 And it's a lot more accessible, which is nice. 702 00:30:57,710 --> 00:31:01,174 Amazon, if you've ever tried to understand what all their products do, 703 00:31:01,174 --> 00:31:04,340 it'll probably quickly turn you off, because there's just too much going on. 704 00:31:04,340 --> 00:31:07,700 And it's not very user friendly, whereas Google doesn't really give you much. 705 00:31:07,700 --> 00:31:10,380 They have servers, they have cloud storage, 706 00:31:10,380 --> 00:31:13,924 they have the whole Firebase platform, but there's not like 10 services 707 00:31:13,924 --> 00:31:17,090 that you don't really understand what the name of the service translates to. 708 00:31:17,090 --> 00:31:18,320 So it's kind of a nice, middle ground. 709 00:31:18,320 --> 00:31:19,890 I haven't personally used Azure. 710 00:31:19,890 --> 00:31:21,723 I'm sure it's also good, but it often caters 711 00:31:21,723 --> 00:31:24,029 to a different kind of technology. 712 00:31:24,029 --> 00:31:26,570 But then we go back up to what we consider more of a platform 713 00:31:26,570 --> 00:31:28,640 as a service which owns more of this. 714 00:31:28,640 --> 00:31:32,270 A lot of these are honestly built on top of AWS, or something else. 715 00:31:32,270 --> 00:31:35,570 Firebase, I think is great, but it's probably better 716 00:31:35,570 --> 00:31:37,700 if you're trying to build a JavaScript application. 717 00:31:37,700 --> 00:31:41,750 Heroku is really good, but it doesn't do a lot for you anymore. 718 00:31:41,750 --> 00:31:45,290 Zeit is just a newer one that we know the people behind. 719 00:31:45,290 --> 00:31:48,110 And it's kind of another JavaScript interesting platform, 720 00:31:48,110 --> 00:31:51,227 where it's sort of if you build your application this specific way, 721 00:31:51,227 --> 00:31:53,560 you don't have to worry about any of the infrastructure. 722 00:31:53,560 --> 00:31:56,060 And that's a really nice thing to get around. 723 00:31:56,060 --> 00:31:59,190 724 00:31:59,190 --> 00:31:59,690 OK. 725 00:31:59,690 --> 00:32:03,560 So, the third step is observability, monitoring invisibility, whatever 726 00:32:03,560 --> 00:32:05,150 you want to call this phase of it. 727 00:32:05,150 --> 00:32:08,450 But knowing when things go wrong. 728 00:32:08,450 --> 00:32:13,010 So, again, the goal is we do this from day one. 729 00:32:13,010 --> 00:32:16,310 Ideally, it doesn't take us much time, but we start with monitoring built in, 730 00:32:16,310 --> 00:32:20,180 so we don't have to add it when there's a problem later. 731 00:32:20,180 --> 00:32:22,280 Honestly, if that's all you take away from this, 732 00:32:22,280 --> 00:32:26,600 just drop in some third party service to whatever you're shipping out there, 733 00:32:26,600 --> 00:32:27,907 it's the best thing. 734 00:32:27,907 --> 00:32:29,990 There's a few important things that we care about. 735 00:32:29,990 --> 00:32:31,281 Like, it needs to be real time. 736 00:32:31,281 --> 00:32:34,940 And when we say real time, we need to know about something-- 737 00:32:34,940 --> 00:32:37,790 ideally it's even better than a minute resolution. 738 00:32:37,790 --> 00:32:39,415 We want to know within like 10 seconds. 739 00:32:39,415 --> 00:32:42,498 And we want to know within seconds, because if we can know within seconds, 740 00:32:42,498 --> 00:32:43,320 we can automate it. 741 00:32:43,320 --> 00:32:46,581 So, for example, we could deploy a change to production, 742 00:32:46,581 --> 00:32:49,080 and Sentry could tell us that there's an increase in errors. 743 00:32:49,080 --> 00:32:50,630 I'm not saying we do this, but we could. 744 00:32:50,630 --> 00:32:52,370 And then we can say, OK that's enough data 745 00:32:52,370 --> 00:32:54,370 that we're going to either roll back that change 746 00:32:54,370 --> 00:32:56,480 or promote the change to the next lev And that's 747 00:32:56,480 --> 00:32:59,620 why I like the real time matters. 748 00:32:59,620 --> 00:33:02,870 Graphs and logs are kind of a classical solution to things. 749 00:33:02,870 --> 00:33:06,262 And they're very useful, but they often don't tell you what's wrong. 750 00:33:06,262 --> 00:33:07,970 They tell you a lot of symptoms, but they 751 00:33:07,970 --> 00:33:09,740 don't go into what the root cause is. 752 00:33:09,740 --> 00:33:13,610 And the way we look at technology now, and the way we build Sentry is, we say, 753 00:33:13,610 --> 00:33:16,174 what is it that we actually are trying to fix? 754 00:33:16,174 --> 00:33:18,090 If we're going to go in and address a problem, 755 00:33:18,090 --> 00:33:20,548 what is the actual problem we're going to need to identify? 756 00:33:20,548 --> 00:33:22,820 And so for Sentry, we say, let's identify that problem 757 00:33:22,820 --> 00:33:25,940 and tell people about it, rather than giving them graphs that says, 758 00:33:25,940 --> 00:33:28,564 like, you're running out of memory, or there's a lot of errors, 759 00:33:28,564 --> 00:33:29,864 or something along those lines. 760 00:33:29,864 --> 00:33:32,780 But there are some good systems out there that will take all this data 761 00:33:32,780 --> 00:33:33,800 and look for anomalies. 762 00:33:33,800 --> 00:33:37,570 That can often help with more complex problems. 763 00:33:37,570 --> 00:33:40,070 And then last, and I think the most important, change that's 764 00:33:40,070 --> 00:33:43,760 happened over the years here, is there's a drastic shift between what 765 00:33:43,760 --> 00:33:46,550 we consider monitoring, where there's application, 766 00:33:46,550 --> 00:33:48,425 and there's also systems monitoring. 767 00:33:48,425 --> 00:33:50,600 And systems monitoring is what has always existed, 768 00:33:50,600 --> 00:33:52,880 and that's really where the graphs and logs come in. 769 00:33:52,880 --> 00:33:55,540 Application monitoring is what we consider Sentry. 770 00:33:55,540 --> 00:33:57,470 It's what, if you've ever looked at New Relic, 771 00:33:57,470 --> 00:33:59,210 a lot of it's app-level monitoring. 772 00:33:59,210 --> 00:34:03,620 It tells you about what's wrong with your code, not sort of what 773 00:34:03,620 --> 00:34:05,990 it looks like on the system level. 774 00:34:05,990 --> 00:34:09,710 And this is where you really get deep insight, because often these systems 775 00:34:09,710 --> 00:34:11,360 are living inside of the code. 776 00:34:11,360 --> 00:34:14,489 So they can actually see full source code, they can know-- 777 00:34:14,489 --> 00:34:17,420 for example, Sentry knows who the user is that's acting on a request. 778 00:34:17,420 --> 00:34:20,476 It knows, in a lot of cases, in like Python, 779 00:34:20,476 --> 00:34:22,850 for example, we can tell you what variables are assigned, 780 00:34:22,850 --> 00:34:23,725 and things like that. 781 00:34:23,725 --> 00:34:26,630 And all of a sudden, we have a lot more context to dig into an issue, 782 00:34:26,630 --> 00:34:28,781 and you'll never get that with systems monitoring. 783 00:34:28,781 --> 00:34:32,449 784 00:34:32,449 --> 00:34:36,355 My goal with this was to sort of overwhelm you with a lot of options. 785 00:34:36,355 --> 00:34:37,480 There's way more than this. 786 00:34:37,480 --> 00:34:41,790 I tried to eliminate any that I don't think are interesting anymore. 787 00:34:41,790 --> 00:34:43,670 You kind of have two categories. 788 00:34:43,670 --> 00:34:47,257 And there's a chunk that are open source, which often means run yourself. 789 00:34:47,257 --> 00:34:49,090 And there's a chunk that are cloud services. 790 00:34:49,090 --> 00:34:53,110 And they often still have a free tier so there's enough to get started. 791 00:34:53,110 --> 00:34:58,600 Personally we only use Datadog in this list, I believe, and Sentry, of course. 792 00:34:58,600 --> 00:35:00,310 I guess we use Elastic for some things. 793 00:35:00,310 --> 00:35:02,518 But all of these sort of solve different constraints. 794 00:35:02,518 --> 00:35:04,480 So if you truly need an encompassing solution, 795 00:35:04,480 --> 00:35:07,907 which, if you're just building a hobby project or a small thing, 796 00:35:07,907 --> 00:35:09,490 you're not going to need most of this. 797 00:35:09,490 --> 00:35:11,410 But if you need to cover all the bases, it's 798 00:35:11,410 --> 00:35:15,060 going to be, like, pick five of these, or something. 799 00:35:15,060 --> 00:35:16,930 A few quick highlights. 800 00:35:16,930 --> 00:35:19,180 Prometheus and Datadog are both systems monitoring. 801 00:35:19,180 --> 00:35:21,280 They provide a lot of graphs and things like that. 802 00:35:21,280 --> 00:35:23,500 Datadog I think is actually very, very compelling. 803 00:35:23,500 --> 00:35:25,747 It's a very cool tool. 804 00:35:25,747 --> 00:35:28,330 Primarily you can do things like, I want to send a metric that 805 00:35:28,330 --> 00:35:29,909 says, here's my response time. 806 00:35:29,909 --> 00:35:32,950 But then I want to say, the response time is actually for this end point. 807 00:35:32,950 --> 00:35:35,500 And I can actually see overall response time. 808 00:35:35,500 --> 00:35:38,090 I can see the top 10 slowest end points, things like that. 809 00:35:38,090 --> 00:35:42,400 So it's like a nice spin on the classic systems monitoring. 810 00:35:42,400 --> 00:35:46,960 Some of these are what we would call APM, which I believe 811 00:35:46,960 --> 00:35:50,371 is application performance monitoring. 812 00:35:50,371 --> 00:35:52,620 But it's focused on performance at the end of the day. 813 00:35:52,620 --> 00:35:54,700 And that's what New Relic is. 814 00:35:54,700 --> 00:35:57,920 It's kind of what Zipkin is. 815 00:35:57,920 --> 00:35:59,780 It's a gray area on some of these others. 816 00:35:59,780 --> 00:36:01,779 Scout and Datadog dog both provide some of that, 817 00:36:01,779 --> 00:36:04,340 but it's really intended to say, my code is slow, why? 818 00:36:04,340 --> 00:36:05,902 Or, my network is slow, why? 819 00:36:05,902 --> 00:36:09,110 Which is an important problem, once you have different services communicating 820 00:36:09,110 --> 00:36:11,300 with each other. 821 00:36:11,300 --> 00:36:12,380 A few of these are logs. 822 00:36:12,380 --> 00:36:15,799 Papertrail is something we used to use, which is a very, 823 00:36:15,799 --> 00:36:19,090 I think they have a free tier, but it's a very cheap, cloud-based log solution, 824 00:36:19,090 --> 00:36:21,404 that you can literally just view the stream of your log 825 00:36:21,404 --> 00:36:23,570 instead of having to jump through a bunch of hurdles 826 00:36:23,570 --> 00:36:26,440 to see what's going on. 827 00:36:26,440 --> 00:36:29,860 Elastic is another version of that that's a little bit more complex. 828 00:36:29,860 --> 00:36:32,839 Stackdriver is any number of things. 829 00:36:32,839 --> 00:36:34,880 I think they do a little bit of what Sentry does. 830 00:36:34,880 --> 00:36:36,921 I think they do a little bit of what Zipkin does, 831 00:36:36,921 --> 00:36:38,604 a little bit of what New Relic does. 832 00:36:38,604 --> 00:36:41,270 But you'll often find, if something tries to do a lot of things, 833 00:36:41,270 --> 00:36:44,750 it doesn't do any of them very well. 834 00:36:44,750 --> 00:36:48,290 I'm not saying it's bad, I'm just saying that's the caveat you get with these. 835 00:36:48,290 --> 00:36:52,090 OpenTSDB is similar to Datadog as well. 836 00:36:52,090 --> 00:36:55,237 There's a mix of things here, I guess is the important point, 837 00:36:55,237 --> 00:36:57,070 that some are systems, some are application, 838 00:36:57,070 --> 00:36:59,170 but very few of these focus on your application. 839 00:36:59,170 --> 00:37:03,119 I would say it's Sentry, It's New Relic, and then some of these 840 00:37:03,119 --> 00:37:04,660 have a couple features that blend in. 841 00:37:04,660 --> 00:37:08,520 842 00:37:08,520 --> 00:37:10,560 The main thing here is just pick a service that 843 00:37:10,560 --> 00:37:13,950 solves the problem the best, and just use that service. 844 00:37:13,950 --> 00:37:17,130 Don't worry about if you need three different ones. 845 00:37:17,130 --> 00:37:20,220 We often have a lot of people that come to us, and they're like, well, 846 00:37:20,220 --> 00:37:21,220 we want to use Sentry. 847 00:37:21,220 --> 00:37:23,880 But if we use Sentry do we still need New Relic? 848 00:37:23,880 --> 00:37:24,740 And it's like, yes. 849 00:37:24,740 --> 00:37:27,726 If we use Sentry, do we still need Elastic or Login? 850 00:37:27,726 --> 00:37:28,350 It's like, yes. 851 00:37:28,350 --> 00:37:29,910 It doesn't solve the same problems. 852 00:37:29,910 --> 00:37:32,650 It solves newer problems that we've created for ourselves. 853 00:37:32,650 --> 00:37:35,230 854 00:37:35,230 --> 00:37:36,520 The other important thing-- 855 00:37:36,520 --> 00:37:37,520 root cause analysis. 856 00:37:37,520 --> 00:37:40,940 So this is actually a slide, I forget which. 857 00:37:40,940 --> 00:37:44,710 We put this together to describe what Sentry could be in a near term future 858 00:37:44,710 --> 00:37:47,062 where you have a complex application, no matter 859 00:37:47,062 --> 00:37:48,770 what these days, where at the very least, 860 00:37:48,770 --> 00:37:52,570 you have a website, or a mobile app, and then you have a server component that's 861 00:37:52,570 --> 00:37:53,867 communicating with it. 862 00:37:53,867 --> 00:37:56,450 And so now all of a sudden you have a distributed application. 863 00:37:56,450 --> 00:38:00,820 In the simplistic form is, I'm loading my mobile app. 864 00:38:00,820 --> 00:38:01,960 I click a button. 865 00:38:01,960 --> 00:38:03,180 It hits an API service. 866 00:38:03,180 --> 00:38:04,327 The API service errors. 867 00:38:04,327 --> 00:38:07,160 In the mobile app I see an error, in the API service I see an error, 868 00:38:07,160 --> 00:38:10,030 but is that enough context on their own? 869 00:38:10,030 --> 00:38:13,180 It often is, but it might help if I'm looking at the API call 870 00:38:13,180 --> 00:38:15,580 and know actually what happened in the mobile app. 871 00:38:15,580 --> 00:38:18,280 So we think of this as tracing, but it's high level. 872 00:38:18,280 --> 00:38:20,520 And this is again a very simplistic version of this. 873 00:38:20,520 --> 00:38:24,460 At larger services these days, this could be a hundred different things 874 00:38:24,460 --> 00:38:26,936 that are communicating with each other. 875 00:38:26,936 --> 00:38:30,459 But the main thing is here, what is actually the cause? 876 00:38:30,459 --> 00:38:32,750 Clicking a button in the mobile app might be the cause. 877 00:38:32,750 --> 00:38:34,315 The cause might be deep in the API. 878 00:38:34,315 --> 00:38:36,690 But we really want to distill down, whatever the error we 879 00:38:36,690 --> 00:38:38,114 saw, what caused it? 880 00:38:38,114 --> 00:38:40,530 And really we want to know what's the code that caused it. 881 00:38:40,530 --> 00:38:42,360 And when did it start happening? 882 00:38:42,360 --> 00:38:46,800 And when did the change set get introduced? 883 00:38:46,800 --> 00:38:47,430 OK. 884 00:38:47,430 --> 00:38:50,160 The last major thing, feedback. 885 00:38:50,160 --> 00:38:54,140 You can take this any way you think about it. 886 00:38:54,140 --> 00:38:57,660 Customer support, monitoring, it sort of all blends together in here. 887 00:38:57,660 --> 00:39:00,990 This is the area that I don't think there's been a lot of development in. 888 00:39:00,990 --> 00:39:04,650 Customer support still works the same way it did, probably 30 years ago. 889 00:39:04,650 --> 00:39:07,590 You email somebody, they email you back. 890 00:39:07,590 --> 00:39:10,930 It's probably a very bad process most of the time. 891 00:39:10,930 --> 00:39:13,110 But there's been a little bit of progress. 892 00:39:13,110 --> 00:39:15,390 Sentry does a little bit in this world. 893 00:39:15,390 --> 00:39:17,530 But let's go again, what are we looking for? 894 00:39:17,530 --> 00:39:19,800 So, we think of this like active and reactive. 895 00:39:19,800 --> 00:39:23,144 Active being, we want to know what's going on 896 00:39:23,144 --> 00:39:24,560 before somebody tells us about it. 897 00:39:24,560 --> 00:39:25,710 So that goes into the monitoring world. 898 00:39:25,710 --> 00:39:28,126 And reactive being, somebody sent us an email, or somebody 899 00:39:28,126 --> 00:39:29,340 complained on Twitter. 900 00:39:29,340 --> 00:39:32,470 It's sort of one of those things where if it's gotten to that stage, 901 00:39:32,470 --> 00:39:34,620 it's not ideal. 902 00:39:34,620 --> 00:39:38,850 I remember, when I was 15 there was some story about customer service 903 00:39:38,850 --> 00:39:42,300 that I'm just going to assume is fact, where it's like 9 out of 10 customers 904 00:39:42,300 --> 00:39:44,610 won't complain. 905 00:39:44,610 --> 00:39:47,820 And if that's any kind of statistic, that could be like 90 out of 100. 906 00:39:47,820 --> 00:39:50,600 Could be 99 out of 100 for all we know. 907 00:39:50,600 --> 00:39:53,850 And so if you're relying on that kind of feedback, It's not really good. 908 00:39:53,850 --> 00:39:56,040 And that's why Sentry focused on error monitoring 909 00:39:56,040 --> 00:40:00,810 so much, because errors are often the first sign of a real customer problem. 910 00:40:00,810 --> 00:40:04,060 So we want all the active, arguably everything we can, 911 00:40:04,060 --> 00:40:06,240 we want automated just like this whole process. 912 00:40:06,240 --> 00:40:09,567 And reactive, really what we want is not somebody raging on Twitter 913 00:40:09,567 --> 00:40:10,650 about their United Flight. 914 00:40:10,650 --> 00:40:13,590 We want them directly talking to United. 915 00:40:13,590 --> 00:40:15,630 So, you want to ask people for it. 916 00:40:15,630 --> 00:40:18,047 Say, like, whenever you have an opportunity-- 917 00:40:18,047 --> 00:40:20,130 and actually I'll show you some examples of this-- 918 00:40:20,130 --> 00:40:23,110 but you want to engage this like, will you give me feedback. 919 00:40:23,110 --> 00:40:26,010 It's not really different than anything else in life. 920 00:40:26,010 --> 00:40:28,967 921 00:40:28,967 --> 00:40:30,550 So Sentry has kind of grown very fast. 922 00:40:30,550 --> 00:40:33,120 And we're doing our first peer reviews in the company. 923 00:40:33,120 --> 00:40:35,250 And really it's just, say, give me feedback. 924 00:40:35,250 --> 00:40:36,990 You're probably not going to give me the feedback day to day, 925 00:40:36,990 --> 00:40:39,490 but I am asking you to give me feedback so I can be better. 926 00:40:39,490 --> 00:40:42,730 It's the same thing when you're doing engineering. 927 00:40:42,730 --> 00:40:45,980 Another thing-- and this is something I think people ignored for a long time-- 928 00:40:45,980 --> 00:40:47,814 is context is very important. 929 00:40:47,814 --> 00:40:49,730 And that's why a lot of this automation helps, 930 00:40:49,730 --> 00:40:51,992 because if we can automate collecting this feedback, 931 00:40:51,992 --> 00:40:53,450 we can automate collecting context. 932 00:40:53,450 --> 00:40:56,345 And the classic example is, customer writes 933 00:40:56,345 --> 00:40:58,220 in about a bug in your application and you're 934 00:40:58,220 --> 00:41:01,261 like, what browser are you using, or what operating system are you using? 935 00:41:01,261 --> 00:41:02,900 And that's such a simple question. 936 00:41:02,900 --> 00:41:06,077 Even if you just have a form that says tell me the browser operating system. 937 00:41:06,077 --> 00:41:07,910 But really, you don't need the form anymore. 938 00:41:07,910 --> 00:41:08,993 We have enough technology. 939 00:41:08,993 --> 00:41:10,674 It's really easy to collect this data. 940 00:41:10,674 --> 00:41:12,590 And so that's one thing Sentry does very well. 941 00:41:12,590 --> 00:41:14,465 We just collect all the data we possibly can. 942 00:41:14,465 --> 00:41:17,090 943 00:41:17,090 --> 00:41:21,140 And I call out here that you probably actually don't need to do a lot here. 944 00:41:21,140 --> 00:41:23,680 If you're a big business, this obviously matters a lot. 945 00:41:23,680 --> 00:41:27,140 If it's a hobby project, who's going to be mad at you? 946 00:41:27,140 --> 00:41:27,720 Your users? 947 00:41:27,720 --> 00:41:29,303 They probably don't pay you any money. 948 00:41:29,303 --> 00:41:31,220 If they do, you can do a little bit here. 949 00:41:31,220 --> 00:41:33,050 I would argue something like Sentry is all 950 00:41:33,050 --> 00:41:35,150 you really need to get off the ground. 951 00:41:35,150 --> 00:41:38,210 I was talking to David earlier and we were talking a little bit 952 00:41:38,210 --> 00:41:42,710 about what we do day to day to monitor our systems. 953 00:41:42,710 --> 00:41:46,824 And my analogy was I haven't had to look at any tool besides Sentry-- 954 00:41:46,824 --> 00:41:48,740 and I don't do development every day anymore-- 955 00:41:48,740 --> 00:41:50,930 but I haven't had to look at any other tool for years at this point. 956 00:41:50,930 --> 00:41:52,850 I don't have to look at system logs anywhere. 957 00:41:52,850 --> 00:41:54,800 I don't really look at graphs anymore. 958 00:41:54,800 --> 00:41:57,740 Sentry solves the major concerns I have, and those 959 00:41:57,740 --> 00:42:01,040 are the concerns of the end user. 960 00:42:01,040 --> 00:42:04,120 If the end user is not seeing a problem, does it really exist? 961 00:42:04,120 --> 00:42:07,590 962 00:42:07,590 --> 00:42:09,160 So going to the reactive feedback. 963 00:42:09,160 --> 00:42:10,410 I think this is the key thing. 964 00:42:10,410 --> 00:42:12,440 So, this is actually a Sentry feature here, 965 00:42:12,440 --> 00:42:15,590 but you've had a browser crash or a mobile app crash. 966 00:42:15,590 --> 00:42:18,170 You open it back up and it's like, hey it crashed 967 00:42:18,170 --> 00:42:20,300 do you want to let us know anything about it? 968 00:42:20,300 --> 00:42:23,690 Now, the truth is, you don't need any of that information. 969 00:42:23,690 --> 00:42:27,312 You already have the details from like that crash report, for example. 970 00:42:27,312 --> 00:42:29,270 The reason we do it, is because we feel there's 971 00:42:29,270 --> 00:42:31,853 a stronger emotional connection-- like a business connection-- 972 00:42:31,853 --> 00:42:36,200 that if the user feels like somebody is aware of that problem, 973 00:42:36,200 --> 00:42:40,970 they're probably going to be far less angry about that poor experience. 974 00:42:40,970 --> 00:42:46,580 But this is no different than a customer support form. 975 00:42:46,580 --> 00:42:51,670 Or, imagine your flight gets delayed and you're really mad, 976 00:42:51,670 --> 00:42:55,420 but when you're notified that your flight is delayed, the airline's like, 977 00:42:55,420 --> 00:42:56,570 hey, we're sorry about it. 978 00:42:56,570 --> 00:42:57,580 Can we help you with something? 979 00:42:57,580 --> 00:42:59,621 Instead of just telling you the flight's delayed. 980 00:42:59,621 --> 00:43:02,650 You want to ask for that feedback. 981 00:43:02,650 --> 00:43:04,510 Again, if you can automate it, that's great. 982 00:43:04,510 --> 00:43:07,480 In this case, we've automatically sent a crash report, 983 00:43:07,480 --> 00:43:09,905 which is just like error logging. 984 00:43:09,905 --> 00:43:13,030 And if they add this information, we'll just associate it with that report. 985 00:43:13,030 --> 00:43:15,238 So we've still automatically collected that feedback, 986 00:43:15,238 --> 00:43:18,010 and we've automatically enriched it with context, 987 00:43:18,010 --> 00:43:21,461 but this is just taking it another level. 988 00:43:21,461 --> 00:43:23,460 Another interesting way you'll get a lot of this 989 00:43:23,460 --> 00:43:25,505 is if you're doing any open source work. 990 00:43:25,505 --> 00:43:27,630 And I highly recommend just open source everything, 991 00:43:27,630 --> 00:43:29,760 unless you have a reason to keep it private, 992 00:43:29,760 --> 00:43:32,014 especially if it's a side project. 993 00:43:32,014 --> 00:43:33,430 Put it behind a protected license. 994 00:43:33,430 --> 00:43:34,950 It doesn't really matter. 995 00:43:34,950 --> 00:43:37,440 But we actually get a lot of people, some good, some bad, 996 00:43:37,440 --> 00:43:40,410 submitting feedback via our GitHub issue trackers. 997 00:43:40,410 --> 00:43:42,120 And this is actually one that was-- 998 00:43:42,120 --> 00:43:45,250 I mean, I took this screen yesterday, so it was submitted like this week, 999 00:43:45,250 --> 00:43:46,984 effectively-- 1000 00:43:46,984 --> 00:43:48,900 and it's just a problem they had with the SDK. 1001 00:43:48,900 --> 00:43:49,720 And they probably pay us money. 1002 00:43:49,720 --> 00:43:51,595 And they're still willing to open this GitHub 1003 00:43:51,595 --> 00:43:53,310 issue, which is really, really nice. 1004 00:43:53,310 --> 00:43:55,590 And it's just another channel that we make 1005 00:43:55,590 --> 00:43:58,440 accessible to allow them to give us feedback. 1006 00:43:58,440 --> 00:44:01,260 Again, as often as possible, how can we make 1007 00:44:01,260 --> 00:44:04,410 the experience good for them, when usually in these situations 1008 00:44:04,410 --> 00:44:06,890 it's really bad. 1009 00:44:06,890 --> 00:44:09,280 This is part of a Sentry crash report. 1010 00:44:09,280 --> 00:44:11,737 We have the basic things I talked about. 1011 00:44:11,737 --> 00:44:13,820 We know the browser, we know the operating system. 1012 00:44:13,820 --> 00:44:15,940 In that case we know the user, which is actually 1013 00:44:15,940 --> 00:44:18,277 super important for a lot of things. 1014 00:44:18,277 --> 00:44:20,110 This might be what a log message looks like. 1015 00:44:20,110 --> 00:44:21,470 It's not really useful. 1016 00:44:21,470 --> 00:44:25,120 We just log when there's a billing failure, because often this 1017 00:44:25,120 --> 00:44:29,376 could be caused, it won't actually throw an error, but it is an error. 1018 00:44:29,376 --> 00:44:31,250 But we actually do a bunch more in this case. 1019 00:44:31,250 --> 00:44:34,450 We're actually collecting a bunch of events that led up to this error. 1020 00:44:34,450 --> 00:44:37,434 And I will argue that most of these have not actually been valuable, 1021 00:44:37,434 --> 00:44:39,100 we don't need them to solve the problem. 1022 00:44:39,100 --> 00:44:42,080 But every so often we're like, oh, this is how it happened, 1023 00:44:42,080 --> 00:44:44,380 or oh, this is how we triggered that problem. 1024 00:44:44,380 --> 00:44:46,960 And it sort of lets us accelerate that process 1025 00:44:46,960 --> 00:44:50,800 of resolving the issue and again, going back to the customer. 1026 00:44:50,800 --> 00:44:54,460 1027 00:44:54,460 --> 00:44:58,716 Again, this is the only goal I think with software, ever. 1028 00:44:58,716 --> 00:45:00,590 There are going to be, some of you, I'm sure, 1029 00:45:00,590 --> 00:45:03,770 will actually go into technology build useful things in the world, 1030 00:45:03,770 --> 00:45:07,075 but most of us are going to build registration forms. 1031 00:45:07,075 --> 00:45:08,950 That's what I always joke about at companies. 1032 00:45:08,950 --> 00:45:13,750 But the reality is, a lot of the problems are not very, very hard. 1033 00:45:13,750 --> 00:45:16,270 And a lot of it's just being able to iterate very quickly, 1034 00:45:16,270 --> 00:45:18,750 and get through the process efficiently. 1035 00:45:18,750 --> 00:45:22,810 And this is actually an idea we proposed to one of our investors at some point. 1036 00:45:22,810 --> 00:45:26,565 You have a user that hits an issue, Sentry's going to notify you via Slack 1037 00:45:26,565 --> 00:45:29,214 or whatever your poison is. 1038 00:45:29,214 --> 00:45:30,130 We're going to fix it. 1039 00:45:30,130 --> 00:45:32,546 And then, what if we just email the customer automatically 1040 00:45:32,546 --> 00:45:33,520 that the bug was fixed. 1041 00:45:33,520 --> 00:45:36,103 We don't do that because there's like some concerns around it, 1042 00:45:36,103 --> 00:45:38,560 but it's an interesting idea. 1043 00:45:38,560 --> 00:45:42,162 Any human that was doing those processes could be doing something else instead. 1044 00:45:42,162 --> 00:45:43,870 One way we think about this is, you often 1045 00:45:43,870 --> 00:45:46,560 have large customer support teams at companies, 1046 00:45:46,560 --> 00:45:49,022 and those customer support are reactive feedback. 1047 00:45:49,022 --> 00:45:50,980 But you also have teams that you would describe 1048 00:45:50,980 --> 00:45:53,160 as what they call customer success. 1049 00:45:53,160 --> 00:45:54,940 And that's more active. 1050 00:45:54,940 --> 00:45:56,480 It's like account management. 1051 00:45:56,480 --> 00:45:58,690 And so what if every customer support person 1052 00:45:58,690 --> 00:46:01,060 could be doing active account management, instead 1053 00:46:01,060 --> 00:46:03,015 of responding to bugs all the time. 1054 00:46:03,015 --> 00:46:08,130 It feels like we'd all be a little bit more successful. 1055 00:46:08,130 --> 00:46:10,580 There's a lot of tools in this space as well. 1056 00:46:10,580 --> 00:46:14,750 There's different approaches here, so that's the main thing. 1057 00:46:14,750 --> 00:46:18,500 I would say these tools are not very well related to each other. 1058 00:46:18,500 --> 00:46:21,830 So Zendesk is a big support application. 1059 00:46:21,830 --> 00:46:24,350 You write an email, it goes into a ticket tracker basically. 1060 00:46:24,350 --> 00:46:26,360 It looks like GitHub issues. 1061 00:46:26,360 --> 00:46:31,330 FullStory actually does visual, sort of, what's your user doing on your website. 1062 00:46:31,330 --> 00:46:37,070 So, maybe you're releasing a feature, and you're not sure 1063 00:46:37,070 --> 00:46:40,494 why anybody's not using it. 1064 00:46:40,494 --> 00:46:42,410 FullStory might be able to paint you a picture 1065 00:46:42,410 --> 00:46:44,069 of what the users are actually doing. 1066 00:46:44,069 --> 00:46:45,860 They're going from this page, to this page, 1067 00:46:45,860 --> 00:46:48,710 and they're actually not even scrolling down like this far on the page. 1068 00:46:48,710 --> 00:46:51,918 But it's really designed to give you a better visual representation of what's 1069 00:46:51,918 --> 00:46:52,590 going on. 1070 00:46:52,590 --> 00:46:55,687 And these are often harder areas to automate. 1071 00:46:55,687 --> 00:46:57,770 You have to give them business logic to say, well, 1072 00:46:57,770 --> 00:46:59,720 what I care about is this goal. 1073 00:46:59,720 --> 00:47:03,120 And then, why is it not being hit, is very subjective. 1074 00:47:03,120 --> 00:47:05,471 So you'll find in the feedback landscape, 1075 00:47:05,471 --> 00:47:07,970 and you know again this blends with monitoring, some of this 1076 00:47:07,970 --> 00:47:11,240 is going to be hands on, and maybe one day we'll solve some of that. 1077 00:47:11,240 --> 00:47:13,820 But there's still a lot of work to do there. 1078 00:47:13,820 --> 00:47:18,239 Sentry, fundamentally, all we do is tell you when your code's broken. 1079 00:47:18,239 --> 00:47:21,030 There's a little bit outside of that, but that's at the core of it. 1080 00:47:21,030 --> 00:47:23,087 Intercom is kind of a blend of Zendesk. 1081 00:47:23,087 --> 00:47:24,920 But it also lets you do proactive messaging. 1082 00:47:24,920 --> 00:47:28,640 So Intercom's almost like a chat widget on a website. 1083 00:47:28,640 --> 00:47:31,670 But it does a few things with that kind of functionality. 1084 00:47:31,670 --> 00:47:33,532 But again, it garnishes reactive feedback. 1085 00:47:33,532 --> 00:47:36,740 So it's always there, somebody can just click it, and they can type something 1086 00:47:36,740 --> 00:47:38,180 in right away. 1087 00:47:38,180 --> 00:47:41,150 But you can also do active requesteds feedback in there. 1088 00:47:41,150 --> 00:47:43,460 So you can say, I want to send a message to everybody. 1089 00:47:43,460 --> 00:47:45,380 I want to tell them about this new product, 1090 00:47:45,380 --> 00:47:46,934 or this new feature of my product. 1091 00:47:46,934 --> 00:47:49,100 And it'll blast anybody that loads your application. 1092 00:47:49,100 --> 00:47:51,516 And then, again, you can engage more and get more feedback 1093 00:47:51,516 --> 00:47:52,870 through that mechanism. 1094 00:47:52,870 --> 00:47:54,620 And then the inevitable, somebody is going 1095 00:47:54,620 --> 00:47:57,356 to email you and be mad, or tweet, or Facebook, or whatever. 1096 00:47:57,356 --> 00:47:59,230 I don't know, do they Snapchat it these days? 1097 00:47:59,230 --> 00:48:01,520 I'm not really sure. 1098 00:48:01,520 --> 00:48:03,241 If not yet, soon, I'm sure. 1099 00:48:03,241 --> 00:48:05,990 But there's a lot of different ways you can collect this feedback. 1100 00:48:05,990 --> 00:48:07,740 I think the big thing that hasn't happened 1101 00:48:07,740 --> 00:48:09,823 in the industry, like I was saying, is there needs 1102 00:48:09,823 --> 00:48:11,570 to be more automation around this. 1103 00:48:11,570 --> 00:48:14,990 Even Sentry's automation around feedback is just, it's just a little bit. 1104 00:48:14,990 --> 00:48:17,620 We've just barely touched this kind of market. 1105 00:48:17,620 --> 00:48:20,850 1106 00:48:20,850 --> 00:48:21,350 OK. 1107 00:48:21,350 --> 00:48:23,600 The last little bit, and I'm not going to dive into this, 1108 00:48:23,600 --> 00:48:25,391 because I don't want to be super technical, 1109 00:48:25,391 --> 00:48:29,000 but if you're really looking to see how something simple could be wired up, 1110 00:48:29,000 --> 00:48:30,770 this is actually an application-- 1111 00:48:30,770 --> 00:48:33,950 I think it took me two days last week. 1112 00:48:33,950 --> 00:48:35,780 All it's plugged into is Firebase. 1113 00:48:35,780 --> 00:48:39,830 I actually didn't even have Travis configured the other day. 1114 00:48:39,830 --> 00:48:42,677 I think I did that after I made this slide. 1115 00:48:42,677 --> 00:48:44,510 And it's connected to Sentry, so we can know 1116 00:48:44,510 --> 00:48:47,547 if there's any kind of obvious thing that went wrong. 1117 00:48:47,547 --> 00:48:50,630 And what it is, and I actually don't have network working on this computer 1118 00:48:50,630 --> 00:48:53,690 right now, but we're doing Hack Week at Sentry right now, 1119 00:48:53,690 --> 00:48:55,766 which is no different than a hackathon. 1120 00:48:55,766 --> 00:48:58,640 It's basically, we say anybody that's in our engineering organization 1121 00:48:58,640 --> 00:49:01,070 can do whatever the hell they want for-- 1122 00:49:01,070 --> 00:49:04,140 unfortunately only four days this week-- but for the entire week. 1123 00:49:04,140 --> 00:49:05,680 And we wanted a product registry. 1124 00:49:05,680 --> 00:49:08,520 So we're like, OK, what's the quickest way we could get something out the door 1125 00:49:08,520 --> 00:49:10,145 so we could start collecting this data. 1126 00:49:10,145 --> 00:49:12,534 So we explored and used Firebase. 1127 00:49:12,534 --> 00:49:14,450 And you can actually load this up if you want. 1128 00:49:14,450 --> 00:49:16,400 I'm pretty sure I made it open source today. 1129 00:49:16,400 --> 00:49:18,770 If not, I can fix that. 1130 00:49:18,770 --> 00:49:21,110 But it's just using a simple Firebase app. 1131 00:49:21,110 --> 00:49:22,610 It's using boilerplate technology. 1132 00:49:22,610 --> 00:49:24,740 It's using React which is, if you're not familiar, 1133 00:49:24,740 --> 00:49:30,050 it's a very good mechanism for building a very responsive, single page 1134 00:49:30,050 --> 00:49:31,410 application. 1135 00:49:31,410 --> 00:49:34,490 It's using Firebase's database on the back end, which is actually 1136 00:49:34,490 --> 00:49:36,870 like, it's fully client connected. 1137 00:49:36,870 --> 00:49:40,010 So, when you load this application, it's actually 1138 00:49:40,010 --> 00:49:41,390 running entirely in your browser. 1139 00:49:41,390 --> 00:49:43,348 There's no server that it's communicating with, 1140 00:49:43,348 --> 00:49:44,230 other than Firebase. 1141 00:49:44,230 --> 00:49:47,990 So it's kind of a neat way to prototype something. 1142 00:49:47,990 --> 00:49:49,430 But it's also wired up to Travis. 1143 00:49:49,430 --> 00:49:52,010 You can sort of see how the lifecycle would work in there. 1144 00:49:52,010 --> 00:49:55,180 There's probably next to nothing in the reading, of course. 1145 00:49:55,180 --> 00:49:57,050 But if you're interested, explore it. 1146 00:49:57,050 --> 00:49:59,091 There's a lot of other stuff that sort of conveys 1147 00:49:59,091 --> 00:50:01,130 some of these ideas on our GitHub. 1148 00:50:01,130 --> 00:50:04,570 I mentioned one of the side projects, we have a project called Zeus 1149 00:50:04,570 --> 00:50:08,870 on GitHub, which sort of is an implementation of reporting 1150 00:50:08,870 --> 00:50:10,390 on top of continuous integration. 1151 00:50:10,390 --> 00:50:13,240 But again, All open source. 1152 00:50:13,240 --> 00:50:19,180 But yes I think that is it. 1153 00:50:19,180 --> 00:50:20,799