Rich Mogull, SVP of Cloud Security at FireMon, joins Corey on Screaming in the Cloud to discuss his career in cybersecurity going back to the early days of cloud. Rich describes how he identified that cloud security would become a huge opportunity in the early days of cloud, as well as how cybersecurity parallels his other jobs in aviation and emergency medicine. Rich and Corey also delve into the history of Rich’s involvement in the TidBITS newsletter, and Rich unveils some of his insights into the world of cloud security as a Gartner analyst. About RichRich is the SVP of Cloud Security at FireMon where he focuses on leading-edge cloud security research and implementation. Rich joined FireMon through the acquisition of DisruptOps, a cloud security automation platform based on his research while as CEO of Securosis. He has over 25 years of security experience and currently specializes in cloud security and DevSecOps, having starting working hands-on in cloud over 12 years ago. He is also the principle course designer of the Cloud Security Alliance training class, primary author of the latest version of the CSA Security Guidance, and actively works on developing hands-on cloud security techniques. Prior to founding Securosis and DisruptOps, Rich was a Research Vice President at Gartner on the security team. Prior to his seven years at Gartner, Rich worked as an independent consultant, web application developer, software development manager at the University of Colorado, and systems and network administrator.Rich is the Security Editor of TidBITS and a frequent contributor to industry publications. He is a frequent industry speaker at events including the RSA Security Conference, Black Hat, and DefCon, and has spoken on every continent except Antarctica (where he's happy to speak for free -- assuming travel is covered).Links Referenced: FireMon: https://www.firemon.com/. Twitter: https://twitter.com/rmogull Mastodon: [https://defcon.social/@rmogull](https://defcon.social/@rmogull) FireMon Blogs: https://www.firemon.com/blogs/ Securosis Blogs: https://securosis.com/blog TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today is Rich Mogull, SVP of Cloud Security over at FireMon now that I’m a bit too old to be super into Pokémon, so I forget which one that is. Rich, thanks for joining me. I appreciate it.Rich: Thank you. Although I think we need to be talking more Digimon than Pokémon. Not that I want to start a flame war on the internet in the first two minutes of the conversation.Corey: I don’t even have the level of insight into that. But I will say one of the first areas where you came to my notice, which I’m sure you’ll blame yourself for later, is that you are the security editor behind TidBITS, which is, more or less, an ongoing newsletter longer than I’ve been in the space, to my understanding. What is that, exactly?Rich: So, TidBITS is possibly the longest-running—one of the longest-running newsletters on the internet these days and it’s focused on all things Apple. So, TidBITS started back in the very early days as kind of more of an email, I think like, 30 years ago or something close to that. And we just write a lot about Apple and I’ve been reading about Apple security there.Corey: That’s got to be a bit of an interesting experience compared to my writing about AWS because people have opinions about AWS, particularly, you know, folks who work there, but let’s be clear, there is nothing approaching the zealotry, I think I want to call it, of certain elements of the Apple ecosystem whenever there is the perception of criticism about the company that they favor. And I want to be clear here to make sure I don’t get letters myself for saying this: if there’s an Apple logo on a product, I will probably buy it. I have more or less surrounded myself with these things throughout the course of the last ten years. So, I say this from a place of love, but I also don’t wind up with people threatening me whenever I say unkind things about AWS unless they’re on the executive team.Rich: So, it’s been a fascinating experience. So, I would say that I’m on the tail end of being involved with kind of the Mac journalist community. But I’ve been doing this for over 15 years is kind of what I first started to get involved over there. And for a time, I wrote most of the security articles for Macworld, or a big chunk of those, I obviously was writing over a TidBITS. I’ve been very lucky that I’ve never been on the end of the death threats and the vitriol in my coverage, even though it was balanced, but I’ve also had to work a lot—or have a lot of conversations with Apple over the years.And what will fascinate you is at what point in time, there were two companies in the world where I had an assigned handler on the PR team, and one was Apple and then the other was AWS. I will say Apple is much better at PR than [laugh] AWS, especially their keynotes, but we can talk about re:Invent later.Corey: Absolutely. I have similar handlers at a number of companies, myself, including of course, AWS. Someone has an impossible job over there. But it’s been a fun and exciting world. You’re dealing with the security side of things a lot more than I am, so there’s that additional sensitivity that’s tied to it.And I want to deviate for a second here, just because I’m curious to get your take on this given that you are not directly representing one of the companies that I tend to, more or less, spend my time needling. It seems like there’s a lot of expectation on companies when people report security issues to them, that you’re somehow going to dance to their tune and play their games the entire time. It’s like, for a company that doesn’t even have a public bug bounties process, that feels like it’s a fairly impressively high bar. On some level, I could just report this via Twitter, so what’s going on over there? That feels like it’s very much an enterprise world expectation that probably means I’m out of step with it. But I’m curious to get your take.Rich: Out of step with which part of it? Having the bug bounty programs or the nature of—Corey: Oh, no. That’s beside the point. But having to deal with the idea of oh, an independent security researcher shows up. Well, now they have to follow our policies and procedures. It’s in my world if you want me to follow your policies and procedures, we need a contract in place or I need to work for you.Rich: Yeah, there is a long history about this and it is so far beyond what we likely have time to get into that goes into my history before I even got involved with dealing with any of the cloud pieces of it. But a lot about responsible disclosure, coordinated disclosure, no more free bugs, there’s, like, this huge history around, kind of, how to handle these pieces. I would say that the core of it comes from, particularly in some of the earlier days, there were researchers who wanted to make their products better, often as you criticize various things, to speak on behalf of the customer. And with security, that is going to trigger emotional responses, even among vendors who are a little bit more mature. Give you an example, let’s talk about Apple.When I first started covering them, they were horrific. I actually, some of the first writing I did that was public about Apple was all around security and their failures on security disclosures and their inability to work with security researchers. And they may struggle still, but they’ve improved dramatically with researcher programs, and—but it was iterative; it really did take a cultural change. But if you really want to know the bad stories, we have to go back to when I was writing about Oracle when I was a Gartner analyst.Corey: Oh, dear. I can only imagine how that played out. They have been very aggressive when it comes to smacking down what they perceive to be negative coverage of anything that they decide they like.Rich: Yeah, you know, if I would look at how culturally some of these companies deal with these things when I was first writing about some of the Oracle stuff—and remember, I was a Gartner analyst, not a vulnerability researcher—but I’m a hacker; I go to Blackhat and DEF CON. I’m friends with the people who are smarter than me at that or have become friends with them over the years. And I wrote a Gartner research note saying, “You probably shouldn’t buy any more Oracle until they fix their vulnerability management process.” That got published under the Gartner name, which that may have gotten some attention and created some headaches and borderline legal threats and shade and all those kinds of things. That’s an organization that looks at security as a PR problem. Even though they say they’re more secure, they look at security as a PR problem. There are people in there who are good at security, but that’s different. Apple used to be like that but has switched. And then Amazon is… learning.Corey: There is a lot of challenge around basically every aspect of communication because again, to me, a big company is one that has 200 people. I think that as soon as you wind up getting into the trillion-dollar company scale, everything you say gets you in trouble with someone, somehow, somewhere, so the easiest thing to do is to say nothing. The counterpoint is that on some point of scale, you hit a level where you need a fair bit of scrutiny; it’s deserved at this point because you are systemically important, and them’s the breaks.Rich: Yeah, and they have improved. A lot of the some of the larger companies have definitely improved. Microsoft learned a bunch of those lessons early on. [unintelligible 00:07:33] the product in Azure, maybe we’ll get there at some point. But you have to—I look at it both sides a little bit.On the vendor side, there are researchers who are unreasonable because now that I’m on the vendor side for the first time in my career, if something gets reported, like, it can really screw up plans and timing and you got to move developer resources. So, you have outside influences controlling you, so I get that piece of it. But the reality is if some researcher discovered it, some China, Russia, random criminals are going to discover it. So, you need to deal with those issues. So, it’s a bit of control. You lose control of your messaging and everything; if marketing gets their hands in this, then it becomes ugly.On the other hand, you have to, as a vendor, always realize that these are people frequently trying to make your products better. Some may be out just to extort you a little bit, whatever. That’s life. Get used to it. And in the end, it’s about putting the customers first, not necessarily putting your ego first and your marketing first.Corey: Changing gears slightly because believe it or not, neither you nor I have our primary day jobs focused on, you know, journalism or analyst work or anything like that these days, we focus on these—basically cloud, for lack of a better term—through slightly different lenses. I look at it through cost—which is of course architecture—and you look at it through the lens of security. And I will point out that only one of us gets called at three in the morning when things get horrible because of the bill is a strictly business-hours problem. Don’t think that’s an accident as far as what I decided to focus on. What do you do these days?Rich: You mean, what do I do in my day-to-day job?Corey: Well, it feels like a fair question to ask. Like, what do you do as far as day job, personal life et cetera. Who is Rich Mogull? You’ve been a name on the internet for a long time; I figured we’d add some color and context to it.Rich: Well, let’s see. I just got back from a flying lesson. I’m honing in on my getting ready for my first solo. My side gig is as a disaster response paramedic. I dressed up as a stormtrooper for the 501st Legion. I’ve got a few kids and then I have a job. I technically have two jobs. So—Corey: I’m envious of some of those things. I was looking into getting into flying but that path’s not open to me, given that I have ADHD. And there are ways around it in different ways. It’s like no, no, you don’t understand. With my given expression of it, I am exactly the kind of person that should not be flying a plane, let’s be very clear here. This is not a regulatory thing so much as it is a, “I’m choosing life.”Rich: Yeah. It’s a really fascinating thing because it’s this combination of a physical and a mental challenge. And I’m still very early in the process. But you know, I cracked 50, it had always been a life goal to do this, and I said, “You know what? I’m going to go do it.”So, first thing, I get my medical to make sure I can actually pass that because I’m over 50, and then from there, I can kind of jump into lessons. Protip though: don’t start taking lessons right as summer is kicking in in Phoenix, Arizona, with winds and heat that messes up your density altitude, and all sorts of fun things like that because it’s making it a little more challenging. But I’m glad I’m doing it.Corey: I have to imagine. That’s got to be an interesting skill set that probably doesn’t have a huge amount of overlap with the ins and outs of the cloud business. But maybe I’m wrong.Rich: Oh God, Corey. The correlations between information security—my specialty, and cloud security as a subset of that—aviation, and emergency medicine are incredible. These are three areas with very similar skill sets required in terms of thought processes. And in the case of both the paramedic and aviation, there’s physical skills and mental skills at the same time. But how you look at incidents, how you process things algorithmically, how you—your response times, checklists, the correlations.And I’ve been talking about two of those three things for years. I did a talk a couple years ago, during Covid, my Blackhat talk on the “Paramedics Guide to Surviving Cybersecurity,” where I talked a lot about these kinds of pieces. And now aviation is becoming another part of that. Amazing parallels between all three. Very similar mindsets are required.Corey: When you take a look at the overall sweep of the industry, you’ve been involved in cloud for a fairly long time. I have, too, but I start off as a cynic. I started originally when I got into the space, 2006, 2007, thinking virtualization was a flash in the pan because of the security potential impact of this. Then cloud was really starting to be a thing and pfff, that’s not likely to take off. I mean, who’s going to trust someone else to run all of their computing stuff?And at this point, I’ve learned to stop trying to predict the future because I generally get it 180 degrees wrong, which you know, I can own that. But I’m curious what you saw back when you got into this that made you decide, yeah, cloud has legs. What was that?Rich: I was giving a presentation with this guy, Chris Hoff, a good friend of mine. And Chris and I joined together are individual kind of research threads and were talking about, kind of, “Disruptive Innovation and the Future of Security.” I think that was the title. And we get that at RSA, we gave that at SOURCE Boston, start kind of doing a few sessions on this, and we talked about grid computing.And we were looking at, kind of, the economics of where things were going. And very early, we also realized that on the SaaS side, everybody was already using cloud; they just didn’t necessarily know it and they called them Application Service Providers. And then the concepts of cloud in the very early days were becoming compelling. It really hit me the first time I used it.And to give you perspective, I’d spent years, you know, seven years as a Gartner analyst getting hammered with vendors all the time. You can’t really test those technologies out because you can never test them in a way that an enterprise would use them. Even if I had a lab, the lab would be garbage; and we know this. I don’t trust things coming out of labs because that does not reflect operational realities at enterprise scale. Coming out of Gartner, they train me to be an enterprise guy. You talk about a large company being 200? Large companies start at 3000 to 5000 employees.Corey: Does that map to cloud services the way that AWS expresses? Because EKS, you’re going to manage that differently in an enterprise environment—or any other random AWS service; I’m just picking EKS as an example on this. But I can spin up a cluster and see what it’s like in 15 minutes, you know, assuming the cluster gets with the program. And it’s the same type of thing I would use in an enterprise, but I’m also not experiencing it in the enterprise-like way with the processes and the gating and the large team et cetera, et cetera, et cetera. Do you think it’s still a fair comparison at that point?Rich: Yeah, I think it absolutely is. And this is what really blew my mind. 11 or 12 years ago, when I got my first cloud account setup. I realized, oh, my God. And that was, there was no VPC, there was no IAM. It was ephemeral—and—no, we just had EBS was relatively new, and IAM was API only, it wasn’t in the console yet.Corey: And the network latency was, we’ll charitably call it non-deterministic.Rich: That was the advantage of not running anything at scale, wasn’t an issue at the time. But getting the hands-on and being able to build what I could build so quickly and easily and with so little friction, that was mind-blowing. And then for me, the first time I’ve used security groups I’m like, “Oh, my God, I have the granularity of a host firewall with the manageability of a network firewall?” And then years later, getting much deeper into how AWS networking and all the other pieces were—Corey: And doesn’t let it hit the host, which I always thought a firewall that lets—Rich: Yes.Corey: —traffic touch the host is like a seatbelt that lets your face touch the dashboard.Rich: Yeah. The first thing they do, they go in, they’re going to change the rules. But you can’t do that. It’s those layers of defense. And then I’m finding companies in the early days who wanted to put virtual appliances in front of everything. And still do. I had calls last week about that.But those are the things that really changed my mind because all of a sudden, this was what the key was, that I didn’t fully realize—and it’s kind of something that’s evolved into something I call the ‘Grand Unified Theory of Cloud Governance,’ these days—but what I realized was those barriers are gone. And there is no way to stop this as people want to build and test and deploy applications because the benefits are going to be too strong. So, grab onto the reins, hold on to the back of the horse, you’re going to get dragged away, and it’s your choice if your arm gets ripped off in the process or if you’re going to be able to ride that thing and at least steer it in the general direction that you need it to go in.Corey: One of the things that really struck me when I started playing around with cloud for more than ten minutes was everything you say is true, but I can also get started today to test out an idea. And most of them don’t work, but if something hits, suddenly I don’t have the data center constraints, whereas today, I guess you’d call it, I built my experiment MVP on top of a Raspberry Pi and now I have to wait six weeks for Dell to send me something that isn’t a piece of crap that I can actually take production traffic on. There’s no okay, and I’ll throw out the junky hardware and get the good stuff in once you start hitting a point of scale because you’re already building on that stuff without the corresponding massive investment of capital to get there.Rich: Yeah well, I mean, look, I lived this, I did a startup that was based on demos at a Blackhat—sorry, at a Blackhat. Blackhat. Did some demos on stage, people were like, “We want your code.” It was about cloud security automation. That led to doing your startup, the thing called DisruptOps, which got acquired, and that’s how I ended up at FireMon. So, that’s the day job route where I ended up.And what was amazing for that is, to add on to what you said, first of all, the friction was low; once we get the architecture right, scalability is not something we are hugely concerned with, especially because we’re CI/CD. Oh, no, we hit limits. Boom, let’s just stand up a new version and redirect people over there. Problem solved. And then the ability to, say, run multiple versions of our platform simultaneously? We’re doing that right now. We just had to release an entirely free version of it.To do that. It required back-end architectural changes for cost, not for scalability so much, but for a lot around cost and scheduling because our thing was event-driven, we’re able to run that and run our other platform fully in parallel, all shared data structures, shared messaging structures. I can’t even imagine how hard that would have not been to do in a traditional data center. So, we have a lot of freedom, still have those cost constraints because that’s [laugh] your thing, but the experimentation, the ability to integrate things, it’s just oh, my God, it’s just exciting.Corey: And let’s be clear, I, having spent a lot of time as a rat myself in these data centers, I don’t regret handing a lot of that responsibility off, just because, let’s not kid ourselves, they are better at replacing failed or failing hardware than I will ever be. That’s part of the benefit you get from the law of large numbers.Rich: Yeah. I don’t want to do all of that stuff, but we’re hovering around something that is kind of—all right, so former Gartner analyst means I have a massive ego, and because of that, I like to come up with my own terms for things, so roll with me here. And it’s something I’m calling the ‘Grand Unified Theory of Cloud Governance’ because you cannot possibly get more egotistical than referring to something as your solution to the biggest problem in all of physics. The idea is, is that cloud, as we have just been discussing, it drops friction and it decentralizes because you don’t have to go ask somebody for the network, you don’t have to ask somebody for the server. So, all of a sudden, you can build a full application stack without having to call somebody for help. We’ve just never had that in IT before.And all of our governance structures—and this includes your own costs, as well as security—are built around scarcity. Scarcity of resources, natural choke points that evolved from the data center. Not because it was bad. It wasn’t bad. We built these things because that’s what we needed for that environment at the data center.Now, we’ve got cloud and it’s this whole new alien technology and it decentralizes. That said, particularly for us on security, you can build your whole application stack, of course, we have completely unified the management interfaces in one place and then we stuck them on the internet, protected with nothing more than a username and password. And if you can put those three things together in your head, you can realize why these are such dramatic changes and so challenging for enterprises, why my kids get to go to Disney a fair bit because we’re in demand as security professionals.Corey: What does FireMon do exactly? That’s something that I’m not entirely up to speed on, just because please don’t take this the wrong way, but I was at RSA this year, and it feels like all the companies sort of blend together as you walk between the different booths. Like, “This is what you should be terrified of today.” And it always turns into a weird sales pitch. Not that that’s what you do, but it at some point just blinds me and overloads me as far as dealing with any of the cloud security space.Rich: Oh, I’ve been going to RSA for 20 years. One of our SEs, I was briefly at our booth—I’m usually in outside meetings—and he goes, “Do you see any fun and interesting?” I go—I just looked at him like I was depressed and I’m like, “I’ve been to RSA for 20 years. I will never see anything interesting here again. Those days are over.” There’s just too much noise and cacophony on that show floor.What do we do? So—Corey: It makes re:Invent’s Expo Hall look small.Rich: Yeah. I mean, it’s, it’s the show over at RSA. And it wasn’t always. I mean, it was—it’s always been big as long as I’ve been there, but yeah, it’s huge, everyone is there, and they’re all saying exactly the same thing. This year, I think the only reason it wasn’t all about AI is because they couldn’t get the printers to reprint the banners fast enough. Not that anybody has any products that would do anything there. So—you look like you want to say something there.Corey: No, no. I like the approach quite a bit. It’s the, everything was about AI this year. It was a hard pivot from trying to sell me a firewall, which it seems like everyone was doing in the previous year. It’s kind of wild. I keep saying that there’s about a dozen companies that exhibit at RSA. A guess, there are hundreds and hundreds of booths, but it all distills down to the same 12 things. They have different logos and different marketing stories, but it does seem like a lot of stuff is very much just like the booth next to it on both sides.Rich: Yeah. I mean, that’s—it’s just the nature. And part of—there’s a lot of reasons for this. We used to, when I was—so prior to doing the startup thing and then ending up at FireMon, I did Securosis, which was an analyst firm, and we used to do the Securosis guide to RSA every year where we would try and pick the big themes. And the reality is, there’s a reason for that.I wrote something once the vendors lied to you because you want them to. It’s the most dysfunctional relationship because as customers, you’re always asking, “Well, what are you doing for [unintelligible 00:22:16]? What are you doing for zero trust? What are you doing for AI?” When those same customers are still just working on fundamental patch management and firewall management. But it doesn’t stop them from asking the questions and the vendors have to have answers because that’s just the nature of that part of the world.Corey: I will ask you, over are past 12 years—I have my own thoughts on this, but I want to hear your take on it—what’s changed in the world of cloud security?Rich: Everything. I mean, I was one of the first to be doing this.Corey: Oh, is that all?Rich: Yeah. So, there’s more people. When I first started, very few people doing it, nobody knew much about it outside AWS, we all knew each other. Now, we’ve got a community that’s developed and there’s people that know what they’re doing. There’s still a shortage of skills, absolutely still a shortage of skills, but we’re getting a handle on that, you know? We’re getting a bit of a pipeline.And I’d say that’s still probably the biggest challenge faced. But what’s improved? Well, it’s a give-and-take. On one hand, we now have strategies, we have tools that are more helpful, unfortunately—I’ll tell you the biggest mistake I made and it ties to the FireMon stuff in my career, in a minute; relates directly to this question, but we’re kind of getting there on some of the tool pieces.On the other hand, that complexity is increasing faster. And that’s what’s made it hard. So, as much as we’re getting more skilled people, better at tooling, for example, we kind of know—and we didn’t have CloudTrail when I started. We didn’t have the fundamental things you need to actually implement security at the start of cloud. Most of those are there; they may not be working the way we wish they always worked, but we’ve got the pieces to assemble it, depending on which platform you’re on. That’s probably the biggest change. Now, we need to get into the maturity phase of cloud, and that’s going to be much more difficult and time-consuming to kind of get over that hump.Corey: It’s easy to wind up saying, “Oh, I saw the future so clearly back then,” but I have to ask, going back 12 years, the path the world would take was far from certain. Did you have doubts?Rich: Like, I had presented with Chris Hoff. We—we’re still friends—presented stuff together, and he got a job that was kind of clouding ancillary. And I remember calling him up once and going, “Chris, I don’t know what to do.” I was running my little analyst firm—little. We were doing very, very well—I could not get paid to do any work around cloud.People wanted me to write shitty papers on DLP and take customer inquiries on DLP because I had covered that at the Gartner days, and data encryption and those pieces. That was hard. And fortunately, a few things started trickling in. And then it was a flood. It completely changed our business and led to me, you know, eventually going down into the vendor path. But that was a tough day when I hit that point. So, absolutely I knew it was the future. I didn’t know if I was going to be able to make a living at it.Corey: It would seem that you did.Rich: Yeah. Worked out pretty well [laugh].Corey: You seem sprightly to me. Good work. You’re not on death’s door.Rich: No. You know, in fact, the analyst side of it exploded over the years because it turns out, there weren’t people who had this experience. So, I could write code to the APIs, but they’ll still talk with CEOs and boards of directors around these cloud security issues and frame them in ways that made sense to them. So, that was wonderful. We partnered up with the Cloud Security Alliance, I actually built a bunch of the CSA training, I wrote the current version of the CSA guidance, we’re writing the next version of that, did a lot of research with them. They’ve been a wonderful partner.So, all that went well. Then I got diverted down onto the vendor path. I had this research idea and then it came out, we ended up founding that as a startup and then it got, as I mentioned, acquired by FireMon, which is interesting because FireMon, you asked what we did, it’s firewall policy management is the core of the company. Yet the investors realize the company was not going in the right direction necessarily, to deal with the future of cloud. They went to their former CEO and said, “Hey, can you come back”—the founder of the company—“And take this over and start moving us in the right direction?”Well, he happened to be my co-founder at the startup. And so, we kind of came in and took over there. And so, now it’s a very interesting position because we have this one cloud-native thing we built for all these years. We made one mistake with that, which I’ll talk about which ties back to your predicting the future piece if you want to go into it, but then we have the network firewall piece now extending into hybrid, and we have an asset management moving into the attack surface management space as well. And both of those products have been around for, like, 15-plus years.Corey: No, I’m curious to your thoughts on it because it’s been one of those weird areas where there’s been so much change and so much evolution, but you also look at today’s “OWASP Top 10” list of vulnerabilities, and yeah, they updated a year or so ago, but it still looks basically like things that—from 2008—would have made sense to me when I’m looking at this. Well, insomuch as they do now. I didn’t know then, nor do I now what a cross-site scripting attack might be, but other than that, I find that there’s, “Oh, you misconfigured something and it winds up causing a problem.” Well, no kidding. Imagine that.Rich: Yeah. Look, the fundamentals don’t change, but it’s still really easy to screw up.Corey: Oh, having done so a lot, I believe you.Rich: There’s a couple of principles, and I’ll break it into two sides. One is, a lot of security sounds simple. There’s nothing simple at scale. Nothing simple scales. The moment you get up to even 200 employees, everything just becomes ridiculously harder. That’s the nature of reality. Simplicity doesn’t scale.The other part is even though it’s always the same, it’s still easy to think you’re going to be different this time and you’re not going to screw it up, and then you do. For example, so cloud, we were talking about the maturity. I assumed CSPM just wasn’t going to be a thing. For real. The Cloud Security Posture Management. Because why would the cloud providers not just make that problem go away and then all the vulnerability assessment vendors and everybody else? It seemed like it was an uninteresting problem.And yet, we were building a cloud security automation thing and we missed the boat because we had everything we needed to be one of the very first CSPM vendors on the market and we’re like, “No, no. That problem is going to go away. We’ll go there.” And it ties back to what you said, which is it’s the same stuff and we just outsmarted ourselves. We thought that people would go further faster. And they don’t and they aren’t.And that’s kind of where we are today. We are dramatically maturing. At the same time, the complexity is increasing dramatically. It’s just a huge challenge for skills and staffing to adjust governance programs. Like I think we’ve got another 10 to 20 years to go on this cloud security thing before we even get close. And then maybe we’ll get down to the being bored by the problems. But probably not because AI will ruin us.Corey: I’d like to imagine, on some level, that AI could be that good. I mean, don’t get me wrong. It has value and it is transformative for a bunch of things, but I also think a lot of the fear-mongering is more than a little overblown.Rich: No, I agree with you. I’m trying to keep a very close eye on it because—I can’t remember if you and I talked about this when we met face-to-face, or… it was somebody at that event—AI is just not just AI. There’s different. There’s the LLMs, there’s the different kinds of technologies that are involved. I mean, we use AI all over the place already.I mean my phone’s got it built in to take better pictures. It’s a matter of figuring out what the use cases and the, honestly, some of the regulatory structure around it in terms of copyright and everything else. I’m not worried about Clippy turning into Skynet, even though I might make jokes about that on Mastodon, maybe someday there will be some challenges, but no, it’s just going to be another tech that we’re going to figure out over time. It is disruptive, so we can’t ignore that part of it.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where’s the best place to find you that isn’t one of the Disney parks?Rich: That really is kind of the best place to find—no. So, these days, I do technically still have a Twitter presence at @rmogull. I’m not on there much, but I will get DMs if people send those over. I’m more on Mastodon. It’s at @rmogull defcon.social. I write over at FireMon these days, as well as occasionally still over Securosis, on those blogs. And I’m in the [Cloud Security Slack community 00:30:49] that is now under the banner for CloudSec. That’s probably the best place if you want to hit me up and get quick answers on anything.Corey: And I will, of course, include links to all of that in the show notes. Thank you so much for taking the time to speak with me today. I really appreciate it.Rich: Thanks, Corey. I was so happy to be here.Corey: Rich Mogull, SVP of Cloud Security at FireMon. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment talking about how at Dell these days, it does not take six weeks to ship a server. And then I will get back to you in six to eight weeks.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Kelly Shortridge, Senior Principal Engineer at Fastly, joins Corey on Screaming in the Cloud to discuss their recently released book, Security Chaos Engineering: Sustaining Resilience in Software and Systems. Kelly explains why a resilient strategy is far preferable to a bubble-wrapped approach to cybersecurity, and how developer teams can use evidence to mitigate security threats. Corey and Kelly discuss how the risks of working with complex systems is perfectly illustrated by Jurassic Park, and Kelly also highlights why it’s critical to address both system vulnerabilities and human vulnerabilities in your development environment rather than pointing fingers when something goes wrong.About KellyKelly Shortridge is a senior principal engineer at Fastly in the office of the CTO and lead author of "Security Chaos Engineering: Sustaining Resilience in Software and Systems" (O'Reilly Media). Shortridge is best known for their work on resilience in complex software systems, the application of behavioral economics to cybersecurity, and bringing security out of the dark ages. Shortridge has been a successful enterprise product leader as well as a startup founder (with an exit to CrowdStrike) and investment banker. Shortridge frequently advises Fortune 500s, investors, startups, and federal agencies and has spoken at major technology conferences internationally, including Black Hat USA, O'Reilly Velocity Conference, and SREcon. Shortridge's research has been featured in ACM, IEEE, and USENIX, spanning behavioral science in cybersecurity, deception strategies, and the ROI of software resilience. They also serve on the editorial board of ACM Queue.Links Referenced: Fastly: https://www.fastly.com/ Personal website: https://kellyshortridge.com Book website: https://securitychaoseng.com LinkedIn: https://www.linkedin.com/in/kellyshortridge/ Twitter: https://twitter.com/swagitda_ Bluesky: https://shortridge.bsky.social TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My guest today is Kelly Shortridge, who is a Senior Principal Engineer over at Fastly, as well as the lead author of the recently released Security Chaos Engineering: Sustaining Resilience in Software and Systems. Kelly, welcome to the show.Kelly: Thank you so much for having me.Corey: So, I want to start with the honest truth that in that title, I think I know what some of the words mean, but when you put them together in that particular order, I want to make sure we’re talking about the same thing. Can you explain that like I’m five, as far as what your book is about?Kelly: Yes. I’ll actually start with an analogy I make in the book, which is, imagine you were trying to rollerblade to some destination. Now, one thing you could do is wrap yourself in a bunch of bubble wrap and become the bubble person, and you can waddle down the street trying to make it to your destination on the rollerblades, but if there’s a gust of wind or a dog barks or something, you’re going to flop over, you’re not going to recover. However, if you instead do what everybody does, which is you know, kneepads and other things that keep you flexible and nimble, the gust you know, there’s a gust of wind, you can kind of be agile, navigate around it; if a dog barks, you just roller-skate around it; you can reach your destination. The former, the bubble person, that’s a lot of our cybersecurity today. It’s just keeping us very rigid, right? And then the alternative is resilience, which is the ability to recover from failure and adapt to evolving conditions.Corey: I feel like I am about to torture your analogy to death because back when I was in school in 2000, there was an annual tradition at the school I was attending before failing out, where a bunch of us would paint ourselves green every year and then bike around the campus naked. It was the green bike ride. So, one year I did this on rollerblades. So, if you wind up looking—there’s the bubble wrap, there’s the safety gear, and then there’s wearing absolutely nothing, which feels—Kelly: [laugh]. Yes.Corey: —kind of like the startup approach to InfoSec. It’s like, “It’ll be fine. What’s the worst that happens?” And you’re super nimble, super flexible, until suddenly, oops, now I really wish I’d done things differently.Kelly: Well, there’s a reason why I don’t say rollerblade naked, which other than it being rather visceral, what you described is what I’ve called YOLOSec before, which is not what you want to do. Because the problem when you think about it from a resilience perspective, again, is you want to be able to recover from failure and adapt. Sure, you can oftentimes move quickly, but you’re probably going to erode software quality over time, so to a certain point, there’s going to be some big incident, and suddenly, you aren’t fast anymore, you’re actually pretty slow. So, there’s this, kind of, happy medium where you have enough, I would like security by design—we can talk about that a bit if you want—where you have enough of the security by design baked in and you can think of it as guardrails that you’re able to withstand and recover from any failure. But yeah, going naked, that’s a recipe for not being able to rollerblade, like, ever again, potentially [laugh].Corey: I think, on some level, that the correct dialing in of security posture is going to come down to context, in almost every case. I’m building something in my spare time in the off hours does not need the same security posture—mostly—as we are a bank. It feels like there’s a very wide gulf between those two extremes. Unfortunately, I find that there’s a certain tone-deafness coming from a lot of the security industry around oh, everyone must have security as their number one thing, ever. I mean, with my clients who I fixed their AWS bills, I have to care about security contractually, but the secrets that I hold are boring: how much money certain companies pay another very large company.Yes, I’ll get sued into oblivion if that leaks, but nobody dies. Nobody is having their money stolen as a result. It’s slightly embarrassing in the tech press for a cycle and then it’s over and done with. That’s not the same thing as a brief stint I did running tech ops at Grindr ten years ago where, leak that database and people will die. There’s a strong difference between those threat models, and on some level, being able to act accordingly has been one of the more eye-opening approaches to increasing velocity in my experience. Does that align with the thesis of your book, since my copy has not yet arrived for this recording?Kelly: Yes. The book, I am not afraid to say it depends on the book, and you’re right, it depends on context. I actually talk about this resilience potion recipe that you can check out if you want, these ingredients so we can sustain resilience. A key one is defining your critical functions, just what is your system’s reason for existence, and that is what you want to make sure it can recover and still operate under adverse conditions, like you said.Another example I give all the time is most SaaS apps have some sort of reporting functionality. Guess what? That’s not mission-critical. You don’t need the utmost security on that, for the most part. But if it’s processing transactions, yeah, probably you want to invest more security there. So yes, I couldn’t agree more that it’s context-dependent and oh, my God, does the security industry ignore that so much of the time, and it’s been my gripe for, I feel like as long as I’ve been in the industry.Corey: I mean, there was a great talk that Netflix gave years ago where they mentioned in passing, that all developers have root in production. And that’s awesome and the person next to him was super excited and I looked at their badge, and holy hell, they worked at an actual bank. That seems like a bad plan. But talking to the Netflix speaker after the fact, Dave Hahn, something that I found that was extraordinarily insightful, was that, yeah, well we just isolate off the PCI environment so the rest and sensitive data lives in its own compartmentalized area. So, at that point, yeah, you’re not going to be able to break much in that scenario.It’s like, that would have been helpful context to put in talk. Which I’m sure he did, but my attention span had tripped out and I missed that. But that’s, on some level, constraining blast radius and not having compliance and regulatory issues extending to every corner of your environment really frees you up to do things appropriately. But there are some things where you do need to care about this stuff, regardless of how small the surface area is.Kelly: Agreed. And I introduced the concept of the effort investment portfolio in the book, which is basically, that is where does it matter to invest effort and where can you kind of like, maybe save some resources up. I think one thing you touched on, though, is, we’re really talking about isolation and I actually think people don’t think about isolation in as detailed or maybe as expansively as they could. Because we want both temporal and logical and spatial isolation. What you talked about is, yeah, there are some cases where you want to isolate data, you want to isolate certain subsystems, and that could be containers, it could also be AWS security groups.It could take a bunch of different forms, it could be something like RLBox in WebAssembly land. But I think that’s something that I really try to highlight in the book is, there’s actually a huge opportunity for security engineers starting from the design of a system to really think about how can we infuse different forms of isolation to sustain resilience.Corey: It’s interesting that you use the word investment. When fixing AWS bills for a living, I’ve learned over the last almost seven years now of doing this that cost and architecture and cloud are fundamentally the same thing. And resilience is something that comes with a very real cost, particularly when you start looking at what the architectural choices are. And one of the big reasons that I only ever work on a fixed-fee basis is because if I’m charging for a percentage of savings or something, it inspires me to say really uncomfortable things like, “Backups are for cowards.” And, “When was the last time you saw an entire AWS availability zone go down for so long that it mattered? You don’t need to worry about that.” And it does cut off an awful lot of cost issues, at the price of making the environment more fragile.That’s where one of the context thing starts to come in. I mean, in many cases, if AWS is having a bad day in a given region, well does your business need that workload to be functional? For my newsletter, I have a publication system that’s single-homed out of the Oregon region. If that whole thing goes down for multiple days, I’m writing that week’s issue by hand because I’m going to have something different to talk about anyway. For me, there is no value in making that investment. But for companies, there absolutely is, but there’s also seems to be a lack of awareness around, how much is a reasonable investment in that area when do you start making that investment? And most critically, when do you stop?Kelly: I think that’s a good point, and luckily, what’s on my side is the fact that there’s a lot of just profligate spending in cybersecurity and [laugh] that’s really what I’m focused on is, how can we spend those investments better? And I actually think there’s an opportunity in many cases to ditch a ton of cybersecurity tools and focus more on some of the stuff he talked about. I agree, by the way that I’ve seen some threat models where it’s like, well, AWS, all regions go down. I’m like, at that point, we have, like, a severe, bigger-than-whatever-you’re-thinking-about problem, right?Corey: Right. So, does your business continuity plan account for every one of your staff suddenly quitting on the spot because there’s a whole bunch of companies with very expensive consulting, like, problems that I’m going to go work for a week and then buy a house in cash. It’s one of those areas where, yeah, people are not going to care about your environment more than they are about their families and other things that are going on. Plan accordingly. People tend to get so carried away with these things with tabletop planning exercises. And then of course, they forget little things like I overwrote the database by dropping the wrong thing. Turns out that was production. [laugh]. Remembering for [a me 00:10:00] there.Kelly: Precisely. And a lot of the chaos experiments that I talk about in the book are a lot of those, like, let’s validate some of those basics, right? That’s actually some of the best investments you can make. Like, if you do have backups, I can totally see your argument about backups are for cowards, but if you do have them, like, maybe you conduct experiments to make sure that they’re available when you need them, and the same thing, even on the [unintelligible 00:10:21] side—Corey: No one cares about backups, but everyone really cares about restores, suddenly, right after—Kelly: Yeah.Corey: —they really should have cared about backups.Kelly: Exactly. So, I think it’s looking at those experiments where it’s like, okay, you have these basic assumptions in place that you assume to be invariance or assume that they’re going to bail you out if something goes wrong. Let’s just verify. That’s a great place to start because I can tell you—I know you’ve been to the RSA hall floor—how many cybersecurity teams are actually assessing the efficacy and actually experimenting to see if those tools really help them during incidents. It’s pretty few.Corey: Oh, vendors do not want to do those analyses. They don’t want you to do those analyses, either, and if you do, for God’s sakes, shut up about it. They’re trying to sell things here, mostly firewalls.Kelly: Yeah, cybersecurity vendors aren’t necessarily happy about my book and what I talk about because I have almost this ruthless focus on evidence and [unintelligible 00:11:08] cybersecurity vendors kind of thrive on a lack of evidence. So.Corey: There’s so much fear, uncertainty, and doubt in that space and I do feel for them. It’s a hard market to sell in without having to talk about here’s the thing that you’re defending against. In my case, it’s easy to sell the AWS bill is high because if I don’t have to explain why more or less setting money on fire as a bad thing, I don’t really know what to tell you. I’m going to go look for a slightly different customer profile. That’s not really how it works in security, I’m sure there are better go-to-market approaches, but they’re hard to find, at least, ones that work holistically.Kelly: There are. And one of my priorities with the book was to really enumerate how many opportunities there are to take software engineering practices that people already know, let’s say something like type systems even, and how those can actually help sustain resilience. Even things like integration testing or infrastructure as code, there are a lot of opportunities just to extend what we already do for systems reliability to sustain resilience against things that aren’t attacks and just make sure that, you know, we cover a few of those cases as well. A lot of it should be really natural to software engineering teams. Again, security vendors don’t like that because it turns out software engineering teams don’t particularly like security vendors.Corey: I hadn’t noticed that. I do wonder, though, for those who are unaware, chaos engineering started off as breaking things on purpose, which I feel like one person had a really good story and thought about it super quickly when they were about to get fired. Like, “No, no, it’s called Chaos Engineering.” Good for them. It’s now a well-regarded discipline. But I’ve always heard of it in the context of reliability of, “Oh, you think your site is going to work if the database falls over? Let’s push it over and see what happens.” How does that manifest in a security context?Kelly: So, I will clarify, I think that’s a slight misconception. It’s really about fixing things in production, and that’s the end goal. I think we should not break things just to break them, right? But I’ll give a simple example, which I know it’s based on what Aaron Rinehart conducted at UnitedHealth Group, which is, okay, let’s inject a misconfigured port as an experiment and see what happens, end-to-end. In their case, the firewall only detected the misconfigured port 60% of the time, so 60% of the time, it works every time.But it was actually the cloud, the very common, like, Cloud configuration management tool that caught the change and alerted responders. So, it’s that kind of thing where we’re still trying to verify those assumptions that we have about our systems and how they behave, again, end-to-end. In a lot of cases, again, with security tools, they are not behaving as we expect. But I still argue security is just a subset of software quality, so if we’re experimenting to verify, again, our assumptions and observe system behavior, we’re benefiting software quality, and security is just a subset of that. Think about C code, right? It’s not like there’s, like, a healthy memory corruption, so it’s bad for both the quality and security reason.Corey: One problem that I’ve had in the security space for a while is—let’s [unintelligible 00:14:05] on this to AWS for a second because that is the area in which I spend the most of my time, which probably explains a lot about my personality challenges. But the problem that I keep smacking into is if I go ahead and configure everything the way that I should according to best practices and the rest, I wind up with a firehose torrent of information in terms of CloudTrail logs, et cetera. And it’s expensive in its own right. But then to sort through it or to do a lot of things in security, there are basically two options. I can either buy a vendor’s product, which generally tends to start around $12,000 a year and goes up rapidly from there on my current $6,000 a year bill, so okay, twice as much as the infrastructure for security monitoring. Okay.Or alternately, find a bunch of different random scripts and tools on GitHub of wildly diverging quality and sort of hope for the best on that. It feels like there’s nothing in between. And the reason I care about this is not because I’m cheap but because when you have an individual learner who is either a student or a career switcher or someone just trying to experiment with this, you want them to begin as you want them to go on, and things that are no money for an enterprise are all the money to them. They’re going to learn to work with the tools that they can afford. That feels like it’s a big security swing and a miss. Do you agree or disagree? What’s the nuance I’m missing here?Kelly: No, I don’t think there’s nuance you’re missing. I think security observability, for one, isn’t a buzzword that particularly exists. I’ve been trying to make it a thing, but I’m solely one individual screaming into the void. But observability just hasn’t been a thing. We haven’t really focused on, okay, so what, like, we get data and what do we do with it?And I think, again, from a software engineering perspective, I think there’s a lot we can do. One, we can just avoid duplicating efforts. We can treat observability, again, of any sort of issue as similar, whether that’s an attack or a performance issue. I think this is another place where security, or any sort of chaos experiment, shines though because if you have an idea of here’s an adverse scenario we care about, you can actually see how does it manifest in the logs and you can start to figure out, like, what signals do we actually need to be looking for, what signals mattered to be able to narrow it down. Which again, it involves time and effort, but also, I can attest when you’re buying the security vendor tool and, in theory, absolving some of that time and effort, it’s maybe, maybe not, because it can be hard to understand what the outcomes are or what the outputs are from the tool and it can also be very difficult to tune it and to be able to explain some of the outputs. It’s kind of like trading upfront effort versus long-term overall overhead if that makes sense.Corey: It does. On that note, the title of your book includes the magic key phrase ‘sustaining resilience.’ I have found that security effort and investment tends to resemble a fire drill in—Kelly: [laugh].Corey: —an awful lot of places, where, “We care very much about security,” says the company, right after they very clearly failed to care about security, and I know this because I’m reading getting an email about a breach that they’ve just sent me. And then there’s a whole bunch of running around and hair-on-fire moments. But then there’s a new shiny that always comes up, a new strategic priority, and it falls to the wayside again. What do you see the drives that sustained effort and focus on resilience in a security context?Kelly: I think it’s really making sure you have a learning culture, which sounds very [unintelligible 00:17:30], but things again, like, experiments can help just because when you do simulate those adverse scenarios and you see how your system behaves, it’s almost like running an incident and you can use that as very fresh, kind of, like collective memory. And I even strongly recommend starting off with prior incidents in simulating those, just to see like, hey, did the improvements we make actually help? If they didn’t, that can be kind of another fire under the butt, so to speak, to continue investing. So, definitely in practice—and there’s some case studies in the book—it can be really helpful just to kind of like sustain that memory and sustain that learning and keep things feeling a bit fresh. It’s almost like prodding the nervous system a little, just so it doesn’t go back to that complacent and convenient feeling.Corey: It’s one of the hard problems because—I’m sure I’m going to get castigated for this by some of the listeners—but computers are easy, particularly compared to the people. There are deterministic ways to solve almost any computer problem, but people are always going to be a little bit different, and getting them to perform the same way today that they did yesterday is an exercise in frustration. Changing the culture, changing the approach and the attitude that people take toward a lot of these things feels, from my perspective, like, something of an impossible job. Cultural transformations are things that everyone talks about, but it’s rare to see them succeed.Kelly: Yes, and that’s actually something that I very strongly weaved throughout the book is that if your security solutions rely on human behavior, they’re going to fail. We want to either reduce hazards or eliminate hazards by design as much as possible. So, my view is very much again, like, can you make processes more repeatable? That’s going to help security. I definitely do not think that if anyone takes away from my book that they need to have, like, a thousand hours of training to change hearts and minds, then they have completely misunderstood most of the book.The idea is very much like, what are practices that we want for other outcomes anyway—again, reliability or faster time to market—and how can we harness those to also be improving resilience or security at the same time? It’s very much trying to think about those opportunities rather than, you know, trying to drill into people’s heads, like, “Thou shalt not,” or, “Thou shall.”Corey: Way back in 2018, you gave a keynote at some conference or another and you built the entire thing on the story of Jurassic Park, specifically Ian Malcolm as one of your favorite fictional heroes, and you tied it into security in a bunch of different ways. You hadn’t written this book then unless the authorship process is way longer than I think it is. So, I’m curious to get your take on what Jurassic Park can teach us about software security.Kelly: Yes, so I talk about Jurassic Park as a reference throughout the book, frequently. I’ve loved that book since I was a very young child. Jurassic Park is a great example of a complex system gone wrong because you can’t point to any one thing. Like there’s Dennis Nedry, you know, messing up the power system, but then there’s also the software was looking for a very specific count of dinosaurs and they didn’t anticipate there could be more in the count. Like, there are so many different factors that influenced it, you can’t actually blame just, like, human error or point fingers at one thing.That’s a beautiful example of how things go wrong in our software systems because like you said, there’s this human element and then there’s also how the humans interact and how the software components interact. But with Jurassic Park, too, I think the great thing is dinosaurs are going to do dinosaur things like eating people, and there are also equivalents in software, like C code. C code is going to do C code things, right? It’s not a memory safe language, so we shouldn’t be surprised when something goes wrong. We need to prepare accordingly.Corey: “How could this happen? Again?” Yeah.Kelly: Right. At a certain point, it’s like, there’s probably no way to sufficiently introduce isolation for dinosaurs unless you put them in a bunker where no one can see them, and it’s the same thing sometimes with things like C code. There’s just no amount of effort you can invest, and you’re just kind of investing for a really unclear and generally not fortuitous outcome. So, I like it as kind of this analogy to think about, okay, where do our effort investments make sense and where is it sometimes like, we really just do need to refactor because we’re dealing with dinosaurs here.Corey: When I was a kid, that was one of my favorite books, too. The problem is, I didn’t realize I was getting a glimpse of my future at a number of crappy startups that I worked at. Because you have John Hammond, who was the owner of the park talking constantly about how, “We spared no expense,” but then you look at what actually happened and he spared every frickin expense. You have one IT person who is so criminally underpaid that smuggling dinosaur embryos off the island becomes a viable strategy for this. He wound up, “Oh, we couldn’t find the right DNA, so we’re just going to, like, splice some other random stuff in there. It’ll be fine.”Then you have the massive overconfidence because it sounds very much like he had this almost Muskian desire to fire anyone who disagreed with him, and yeah, there was a certain lack of investment that could have been made, despite loud protestations to the contrary. I’d say that he is the root cause, he is the proximate reason for the entire failure of the park. But I’m willing to entertain disagreement on that point.Kelly: I think there are other individuals, like Dr. Wu, if you recall, like, deciding to do the frog DNA and not thinking that maybe something could go wrong. I think there was a lot of overconfidence, which you’re right, we do see a lot in software. So, I think that’s actually another very important lesson is that incentives matter and incentives are very hard to change, kind of like what you talked about earlier. It doesn’t mean that we shouldn’t include incentives in our threat model.So like, in the book I talked about, our threat models should include things like maybe yeah, people are underpaid or there is a ton of pressure to deliver things quickly or, you know, do things as cheaply as possible. That should be just as much of our threat models as all of the technical stuff too.Corey: I think that there’s a lot that was in that movie that was flat-out wrong. For example, one of the kids—I forget her name; it’s been a long time—was logging in and said, “Oh, this is Unix. I know Unix.” And having learned Unix as my first basically professional operating system, “No, you don’t. No one knows Unix. They get very confused at some point, the question is, just how far down what rabbit hole it is.”I feel so sorry for that kid. I hope she wound up seeking therapy when she was older to realize that, no, you don’t actually know Unix. It’s not that you’re bad at computers, it’s that Unix is user-hostile, actively so. Like, the raptors, like, that’s the better metaphor when everything winds up shaking out.Kelly: Yeah. I don’t disagree with that. The movie definitely takes many liberties. I think what’s interesting, though, is that Michael Creighton, specifically, when he talks about writing the book—I don’t know how many people know this—dinosaurs were just a mechanism. He knew people would want to read it in airport.What he cared about was communicating really the danger of complex systems and how if you don’t respect them and respect that interactivity and that it can baffle and surprise us, like, things will go wrong. So, I actually find it kind of beautiful in a way that the dinosaurs were almost like an afterthought. What he really cared about was exactly what we deal with all the time in software, is when things go wrong with complexity.Corey: Like one of his other books, Airframe, talked about an air disaster. There’s a bunch of contributing factors in the rest, and for some reason, that did not receive the wild acclaim that Jurassic Park did to become a cultural phenomenon that we’re still talking about, what, 30 years later.Kelly: Right. Dinosaurs are very compelling.Corey: They really are. I have to ask though—this is the joy of having a kid who is almost six—what is your favorite dinosaur? Not a question most people get asked very often, but I am going to trot that one out.Kelly: No. Oh, that is such a good question. Maybe a Deinonychus.Corey: Oh, because they get so angry they spit and kill people? That’s amazing.Kelly: Yeah. And I like that, kind of like, nimble, smarter one, and also the fact that most of the smaller ones allegedly had feathers, which I just love this idea of, like, feather-ful murder machines. I have the classic, like, nerd kid syndrome, though, where I read all these dinosaur names as a kid and I’ve never pronounced them out loud. So, I’m sure there are others—Corey: Yep.Kelly: —that I would just word salad. But honestly, it’s hard to go wrong with choosing a favorite dinosaur.Corey: Oh, yeah. I’m sure some paleontologist is sitting out there in the field on the dig somewhere listening to this podcast, just getting very angry at our pronunciation and things. But for God’s sake, I call the database Postgres-squeal. Get in line. There’s a lot of that out there where looking at a complex system failures and different contributing factors and the rest makes stuff—that’s what makes things interesting.I think that there’s this the idea of a root cause is almost always incorrect. It’s not, “Okay, who tripped over the buried landmine,” is not the interesting question. It’s, “Who buried the thing?” What were all the things that wound up contributing to this? And you can’t even frame it that way in the blaming context, just because you start doing that and people clam up, and good luck figuring out what really happened.Kelly: Exactly. That’s so much of what the cybersecurity industry is focused on is how do we assign blame? And it’s, you know, the marketing person clicked on a link. And it’s like, they do that thousands of times, like a month, and the one time, suddenly, they were stupid for doing it? That doesn’t sound right.So, I’m a big fan of, yes, vanquishing root cause, thinking about contributing factors, and in particular, in any sort of incident review, you have to think about, was there a designer process problem? You can’t just think about the human behavior; you have to think about where are the opportunities for us to design things better, to make this secure way more of the default way.Corey: When you talk about resilience and reliability and big, notable outages, most forward-thinking companies are going to go and do a variety of incident reviews and disclosures around everything that happened to it, depending upon levels of trust and whether your NDA’ed or not, and how much gets public is going to vary from place to place. But from a security perspective, that feels like the sort of thing that companies will clam up about and never say a word.Kelly: Yes.Corey: Because I can wind up pouring a couple of drinks into people and get the real story of outages, or the AWS bill, but security stuff, they start to wonder if I’m a state actor, on some level. When you were building all of this, how did you wind up getting people to talk candidly and forthrightly about issues that if it became tied to them that they were talking to this in public would almost certainly have negative career impact for them?Kelly: Yes, so that’s almost like a trade secret, I feel like. A lot of it is yes, over the years talking with people over, generally at a conference where you know, things are tipsy. I never want to betray confidentiality, to be clear, but certainly pattern-matching across people’s stories.Corey: Yeah, we’re both in positions where if even the hint of they can’t be trusted enters the ecosystem, I think both of our careers explode and never recover. Like it’s—Kelly: Exactly.Corey: —yeah. Oh, yeah. They play fast and loose with secrets is never the reputation you want as a professional.Kelly: No. No, definitely not. So, it’s much more pattern matching and trying to generalize. But again, a lot of what can go wrong is not that different when you think about a developer being really tired and making a bunch of mistakes versus an attacker. A lot of times they’re very much the same, so luckily there’s commonality there.I do wish the security industry was more forthright and less clandestine because frankly, all of the public postmortems that are out there about performance issues are just such, such a boon for everyone else to improve what they’re doing. So, that’s a change I wish would happen.Corey: So, I have to ask, given that you talk about security, chaos engineering, and resilience-and of course, software and systems—all in the title of the O’Reilly book, who is the target audience for this? Is it folks who have the word security featured three times in their job title? Is it folks who are new to the space? What is your target audience start and stop?Kelly: Yes, so I have kept it pretty broad and it’s anyone who works with software, but I’ll talk about the software engineering audience because that is, honestly, probably out of anyone who I would love to read the book the most because I firmly believe that there’s so much that software engineering teams can do to sustain resilience and security and they don’t have to be security experts. So, I’ve tried to demystify security, make it much less arcane, even down to, like, how attackers, you know, they have their own development lifecycle. I try to demystify that, too. So, it’s very much for any team, especially, like, platform engineering teams, SREs, to think about, hey, what are some of the things maybe I’m already doing that I can extend to cover, you know, the security cases as well? So, I would love for every software engineer to check it out to see, like, hey, what are the opportunities for me to just do things slightly differently and have these great security outcomes?Corey: I really want to thank you for taking the time to talk with me about how you view these things. If people want to learn more, where’s the best place for them to find you?Kelly: Yes, I have all of the social media which is increasingly fragmented, [laugh] I feel like, but I also have my personal site, kellyshortridge.com. The official book site is securitychaoseng.com as well. But otherwise, find me on LinkedIn, Twitter, [Mastodon 00:30:22], Bluesky. I’m probably blanking on the others. There’s probably already a new one while we’ve spoken.Corey: Blue-ski is how I insist on pronouncing it as well, while we’re talking about—Kelly: Blue-ski?Corey: Funhouse pronunciation on things.Kelly: I like it.Corey: Excellent. And we will, of course, put links to all of those things in the [show notes 00:30:37]. Thank you so much for being so generous with your time. I really appreciate it.Kelly: Thank you for having me and being a fellow dinosaur nerd.Corey: [laugh]. Kelly Shortridge, Senior Principal Engineer at Fastly. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment about how our choice of dinosaurs is incorrect, then put the computer away and struggle to figure out how to open a door.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Brooke Sargent, Software Engineer at Honeycomb, joins Corey on Screaming in the Cloud to discuss how she fell into the world of observability by adopting Honeycomb. Brooke explains how observability was new to her in her former role, but she quickly found it to enable faster learning and even a form of self care for herself as a developer. Corey and Brooke discuss the differences of working at a large company where observability is a new idea, versus an observability company like Honeycomb. Brooke also reveals the importance of helping people reach a personal understanding of what observability can do for them when trying to introduce it to a company for the first time. About BrookeBrooke Sargent is a Software Engineer at Honeycomb, working on APIs and integrations in the developer ecosystem. She previously worked on IoT devices at Procter and Gamble in both engineering and engineering management roles, which is where she discovered an interest in observability and the impact it can have on engineering teams.Links Referenced: Honeycomb: https://www.honeycomb.io/ Twitter: https://twitter.com/codegirlbrooke TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. This promoted guest episode—which is another way of saying sponsored episode—is brought to us by our friends at Honeycomb. And today’s guest is new to me. Brooke Sargent is a software engineer at Honeycomb. Welcome to the show, Brooke.Brooke: Hey, Corey, thanks so much for having me.Corey: So, you were part of I guess I would call it the new wave of Honeycomb employees, which is no slight to you, but I remember when Honeycomb was just getting launched right around the same time that I was starting my own company and I still think of it as basically a six-person company versus, you know, a couple of new people floating around. Yeah, turns out, last I checked, you were, what, north of 100 employees and doing an awful lot of really interesting stuff.Brooke: Yeah, we regularly have, I think, upwards of 100 in our all-hands meeting, so definitely growing in size. I started about a year ago and at that point, we had multiple new people joining pretty much every week. So yeah, a lot of new people.Corey: What was it that drove you to Honeycomb? Before this, you spent a bit of time over Procter and Gamble. You were an engineering manager and now you’re going—you went from IC to management and now you’re IC again. There’s a school of thought that I vehemently disagree with, that that’s a demotion. I think they are orthogonal skill sets to my mind, but I’m curious to hear your journey through your story.Brooke: Yeah, absolutely. So yeah, I worked at Procter and Gamble, which is a big Cincinnati company. That’s where I live and I was there for around four years. And I worked in both engineering and engineering management roles there. I enjoy both types of roles.What really drove me to Honeycomb is, my time at Procter and Gamble, I spent probably about a year-and-a-half, really diving into observability and setting up an observability practice on the team that I was on, which was working on connected devices, connected toothbrushes, that sort of thing. So, I set up an observability practice there and I just saw so much benefit to the engineering team culture and the way that junior and apprentice engineers on the team were able to learn from it, that it really caught my attention. And Honeycomb is what we were using and I kind of just wanted to spend all of my time working on observability-type of stuff.Corey: When you say software engineer, my mind immediately shortcuts to a somewhat outdated definition of what that term means. It usually means application developer, to my mind, whereas I come from the world of operations, historically sysadmins, which it still is except now, with better titles, you get more money. But that’s functionally what SRE and DevOps and all the rest of the terms still currently are, which is, if it plugs into the wall, congratulations. It’s your problem now to go ahead and fix that thing immediately. Were you on the application development side of the fence? Were you focusing on the SRE side of the world or something else entirely?Brooke: Yeah, so I was writing Go code in that role at P&G, but also doing what I call it, like, AWS pipe-connecting, so a little bit of both writing application code but also definitely thinking about the architecture aspects and lining those up appropriately using a lot of AWS serverless and managed services. At Honeycomb, I definitely find myself—I’m on the APIs and partnerships team—I find myself definitely writing a lot more code and focusing a lot more on code because we have a separate platform team that is focusing on the AWS aspects.Corey: One thing that I find interesting is that it is odd, in many cases, to see, first, a strong focus on observability coming from the software engineer side of the world. And again, this might be a legacy of where I was spending a lot of my career, but it always felt like getting the application developers to instrument whatever it was that they were building felt in many ways like it was pulling teeth. And in many further cases, it seemed that you didn’t really understand the value of having that visibility or that perspective into what’s going on in your environment, until immediately after. You really wished you had that perspective into what was going on in your environment, but didn’t. It’s similar to, no one is as zealous about backups as someone who’s just suffered a data loss. Same operating theory. What was it that you came from the software engineering side to give a toss about the idea of observability?Brooke: Yeah, so working on the IoT—I was working on, like, the cloud side of things, so in Internet of Things, you’re keeping a mobile application, firmware, and cloud synced up. So, I was focused on the cloud aspect of that triangle. And we got pretty close to launching this greenfield IoT cloud that we were working on for P&G, like, we were probably a few months from the initial go-live date, as they like to call it, and we didn’t have any observability. We were just kind of sending things to CloudWatch logs. And it was pretty painful to figure out when something went wrong, from, like, you know, hearing from a peer on a mobile app team or the firmware team that they sent us some data and they’re not seeing it reflected in the cloud that is, like, syncing it up.Figuring out where that went wrong, just using CloudWatch logs was pretty difficult and syncing up the requests that they were talking about to the specific CloudWatch log that had the information that we needed, if we had even logged the right thing. And I was getting a little worried about the fact that people were going to be going into stores and buying these toothbrushes and we might not have visibility into what could be going wrong or even being able to be proactive about what is going wrong. So, then I started researching observability. I had seen people talking about it as a best practice thing that you should think about when you’re building a system, but I just hadn’t had the experience with it yet. So, I experimented with Honeycomb a bit and ended up really liking their approach to observability. It fit my mental model and made a lot of sense. And so, I went full-steam ahead with implementing it.Corey: I feel what you just said is very key: the idea of finding an observability solution that keys into the mental model that someone’s operating with. I found that a lot of observability talk sailed right past me because it did not align with that, until someone said, “Oh yeah, and then here’s events.” “Well, what do you mean by event?” It distills down to logs. And oh, if you start viewing everything as a log event, then yeah, that suddenly makes a lot of sense, and that made it click for me in a way that, honestly, is a little embarrassing that it didn’t before then.But I come from a world before containers and immutable infrastructure and certainly before the black boxes that are managed serverless products, where I’m used to, oh, something’s not working on this Linux box. Well, I have root, so let’s go ahead and fix that and see what’s going on. A lot of those tools don’t work, either at scale or in ephemeral environments or in scenarios where you just don’t have the access to do the environment. So, there’s this idea that if you’re trying to diagnose something that happened and the container that it happened on stopped existing 20 minutes ago, your telemetry game has got to be on point or you’re just guessing at that point. That is something that I think I did myself a bit of a disservice by getting out of hands-on keyboard operations roles before that type of architecture really became widespread.Brooke: Yeah, that makes a lot of sense. On the team that I was on, we were using a lot of AWS Lambda and similarly, tracking things down could be a little bit challenging. And emitting telemetry data also has some quirks [laugh] with Lambda.Corey: There certainly is. It’s also one of those areas that, on some level, being stubborn to adopt it works to your benefit. Because when Lambda first came out, it was a platform that was almost entirely identified by its constraints. And Amazon didn’t do a terrific job, at least in the way that I tend to learn, of articulating what those constraints are. So, you learn by experimenting and smacking face first into a lot of those things.What the hell do you mean you can’t write to the file? Oh, it’s a read-only file system. [except slash tap 00:08:39]. What do you mean, it’s only half a gigabyte? Oh, that’s the constraint there. Well, what do you mean, it automatically stops after—I think back in that point it was five or ten minutes; it’s 15 these days. But—Brooke: Right.Corey: —I guess it’s their own creative approach to solving the halting problem from computer science classes, where after 15 minutes, your code will stop executing, whether you want it to or not. They’re effectively evolving these things as we go and once you break your understanding in a few key ways, at least from where I was coming from, it made a lot more sense. But ugh, that was a rough couple of weeks for me.Brooke: Yeah [laugh]. Agreed.Corey: So, a topic that you have found personally inspiring is that observability empowers junior engineers in a bunch of ways. And I do want to get into that, but beforehand, I am curious as to the modern-day path for SREs because it doesn’t feel to me like there is a good answer for, “What does a junior SRE look like?” Because the answer is, “Oh, they don’t.” It goes back to the old sysadmin school of thought, which is that, oh, you basically learn by having experience. I’ve lost count a number of startups I’ve encountered where you have a bunch of early-20-something engineers but the SRE folks are all generally a decade into what they’re what they’ve been doing because the number-one thing you want to hear from someone in that role is, “Oh, the last time I saw it, here’s what it was.” What is the observability story these days for junior engineers?Brooke: So, with SRE I agr—like, that’s a conversation that I’ve had a lot of times on different teams that I’ve been on, is just can a junior SRE exist? And I think that they can.Corey: I mean, they have to because otherwise, it’s well, where does it SRE come from? Oh, they spring—Brooke: [laugh].Corey: —fully formed from the forehead of some God out of mythology. It doesn’t usually work that way.Brooke: Right. But you definitely need a team that is ready to support a junior SRE. You need a robust team that is interested in teaching and mentoring. And not all teams are like that, so making sure that you have a team culture that is receptive to taking on a junior SRE is step number one. And then I think that the act of having an observability practice on a team is very empowering to somebody who is new to the industry.Myself, I came from a self-taught background, learning to code. I actually have a music degree; I didn’t go to school for computer science. And when I finally found my way to observability, it made so many, kind of, light bulbs go off of just giving me more visuals to go from, “I think this is happening,” to, “I know this is happening.” And then when I started mentoring juniors and apprentices and putting that same observability data in front of them, I noticed them learning so much faster.Corey: I am curious in that you went from implementing a lot of these things and being in a management role of mentoring folks on observability concepts to working for an observability vendor, which is… I guess I would call Honeycomb the observability vendor. They were the first to really reframe a lot of how we considered what used to be called monitoring and now it’s called observability, or as I think of it, hipster monitoring.Brooke: [laugh].Corey: But I am curious as to when you look at this, my business partner wrote a book for O’Reilly, Practical Monitoring, and he loved it so much that by the end of that book, he got out of the observability monitoring space entirely and came to work on AWS bills with me. Did you find that going to Honeycomb has changed your perspective on observability drastically?Brooke: I had definitely consumed a lot of Honeycomb’s blog posts, like, that’s one of the things that I had loved about the company is they put out a lot of interesting stuff, not just about observability but about operating healthy teams, and like you mentioned, like, a pendulum between engineering management and being an IC and just interesting concepts within our industry overall as, like, software engineers and SREs. So, I knew a lot of the thought leadership that the company put out, and that was very helpful. It was a big change going from an enterprise like Procter and Gamble to a startup observability company like Honeycomb, just—and also, going from a company that very much believes in in-person work to remote-first work at Honeycomb, now. So, there were a lot of, like, cultural changes, but I think I kind of knew what I was getting myself into as far as the perspective that the company takes on observability.Corey: That is always the big, somewhat awkward question because of the answer goes a certain way, it becomes a real embarrassment, but I’m confident enough, having worked with Honeycomb as a recurring sponsor and having helped out on the AWS bill side of the world since you were a reference client on both sides of that business, I want to be very clear that I don’t think I’m throwing you under a bus on this one. But do you find that the reality, now that you’ve been there for a year, has matched the external advertising and the ethos of the story they tell about Honeycomb from the outside?Brooke: I definitely think it matches up. One thing that is just different about working inside of a company like Honeycomb versus working at a company that doesn’t have any observability at all yet, is that there are a lot of abstraction layers in our codebase and things like that. So, me being a software engineer and writing code Honeycomb compared to P&G, I don’t have to think about observability as much because everybody in the company is thinking about observability and had thought about it before I joined and had put in a lot of thought to how to make sure that we consistently have telemetry data that we need to solve problems versus I was thinking about this stuff on the daily at P&G.Corey: Something I’ve heard from former employees of a bunch of different observability companies has a recurring theme to it, and that it’s hard to leave. Because when you’re at an observability company, everything is built with an eye toward observability. And there’s always the dogfooding story of, we instrument absolutely everything we have with everything that we sell the customers. Now, in practice, you leave and go to a different company, that is almost never going to be true, if for no other reason than based on simple economics. Turning on every facet of every observability tool that a given company sells becomes extraordinarily expensive and is an investment decision, so companies say yes to some, no to others. Do you think you’re going to have that problem if and when you decide it’s time to move on to your next role, assuming of course, that it’s not at a competing observability company?Brooke: I’m sure there will be some challenges if I decide to move on from working for observability platforms in the future. The one that I think would be the most challenging is joining a team where people just don’t understand the value of observability and don’t want to invest, like, the time and effort into actually instrumenting their code, and don’t see why they need to do it, versus just, like, they haven’t gotten there yet or they haven’t had enough people hired to do it just yet. But if people are actively, like, kind of against the idea of instrumenting your code, I think that would be really challenging to kind of shift to especially after, over the last two-and-a-half years or so, being so used to having this, like, extra sense when I’m debugging problems and dealing with outages.Corey: I will say, it was a little surreal the first time I wound up taking a look at Honeycomb’s environment—because I do believe that cost and architecture are fundamentally the same thing when it comes to cloud—and you had clear lines of visibility into what was going on in your AWS bill by way of Honeycomb as a product. And that’s awesome. I haven’t seen anyone else do that yet and I don’t know that it would necessarily work as well because, as you said, there, everyone’s thinking about it through this same shared vision, whereas in a number of other companies, it flat out does not work that way. There are certain unknowns and questions. And from the outside, and when you first start down this path, it feels like a ridiculous thing to do, until you get to a point of seeing the payoff, and yeah, this makes an awful lot of sense.I don’t know that it would, for example, work as a generic solution for us to roll out to our various clients and say, oh, let’s instrument your environment with this and see what’s going on because first, we don’t have that level of ability to make change in customer environments. We are read-only for some very good reasons. And further, it also seems like it’s a, “Step one: change your entire philosophy around these sorts of things so we can help you spend less on AWS,” seems like a bit of a tall order.Brooke: Yeah, agreed. And yeah, on previous teams that I’ve been on, I definitely—and I think it’s fair, absolutely fair, that there were things where, especially using AWS serverless services, I was trying to get as much insight as possible into adding some of these services to our traces, like, AppSync was one example where I could not for the life of me figure out how to get AppSync API requests onto my Honeycomb trace. And I spent a lot of time trying to figure it out. And I had team members that would just be, like, you know, “Let’s timebox this; let’s not, like, sink all of our time into it.” And so, I think as observability evolves, hopefully, carving out those patterns continues to get easier so that engineers don’t have to spend all of their time, kind of, carving out those patterns.Corey: It feels like that’s the hard part, is the shift in perspective. Instrumenting a given tool into an environment is not the heavy lift compared to appreciating the value of it. Do you find that that was an easy thing for you to overcome, back when you were at Procter and Gamble, as far as people already have bought in, on some level, to observability from having seen it in some kind of scenarios where it absolutely save folks’ bacon? Or was it the problem of, first you have to educate people about the painful problem that they have before they realize it is in fact, A, painful, and B, a problem, and then C, that you have something to sell them that will solve that? Because that pattern is a very hard sales motion to execute in most spaces. But you were doing it at it, from the customer side first.Brooke: Yeah. Yeah, doing it from the customer side, I was able to get buy-in on the team that I was on, and I should also say, like, the team that I was on was considered an innovation team. We were in a separate building from, like, the corporate building and things like that, which I’m sure played into some of those cultural aspects and dynamics. But trying to educate people outside of our team and trying to build an observability practice within this big enterprise company was definitely very challenging, and it was a lot of spending time sharing information and talking to people about their stack and what languages and tools that they’re using and how this could help them. I think until people have had that, kind of, magical moment of using observability data to solve a problem for themselves, it’s very hard, it can be very hard to really make them understand the value.Corey: That was is always my approach because it feels like observability is a significant and sizable investment in infrastructure alone, let alone mental overhead, the teams to manage these things, et cetera, et cetera. And until you have a challenge that observability can solve, it feels like it is pure cost, similar to backups, where it’s just a whole bunch of expense for no benefit until suddenly, one day, you’re very glad you had it. Now, the world is littered with stories that are very clear about what happens when you don’t have backups. Most people have a personal story around that, but it feels like it’s less straightforward to point at a visceral story where not having observability really hobbled someone or something.It feels like—because in the benefit of perfect hindsight, oh yeah, like a disk filled up and we didn’t know about that. Like, “Ah, if we just had the right check, we would have caught that early on.” Yeah, coulda, woulda shoulda, but it was a cascading failure that wasn’t picked up until seven levels downstream. Do you think that that's the situation these days or am I misunderstanding how people are starting to conceive about this stuff?Brooke: Yeah. I mean, I definitely have a couple of stories of even once I was on the journey to observability adoption—which I call it a journey because you don’t just—kind of—snap your fingers and have observability—I started with one service, instrumenting that and just, like, gradually, over sprint’s would instrument more services and pull more team members in to do that as well. But when we were in that process of instrumenting services, there was one service which was our auth service—which maybe should have been the first one that we instrumented—that a code change was made and it was erroring every time somebody tried to sign up in the app. And if we had observability instrumentation in place for that service, it wouldn’t have taken us, like, the four or five hours to find the problem of the one line of code that we had changed; we would have been able to see more clearly what error was happening and what line of code it was happening on and probably fix it within an hour.And we had a similar issue with a Redshift database that we were running more on the metrics side of things. We were using it to send analytics data to other people in the company and that Redshift database just got maxed out at a certain point. The CPU utilization was at, like, 98% and people in the company were very upset and [laugh] having a very bad time querying their analytics data.Corey: It’s a terrific sales pitch for Snowflake, to be very direct, because you hear that story kind of a lot.Brooke: Yeah, it was not a fun time. But at that point, we started sending Redshift metrics data over to Honeycomb as well, so that we could keep a better pulse on what exactly was happening with that database.Corey: So, here’s sort of the acid test: people tend to build software when they’re starting out greenfield, in ways that emphasize their perspective on the world. For example, when I’m building something new, doesn’t matter if it’s tiny or for just a one-off shitposting approach, and it touches anything involving AWS, first thing I do out of the gate is I wind up setting tags so that I can do cost allocation work on it; someday, I’m going to wonder how much this thing cost. That is, I guess my own level of brokenness.Brooke: [laugh].Corey: When you start building something at work from scratch, I guess this is part ‘you,’ part ‘all of Honeycomb,’ do you begin from that greenfield approach of Hello World of instrumenting it for observability, even if it’s not explicitly an observability-focused workload? Or is it something that you wind up retrofitting with observability insights later, once it hits a certain point of viability?Brooke: Yeah. So, if I’m at the stage of just kind of trying things out locally on my laptop, kind of outside of, like, the central repo for the company, I might not do observability data because I’m just kind of learning and trying things out on my laptop. Once I pull it into our central repo, there is some observability data that I am going to get, just in the way that we kind of have our services set up. And as I’m going through writing code to do this whatever new feature I’m trying to do, I’m thinking about what things, when this breaks—not if it breaks; when it breaks [laugh]—am I going to want to know about in the future. And I’ll add those things, kind of, on the fly just to make things easier on myself, and that’s just kind of how my brain works at this point of thinking about my future self, which is, kind of like, the same definition of self-care. So, I think of observability as self-care for developers.But later on, when we’re closer to actually launching a thing, I might take another pass at just, like, okay, let’s once again take a look at the error paths and how this thing can break and make sure that we have enough information at those points of error to know what is happening within a trace view of this request.Corey: My two programming languages that I rely on the most are enthusiasm and brute force, and I understand this is not a traditional software engineering approach. But I’ve always found that having to get observability in involved a retrofit, on some level. And it always was frustrating to me just because it felt like it was so much effort in various ways that I’ve just always kicked myself: I should have done this early on. But I’ve been on the other side of that, and it’s like, should I instrument this with good observability? No, that sounds like work. I want to see if this thing actually works at all, or not first.And I don’t know what side of the fence is the correct one to be on, but I always find that I’m on the wrong one. Like, I don’t know if it’s, like, one of those, there’s two approaches and neither one works. I do see in client environments where observability is always, always, always something that has to be retrofit into what it is that they’re doing. Does it stay that way once companies get past a certain point? Does observability of adoption among teams just become something that is ingrained into them or do people have to consistently relearn that same lesson, in your experience?Brooke: I think it depends, kind of, on the size of your company. If you are a small company with a, you know, smaller engineering organization where it’s not, I won’t say easy, but easier to get kind of full team buy-in on points of view and decisions and things like that, it becomes more built-in. If you’re in a really big company like the one that I came from, I think it is continuously, like, educating people and trying to show the value of, like, why we are doing this—coming back to that why—and like, the magical moment of, like, stories of problems that have been solved because of the instrumentation that was in place. So, I guess, like most things, it’s an, ‘it depends.’ But the larger that your company becomes, I think the harder it gets to keep everybody on the same page.Corey: I am curious, in that I tend to see the world through AWS bills, which is a sad, pathetic way to live that I don’t recommend to basically anyone, but I do see the industry, or at least my client base, forming a bit of a bimodal distribution. On one side, you have companies like Honeycomb, including, you know, Honeycomb, where the majority of your AWS spend is driven by the application that is Honeycomb, you know, the SaaS thing you sell to people to solve their problems. The other side of the world are companies that look a lot more like Procter and Gamble, presumably, where—because I think of oh, what does Procter and Gamble do? And the answer is, a lot. They’re basically the definition of conglomerate in some ways.So, you look at that, a bill at a big company like that and it might be hundreds of millions of dollars, but the largest individual workload is going to be a couple million at best. So, it feels very much like it’s this incredibly diffuse selection of applications. And in those environments, you have to start thinking a lot more about centralization things you can do, for example, for savings plan purchases and whatnot, whereas at Honeycomb-like companies, you can start looking at, oh, well, you have this single application that’s the lion’s share of everything. We can go very deep into architecture and start looking at micro-optimizations here that will have a larger impact. Having been an engineer at both types of companies, do you find that there’s a different internal philosophy, or is it that when you’re working in a larger company on a specific project, that specific project becomes your entire professional universe?Brooke: Yeah, definitely at P&G, for the most part, IoT was kind of the center of my universe. But one philosophy that I noticed as being different—and I think this is from being an enterprise in a startup—is just the way that thinking about cost and architecture choices, kind of, happened. So, at P&G, like I said, we were using a lot of Lambda, and pretty much any chance we got, we used a serverless or managed offering from AWS. And I think a big part of that reasoning was because, like I said earlier, P&G is very interested in in-person work. So, everybody that we hired her to be located in Cincinnati.And it became hard to hire for people who had Go and Terraform experience because a lot of people in the Midwest are much more comfortable in .NET and Java; there’s just a lot more jobs using those technologies. So, we had a lot of trouble hiring and would choose—because P&G had a lot of money to spend—to give AWS that money because we had trouble finding engineers to hire, whereas Honeycomb really does not seem to have trouble hiring engineers. They hire remote employees and lots of people are interested in working at Honeycomb and they also do not have the bank account [laugh] that Procter and Gamble has, so just thinking about cost and architecture is kind of a different beast. So, at Honeycomb, we are building a lot more services versus just always choosing a serverless or easy, like, AWS managed option to think about it less.Corey: Yeah, at some level, it’s an unfair question, just because it comes down to, in the fullness of time, even Honeycomb turns into something that looks a lot more like Procter and Gamble. Because, okay, you have the Honeycomb application. That’s great, but as the company continues to grow and offer different things to different styles of customers, you start seeing a diffusion where, yeah, everything stills observability focused, but I can see a future where it becomes a bunch of different subcomponents. You make acquisitions of other companies that wind up being treated as separate environments and the rest. And in the fullness of time, I can absolutely see that that is the path that a lot of companies go down.So, it might also just be that I’m looking at this through a perspective lens of… just company stage, as opposed to what the internal story of the company is. I mean, Procter and Gamble’s, what, a century old give or take? Whereas Honeycomb is an ancient tech company, by which I mean it’s over 18 months old.Brooke: Yeah, P&G was founded in 1837. So that’s—Corey: Almost 200 years old. Wonderful.Brooke: —quite old [laugh]. Yeah [laugh].Corey: And for some reason, they did not choose to build their first technical backbone on top of AWS back then. I don’t understand why, for the life of me.Brooke: [laugh]. Yeah, but totally agree on your point that the kind of difference of thinking about cost and architecture definitely comes from company’s stage rather than necessarily the industry.Corey: I really want to thank you for taking the time out of your day to talk with me about what you’re up to and how you view these things. If people want to learn more, what’s the best place for them to find you?Brooke: Yeah, so I think the main place that I still sometimes am, is Twitter: @codegirlbrooke is my username. But I’m only there sometimes, now [laugh].Corey: I feel like that’s a problem a lot of us are facing right now. Like, I’m more active on Bluesky these days, but it’s still invite only and it feels like it’s too much of a weird flex to wind up moving people to just yet. I’m hoping that changes soon, but we’ll see how it plays. We’ll, of course, put links to that in the [show notes 00:31:53]. I really want to thank you for taking the time out of your day to talk with me.Brooke: Yeah, thanks so much for chatting with me. It was a good time.If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Mike Brevoort, Chief Product Officer at Gitpod, joins Corey on Screaming in the Cloud to discuss all the intricacies of remote development and how Gitpod is simplifying the process. Mike explains why he feels the infinite resources cloud provides can be overlooked when discussing remote versus local development environments, and how simplifying build abstractions is a fantastic goal, but that focusing on the tools you use in a build abstraction in the meantime can be valuable. Corey and Mike also dive into the security concerns that come with remote development, and Mike reveals the upcoming plans for Gitpod’s local conference environment, CDE Universe. About MikeMike has a passion for empowering people to be creative and work together more effectively. He is the Chief Product Officer at Gitpod striving to remove the friction and drudgery from software development through Cloud Developer Environments. He spent the previous four years at Slack where he created Workflow Builder and “Platform 2.0” after his company Missions was acquired by Slack in 2018. Mike lives in Denver, Colorado and enjoys cycling, hiking and being outdoors.Links Referenced: Gitpod: https://www.gitpod.io/ CDE Universe: https://cdeuniverse.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It’s easy to **BEEP** up on AWS. Especially when you’re managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. I have had loud, angry, and admittedly at times uninformed opinions about so many things over the past few years, but something that predates that a lot is my impression on the idea of using remote systems for development work as opposed to doing local dev, and that extends to build and the rest. And my guest today here to argue with me about some of it—or agree; we’ll find out—is Mike Brevoort, Chief Product Officer at Gitpod, which I will henceforth be mispronouncing as JIT-pod because that is the type of jerk I am. Mike, thank you for joining me.Mike: Thank you for insulting my company. I appreciate it.Corey: No, by all means, it’s what we do here.Mike: [laugh].Corey: So, you clearly have opinions on the idea of remote versus local development that—I am using the word remote development; I know you folks like to use the word cloud, in place of remote, but I’m curious to figure out is, is that just the zeitgeist that has shifted? Do you have a belief that it should be in particular places, done in certain ways, et cetera? Where do your opinion on this start and stop?Mike: I think that—I mean, remote is accurate, an accurate description. I don’t like to emphasize the word remote because I don’t think it’s important that it’s remote or local. I think that the term cloud connotes different values around the elasticity of environments and the resources that are more than what you might have on your local machine versus a remote machine. It’s not so much whether the one machine is local or remote as much of it is that there are infinite numbers of resources that you can develop across in the cloud. That’s why we tend to prefer our cloud development environments.Corey: From my perspective, I’ve been spending too many years now living in basically hotels and airports. And when I was doing that, for a long time, the only computer I bring with me has been my iPad Pro. That used to be a little bit on the challenging side and these days, that’s gotten capable enough where it’s no longer interesting in isolation. But there’s no local development environment that is worth basically anything on that. So, I’ve been SSHing into things and using VI as my development environment for many years.When I started off as a grumpy Unix sysadmin, there was something reassuring about the latest state of whatever it is I’m working on lives in a data center somewhere rather than on a laptop, I’m about to leave behind a coffee shop because I’m careless. So, there’s a definite value and sense that I am doing something virtuous, historically. But it didn’t occur to me till I started talking to people about this, just how contentious the idea was. People would love to ask all kinds of fun objections to this where it was, “Oh, well, what about when you’re on a plane and need to do work?” It’s, well, I spend an awful lot of time on planes and that is not a limiting factor in me writing the terrible nonsense that I will charitably called code, in my case. I just don’t find that that idea holds up anywhere. The world has become so increasingly interconnected that that seems unlikely. But I do live in San Francisco, so here, every internet is generally pretty decent; not every place is. What are your thoughts?Mike: I agree. I mean, I think one thing is, I would just like not to think about it, whether I can or can’t develop because I’m connected or not. And I think that we tend to be in a world where that is moreso the case. And I think a lot of times when you’re not connected, you become reconnected soon, like if your connection is not reliable or if you’re going in and out of connectivity issues. And when you’re trying to work on a local laptop and you’re connecting and disconnecting, it’s not like we develop these days, and everything is just isolated on our local laptop, especially we talk about cloud a lot on this podcast and a lot of apps now go way beyond just I’m running a process on my machine and I’m connecting to data on my machine.There are local emulators you could use for some of these services, but most of them are inferior. And if you’re using SQS or using any other, like, cloud-based service, you’re usually, as a developer, connecting to some version of that and if you’re disconnected anyway, you’re not productive either. And so, I find that it’s just like an irrelevant conversation in this new world. And that the way we’ve developed traditionally has not followed along with this view of I need to pile everything in on my laptop, to be able to develop and be productive has not, like, followed along with the trend that moved into the cloud.Corey: Right. The big problem for a long time has been, how do I make this Mac or Windows laptop look a lot like Linux EC2 instance? And there have been a bunch of challenges and incompatibility issues and the rest, and from my perspective, I like to develop in an environment that at least vaguely resembles the production environment it’s going to run in, which in AWS’s case, of course, comes down to expensive. Bu-dum-tss.Mike: Yeah, it’s a really big challenge. It’s been a challenge, right? When you’ve worked with coworkers that were on a Windows machine and you were on a Mac machine, and you had the one person on their Linux machine forever, and we all struggled with trying to mimic these development environments that were representative, ultimately, of what we would run in production. And if you’re counting costs, we can count the cost of those cloud resources, we can count the cost of those laptops, but we also need to count the cost of the people who are using those laptops and how inefficient and how much churn they have, and how… I don’t know, there was for years of my career, someone would show up every morning to the stand-up meeting and say, it’s like, “Well, I wasted all afternoon yesterday trying to work out my, you know, issues with my development environment.” And it’s, like, “I hope I get that sorted out later today and I hope someone can help me.”And so, I think cost is one thing. I think that there’s a lot of inconsistencies that lead to a lot of inefficiencies and churn. And I think that, regardless of where you’re developing, the more that you can make your environments more consistent and sound, not for you, but for your own team and have those be more representative of what you are running in production, the better.Corey: We should disambiguate here because I fear this is one of the areas where my use case tends to veer off into the trees, which is I tend to operate largely in isolation, from a development point of view. I build small, micro things that wind up doing one thing, poorly. And that is, like, what I do is a proof of concept, or to be funny, or to kick the tires on a new technology. I’ll also run a bunch of random things I find off of JIF-ub—yes, that’s how I pronounce GitHub. And that’s great, but it also feels like I’m learning as a result, every stack, and every language, in every various version that it has, and very few of the cloud development environments that I’ve seen, really seems to cater to the idea that simultaneously, I want to have certain affordances in my shell environment set up the way that I want them, tab complete this particular suite of tools generically across the board, but then reset to that baseline and go in a bunch of different directions of, today, it’s Python in this version and tomorrow, it’s Node in this other version, and three, what is a Typescript anyway, and so on and so forth.It feels like it’s either, in most cases, you either get this generic, one-size-fits-everyone in this company, for this project, approach, or it’s, here’s a very baseline untuned thing that does not have any of your dependencies installed. Start from scratch every time. And it’s like, feels like there are two paths, and they both suck. Where are you folks at these days on that spectrum?Mike: Yeah, I think that, you know, one, if you do all of that development across all these different libraries and technology stacks and you’re downloading all these repos from JIF-hub—I say it right—and you’re experimenting, you tend to have a lot of just collision of things. Like if you’re using Python, it’s, like, really a pain to maintain isolation across projects and not have—like, your environment is, like, one big bucket of things on your laptop and it’s very easy to get that into a state where things aren’t working, and then you’re struggling. There’s no big reset on your laptop. I mean, there is but it takes—it’s a full reset of everything that you have.And I think the thing that’s interesting to me about cloud development environments is I could spin one of these up, I could trash it to all hell and just throw it away and get another one. And I could get another one of those at a base of which has been tuned for whatever project or technology I’m working on. So, I could take—you know, do the effort to pre-setup environments, one that is set up with all of my, like, Python tooling, and another one that’s set up with all my, like, Go or Rust tooling, or our front-end development, even as a base repo for what I tend to do or might tend to experiment with. What we find is that, whether you’re working alone or you’re working with coworkers, that setting up a project and all the resources and the modules and the libraries and the dependencies that you have, like, someone has to do that work to wire that up together and the fact that you could just get an environment and get another one and another one, we use this analogy of, like, tissue boxes where, like, you should just be able to pull a new dev environment out of a tissue box and use it and throw it away and pull as many tissues out of the box as you want. And they should be, like, cheap and ephemeral because—and they shouldn’t be long-lived because they shouldn’t be able to drift.And whether you’re working alone or you’re working in a team, it’s the same value. The fact that, like, I could pull on these out, I have it. I’m confident in it of what I got. Like for example, ideally, you would just start a dev environment, it’s available instantly, and you’re ready to code. You’re in this project with—and maybe it’s a project you’ve never developed on. Maybe it’s an open-source project.This is where I think it really improves the sort of equitability of being able to develop, whether it’s in open-source, whether it’s inner-source in companies, being able to approach any project with a click of a button and get the same environment that the tech lead on the project who started it five years ago has, and then I don’t need to worry about that and I get the same environment. And I think that’s the value. And so, whether you’re individual or you’re on a team, you want to be able to experiment and thrash and do things and be able to throw it away and start over again, and not have to—like for example, maybe you’re doing that on your machine and you’re working on this thing and then you actually have to do some real work, and then now that you’ve done something that conflicts with the thing that you’re working on and you’re just kind of caught in this tangled mess, where it’s like, you should just be able to leave that experiment there and just go work on the thing you need to work on. And why can’t you have multiples of these things at any given time?Corey: Right. One of the things I loved about EC2 dev environments has been that I can just spin stuff up and okay, great, it’s time for a new project. Spin up another one and turn it off when I’m done using it—which is the lie we always tell ourselves in cloud and get charged for things we forget to turn off. But then, okay, I need an Intel box one day. Done. Great, awesome. I don’t have any of those lying around here anymore but clickety, clickety, and now I do.It’s nice being able to have that flexibility, but it’s also sometimes disconcerting when I’m trying to figure out what machine I was on when I was building things and the rest, and having unified stories around this becomes super helpful. I’m also finding that my overpowered desktop is far more cost-efficient when I need to compile something challenging, as opposed to finding a big, beefy, EC2 box for that thing as well. So, much of the time, what my remote system is doing is sitting there bored. Even when I’m developing on it, it doesn’t take a lot of modern computer resources to basically handle a text editor. Unless it’s Emacs, in which case, that’s neither here nor there.Mike: [laugh]. I think that the thing that becomes costly, especially when using cloud development environments, is when you have to continue to run them even when you’re not using them for the sake of convenience because you’re not done with it, you’re in the middle of doing some work and it still has to run or you forget to shut it off. If you are going to just spin up a really beefy EC2 instance for an hour to do that big compile and it costs you 78 cents. That’s one thing. I mean, I guess that adds up over time and yes, if you’ve already bought that Mac Studio that’s sitting under your desk, humming, it’s going to be more cost-efficient to use that thing.But there’s, like, an element of convenience here that, like, what if I haven’t bought the Mac Studio, but I still need to do that big beefy compilation? And maybe it’s not on a project I work on every single day; maybe it’s the one that I’m just trying to help out with or just starting to contribute to. And so, I think that we need to get better about, and something that we’re very focused on at JIT-pod, is—Gitpod—is—Corey: [laugh]. I’m going to get you in trouble at this rate.Mike: —[laugh]—is really to optimize that underlying runtime environment so that we can optimize the resources that you’re using only when you’re using it, but also provide a great user experience. Which is, for me, as someone who’s responsible for the product at Gitpod, the thing I want to get to is that you never have to think about a machine. You’re not thinking about this dev environment as something that lives somewhere, that you’re paying for, that there’s a meter spinning that if you forget it, that you’re like, ah, it’s going to cost me a lot of money, that I have to worry about ever losing it. And really, I just want to be able to get a new environment, have one, use it, come back to it when I need it, have it not cost me a lot of money, and be able to have five or ten of those at a time because I’m not as worried about what it’s going to cost me. And I’m sure it’ll cost something, but the convenience factor of being able to get one instantly and have it and not have to worry about it ultimately saves me a lot of time and aggravation and improves my ability to focus and get work done.And right now, we’re still in this mode where we’re still thinking about, is it on my laptop? Is it remote? Is it on this EC2 instance or that EC2 instance? Or is this thing started or stopped? And I think we need to move beyond that and be able to just think of these things as development environments that I use and need and they’re there when I want to, when I need to work on them, and I don’t have to tend to them like cattle.Corey: Speaking of tending large things in herds—I guess that’s sort of for the most tortured analogy slash segway I’ve come up with recently—you folks have a conference coming up soon in San Francisco. What’s the deal with that? And I’ll point out, it’s all on-site, locally, not in the cloud. So, hmm…Mike: Yeah, so we have a local conference environment, a local conference that we’re hosting in San Francisco called CDE Universe on June 1st and 2nd, and we are assembling all the thought leaders in the industry who want to get together and talk about where not just cloud development is going, but really where development is going. And so, there’s us, there’s a lot of companies that have done this themselves. Like, before I joined Gitpod, I was at Slack for four years and I got to see the transition of a, sort of, remote development hosted on EC2 instances transition and how that really empowered our team of hundreds of engineers to be able to contribute and like work together better, more efficiently, to run this giant app that you can’t run just alone on your laptop. And so, Slack is going to be there, they’re going to be talking about their transition to cloud development. The Uber team is going to be there, there’s going to be some other companies.So, Nathan who’s building Zed, he was the one that originally built Adam at GitHub is now building Zed, which is a new IDE, is going to be there. And I can’t mention all the speakers, but there’s going to be a lot of people that are really looking at how do we drive forward development and development environments. And that experience can get a lot better. So, if you’re interested in that, if you’re going to be in San Francisco on June 1st and 2nd and want to talk to these people, learn from them, and help us drive this vision forward for just a better development experience, come hang out with us.Corey: I’m a big fan of collaborating with folks and figuring out what tricks and tips they’ve picked up along the way. And this is coming from the perspective of someone who acts as a solo developer in many cases. But it always drove me a little nuts when you see people spending weeks of their lives configuring their text editor—VIM in my case because I’m no better than these people; I am one of them—and getting it all setup and dialed in. It’s, how much productivity you gaining versus how much time are you spending getting there?And then when all was said and done a few years ago, I found myself switching to VS Code for most of what I do, and—because it’s great—and suddenly the world’s shifting on its axis again. At some point, you want to get away from focusing on productivity on an individualized basis. Now, the rules change when you’re talking about large teams where everyone needs a copy of this running locally or in their dev environment, wherever happens to be, and you’re right, often the first two weeks of a new software engineering job are, you’re now responsible for updating the onboarding docs because it’s been ten minutes since the last time someone went through it. And oh, the versions bumped again of what we would have [unintelligible 00:16:44] brew install on a Mac and suddenly everything’s broken. Yay. I don’t miss those days.Mike: Yeah, the new, like, ARM-based Macs came out and then you were—now all of a sudden, all your builds are broken. We hear that a lot.Corey: Oh, what I love now is that, in many cases, I’m still in a process of, okay, I’m developing locally on an ARM-based Mac and I’m deploying it to a Graviton2-based Lambda or instance, but the CI/CD builder is going to run on Intel, so it’s one of those, what is going on here? Like, there’s a toolchain lag of round embracing ARM as an architecture. That’s mostly been taken care of as things have evolved, but it’s gotten pretty amusing at some point, just as quickly that baseline architecture has shifted for some workloads. And for some companies.Mike: Yeah, and things just seem to be getting more [laugh] and more complicated not less complicated, and so I think the more that we can—Corey: Oh, you noticed?Mike: Try to simplify build abstractions [laugh], you know, the better. But I think in those cases where, I think it’s actually good for people to struggle with setting up their environment sometime, with caring about the tools that they use and their experience developing. I think there has to be some ROI with that. If it’s like a chronic thing that you have to continue to try to fix and make better, it’s one thing, but if you spend a whole day improving the tools that you use to make you a better developer later, I think there’s a ton of value in that. I think we should care a lot about the tools we use.However, that’s not something we want to do every day. I mean, ultimately, I know I don’t build software for the sake of building software. I want to create something. I want to create some value, some change in the world. There’s some product ultimately that I’m trying to build.And, you know, early on, I’ve done a lot of work in my career on, like, workflow-type builders and visual builders and I had this incorrect assumption somewhere along the way—and this came around, like, sort of the maker movement, when everybody was talking about everybody should learn how to code, and I made this assumption that everybody really wants to create; everybody wants to be a creator, and if given the opportunity, they will. And I think what I finally learned is that, actually most people don’t like to create. A lot of people just want to be served; like, they just want to consume and they don’t want the hassle of it. Some people do, if they have the opportunity and the skillsets, too, but it’s also similar to, like, if I’m a professional developer, I need to get my work done. I’m not measured on how well my local tooling is set up; I’m sort of measured on my output and the impact that I have in the organization.I tend to think about, like, chefs. If I’m a chef and I work 60 hours in a restaurant, 70 hours in a restaurant, the last thing I want to do is come home and cook myself a meal. And most of the chefs I know actually don’t have really nice kitchens at home. They, like, tend to, they want other people to cook for them. And so, I think, like, there’s a place in professional setting where you just need to get the work done and you don’t want to worry about all the meta things and the time that you could waste on it.And so, I feel like there’s a happy medium there. I think it’s good for people to care about the tools that they use the environment that they develop in, to really care for that and to curate it and make it better, but there’s got to be some ROI and it’s got to have value to you. You have to enjoy that. Otherwise, you know, what’s the point of it in the first place?Corey: One thing that I used to think about was that if you’re working in regulated industries, as I tended to a fair bit, there’s something very nice about not having any of the data or IP or anything like that locally. Your laptop effectively just becomes a thin client to something that’s already controlled by the existing security and compliance apparatus. That’s very nice, where suddenly it’s all someone steals my iPad, or I drop it into the bay, it’s locked, it’s encrypted. Cool, I go to the store, get myself a new one, restore a backup from iCloud, and I’m up and running again in a very short period of time as if nothing had ever changed. Whereas when I was doing a lot of local development and had bad hard drive issues in the earlier part of my career, well, there goes that month.Mike: Yeah, it’s a really good point. I think that we’re all walking around with these laptops with really sensitive IP on it and that those are in bars and restaurants. And maybe your drives are encrypted, but there’s a lot of additional risks, including, you know, everything that is going over the network, whether I’m on a local coffee shop, and you know, the latest vulnerability that, an update I have to do on my Mac if I’m behind. And there’s actually a lot of risk and having all that just sort of thrown to the wind and spread across the world and there’s a lot of value in having that in a very safe place. And what we’ve even found that, at Gitpod now, like, the latest product we’re working on is one that we called Gitpod Dedicated, which gives you the ability to run inside your own cloud perimeter. And we’re doing that on AWS first, and so we can set up and manage an installation of Gitpod inside your own AWS account.And the reason that became important to us is that a lot of companies, a lot of our customers, treat their source code as their most sensitive intellectual property. And they won’t allow it to leave their perimeter, like, they may run in AWS, but they have this concept of, sort of like, our perimeter and you’re either inside of that and outside of it. And I think this speaks a little bit to a blog post that you wrote a few months ago about the lagging adoption of remote development environments. I think one of those aspects is, sort of, convenience and the user experience, but the other is that you can’t use them very well with your stack and all the tools and resources that you need to use if they’re not running, sort of, close within your perimeter. And so, you know, we’re finding that companies have this need to be able to have greater control, and now with the, sort of, trends around, like, coding assistance and generative AI and it’s even the perfect storm of not only am I like sending my source code from my editor out into some [LM 00:22:36], but I also have the risk of an LM that might be compromised, that’s injecting code and I’m committing on my behalf that may be introducing vulnerabilities. And so, I think, like, getting that off to a secure space that is consistent and sound and can be monitored, to be kept up-to-date, I think it has the ability to, sort of, greatly increase a customer’s security posture.Corey: While we’re here kicking the beehive, for lack of a better term, your support for multiple editors in Gitpod the product, I assumed that most people would go with VS Code because I tend to see it everywhere, and I couldn’t help but notice that neither VI nor Emacs is one of the options, the last time I checked. What are you seeing as far as popularity contests go? And that might be a dangerous question because I’m not suggesting you alienate many of the other vendors who are available, but in the world I live in, it’s pretty clear where the zeitgeist of my subculture is going.Mike: Yeah, I mean, VS Code is definitely the most popular IDE. The majority of people that use Gitpod—and especially we have a, like, a pretty heavy free usage tier—uses it in the browser, just for the convenience of having that in the browser and having many environments in the browser. We tend to find more professional developers use VS Code desktop or the JetBrains suite of IDEs.Corey: Yeah, JetBrains I’m seeing a fair bit of in a bunch of different ways and I think that’s actually most of what your other options are. I feel like people have either gone down the JetBrains path or they haven’t and it seems like it’s very, people who are into it are really into it and people who are not are just, never touch it.Mike: Yeah, and we want to provide the options for people to use the tools that they want to use and feel comfortable on. And we also want to provide a platform for the next generation of IDEs to be able to build on and support and to be able to support this concept of cloud or remote development more natively. So, like I mentioned, Nathan Sobo at Zed, I met up with him last week—I’m in Denver; he’s in Boulder—and we were talking about this and he’s interested in Zed working in the browser, and he’s talked about this publicly. And for us, it’s really interesting because, like, IDEs working in the browser is, like, a really great convenience. It’s not the perfect way to work, necessarily, in all circumstances.There’s some challenges with, like, all this tab sprawl and stuff, but it gives us the opportunity, if we can make Zed work really well in for Gitpod—or anybody else building an IDE—for that to work in the browser. Ultimately what we want is that if you want to use a terminal, we want to create a great experience for you for that. And so, we’re working on this ability in Gitpod to be able to effectively, like, bring your own IDE, if you’re building on that, and to be able to offer it and distribute on Gitpod, to be able to create a new developer tool and make it so that anybody in their Gitpod workspace can launch that as part of their workspace, part of their tool. And we want to see developer tools and IDEs flourish on top of this platform that is cloud development because we want to give people choice. Like, at Gitpod, we’re not building our own IDE anymore.The team started to. They created Theia, which was one of the original cloud, sort of, web-based IDEs that now has been handed over to the Eclipse Foundation. But we moved to VS Code because we found that that’s where the ecosystem were. That’s where our users were, and our customers, and what they wanted to use. But we want to expand beyond that and give people the ability to choose, not only the options that are available today but the options that should be available in the future. And we think that choice is really important.Corey: When you see people kicking the tires on Gitpod for the first time, where does the bulk of their hesitancy come from? Like, what is it where—people, in my experience, don’t love to embrace change. So, it’s always this thing, “This thing sucks,” is sort of the default response to anything that requires them to change their philosophy on something. So okay, great. That is a thing that happens. We’ll see what people say or do. But are they basing it on anything beyond just familiarity and comfort with the old way of doing things or are there certain areas that you’re finding the new customers are having a hard time wrapping their head around?Mike: There’s a couple of things. I think one thing is just habit. People have habits and preferences, which are really valuable because it’s the way that they’ve learned to be successful in their careers and the way that they expect things. Sometimes people have these preferences that are fairly well ingrained that maybe are irrational or rational. And so, one thing is just people’s force of habit.And then getting used to this idea that if it’s not on my laptop, it means—like what you mentioned before, it’s always what-ifs of, like, “What if I’m on a plane?” Or like, “What if I’m at the airport in a hurricane?” “What if I’m on a train with a spotty internet connection?” And so, there’s all these sort of what-if situations. And once people get past that and they start actually using Gitpod and trying to set their projects up, the other limiting factor we have is just connectivity.And that’s, like, connectivity to the other resources that you use to develop. So, whether that’s, you know, package or module repositories or that some internal services or a database that might be running behind a firewall, it’s like getting connectivity to those things. And that’s where the dedicated deployment model that I talked about, running inside of your perimeter on our network, they have control over, kind of helps, and that’s why we’re trying to overcome that. Or if you’re using our SaaS product, using something like Tailscale or a more modern VPN that way. But those are the two main things.It’s like familiarity, this comfort for how to work, sort of, in this new world and not having this level of comfort of, like, it’s running on this thing I can hold, as well as connectivity. And then there is some cost associated with people now paying for this infrastructure they didn’t have to pay for before. And I think it’s a, you know, it’s a mistake to say that we’re going to offset the cost of laptops. Like, that shouldn’t be how you justify a cloud development environment. Like—Corey: Yeah, I feel like people are not requesting under-specced laptops much these days anymore.Mike: It’s just like, I want to use a good laptop; I want to use a really nice laptop with good hardware and that shouldn’t be the cost. The proposition shouldn’t be, it’s like, “Save a thousand dollars on every developer’s laptop by moving this off to the cloud.” It’s really the time savings. It’s the focus. It’s the, you know, removing all of that drift and creating these consistent environments that are more secure, and effectively, like, automating your development environment that’s the same for everybody.But that’s the—I think habits are the big thing. And there is, you know, I talked about a little bit that element of, like, we still have this concept of, like, I have this environment and I start it and it’s there, and I pay for it while it’s there and I have to clean it up or I have to make sure it stopped. I think that still exists and it creates a lot of sort of cognitive overhead of things that I have to manage that I didn’t have to manage before. And I think that we have to—Gitpod needs to be better there and so does everybody else in the industry—about removing that completely. Like, there’s one of the things that I really love that I learned from, like, Stewart Butterfield when I was at Slack was, he always brought up this concept called the convenience threshold.And it was just the idea that when a certain threshold of convenience is met, people’s behavior suddenly changes. And as we thought about products and, like, the availability of features, that it really drove how we thought about even how to think about you know, adoption or, like, what is the threshold, what would it take? And, like, a good example of this is even, like, the way we just use credit cards now or debit cards to pay for things all the time, where we’re used to carry cash. And in the beginning, when it was kind of novel that you could use a credit card to pay for things, like even pay for gas, you always had to have cash because you didn’t know if it’d be accepted. And so, you still had to have cash, you still had to have it on hand, you still had to get it from the ATM, you still have to worry about, like, what if I get there and they don’t accept my cards and how much money is it going to be, so I need to make sure I have enough of it.But the convenience of having this card where I don’t have to carry cash is I don’t have to worry about that anymore, as long as they have money in my bank account. And it wasn’t until those cards were accepted more broadly that I could actually rely on having that card and not having the cash. It’s similar when it comes to cloud development environments. It needs to be more convenient than my local development environment. It needs to be—it’s kind of like early—I remember when laptops became more common, I was used to developing on a desktop, and people were like, nobody’s ever going to develop on a laptop, it’s not powerful enough, the battery runs out, I have to you know, when I close the lid, when you open the lid, it used to take, like, five minutes before, like, it would resume an unhibernate and stuff, and it was amazing where you could just close it and open it and get back to where you were.But like, that’s the case where, like, laptops weren’t convenient as desktops were because they were always plugged in, powered on, you can leave them and you can effectively just come back and sit down and pick up where you left off. And so, I think that this is another moment where we need to make these cloud development environments more convenient to be able to use and ultimately better. And part of that convenience is to make it so that you don’t have to think about all these parts of them of whether they’re running, not running, how much they cost, whether you’re going to be there [unintelligible 00:31:35] or lose their data. Like, that should be the value of it that I don’t have to think about any of that stuff.Corey: So, my last question for you is, when you take a look at people who have migrated to using Gitpod, specifically from the corporate perspective, what are their realizations after the fact—I mean, assuming they still take your phone calls because that’s sort of feedback of a different sort—but what have they realized has worked well? What keeps them happy and coming back and taking your calls?Mike: Yeah, our customers could focus on their business instead of focusing on all the issues that they have with configuring development environments, everything that could go wrong. And so, a good example of this is a customer they have, Quizlet, Quizlet saw a 45-point increase in developer satisfaction and a 60% reduction in incidents, and the time that it takes to onboard new engineers went down to ten minutes. So, we have some customers that we talk to that come to us and say, “It takes us 20 days to onboard an engineer because of all the access they need and everything you need to set up and credentials and things, and now we could boil that down to a button click.” And that’s the thing that we tend to hear from people is that, like, they just don’t have to worry about this anymore and they tend to be able to focus on their business and what the developers are actually trying to do, which is build their product.And in Quizlet’s example, it was really cool to see them mention in one of the recent OpenAI announcements around GPT4 and plugins is they were one of the early customers that built GPT4 plugins, or ChatGPT, and they mentioned that they were sharing a lot of Gitpod URLs around when we reached out to congratulate them. And the thing that was great about that, for us is, like, they were talking about their business and what they were developing and how they were being successful. And we’d rather see Gitpod in your development environment just sort of disappear into the background. We’d actually like to not hear from customers because it’s just working so well from them. So, that’s what we found is that customers are just able to get to this point where they could just focus on their business and focus on what they’re trying to develop and focus on making their customers successful and not have to worry about infrastructure for development.Corey: I think that really says it all. On some level, when you have customers who are happy with what’s happening and how they’re approaching this, that really is the best marketing story I can think of because you can say anything you want about it, but when customers will go out and say, “Yeah, this has made our lives better; please keep doing what you’re doing,” it counts for a lot.Mike: Yeah, I agree. And that’s what we’re trying to do. You know, we’re not trying to win, sort of, a tab versus spaces debate here around local or cloud or—I actually just want to enable customers to be able to do their work of their business and develop software better. We want to try to provide a method and a platform that’s extensible and customizable and gives them all the power they need to be able to just be ready to code, to get to work as soon as they can.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where’s the best place for them to find you, other than at your conference in San Francisco in a few weeks?Mike: [laugh]. Yeah, thank you. I really appreciate the banter back and forth. And I hope to see you there at our conference. You should come. Consider this an invite for June 1st and 2nd in San Francisco at CDE Universe.Corey: Of course. And we will put links to this in the [show notes 00:34:53]. Thank you so much for being so generous with your time. I appreciate it.Mike: Thanks, Corey. That was really fun.Corey: Mike Brevoort, Chief Product Officer at Gitpod. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment detailing exactly why cloud development is not the future, but then lose your content halfway through because your hard drive crashed.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Tom Krazit, Editor in Chief at Runtime, joins Corey on Screaming in the Cloud to discuss what it’s like being a journalist in tech. Corey and Tom discuss how important it is to find your voice as a media personality, and Tom explains why he feels one should never compromise their voice for sponsor approval. Tom reveals how he’s covering tech news at his new publication, Runtime, and how he got his break in the tech journalism industry. Tom also talks about why he decided to build his own publication rather than seek out a corporate job, the value of digging deeper for stories, and why he feels it’s so valuable to be able to articulate the issues engineers care about in simple terms. About TomTom Krazit has written and edited stories about the information technology industry for over 20 years. For the last ten years he has focused specifically on enterprise technology, including all three as-a-service models developed around infrastructure, platform, and enterprise software technologies, security, software development techniques and practices, as well as hardware and chips.Links Referenced: Runtime: https://www.runtime.news/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there’s a problem. With Chronosphere, you can shape and transform observability data based on need, context and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io/corey-quinn. That’s chronosphere.io/corey-quinn. And my thanks to them for sponsor ing my ridiculous nonsense. Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn, and people sometimes confuse me for a journalist. I am most assuredly not one of those. I’m just loud and have opinions and every once in a while I tell people things they didn’t already know. That’s not journalism. My guest today, however, is a journalist, Tom Krazit, is the Editor in Chief of the just launched Runtime. Tom, thank you for joining me.Tom: Thanks, Corey. It’s a long-time listener, first-time guest.Corey: We’ve been talking for years now and I’m sort of embarrassed I haven’t had you on the show before now. But the journalists has always felt, to me at least, like they’re a step apart from the typical, you know, rank and file of those of us working in industry. You folks are different from us, and inviting you all just feels like a faux pas, even though it’s very clearly not. Well, how did you get here, I guess is the short version. I know that you’re at Runtime, now; you were at Protocol until its demise recently. Before that, when I first started tracking you, you were over at GeekWire. Where do you come from?Tom: [laugh]. Well, I’ve been doing this for 20 years, which is a long, long time, and it’s amazing how much has changed in that time. I started off doing consumer stuff, I was covering Apple during the launch of the iPhone, I was covering Google as they sort of turned into the Borg. And then I joined GigaOm in 2012 and I joined them as an editor. And it became pretty clear that I needed to learn this enterprise stuff real fast because that was like the largest part of GigaOm’s business at the time.And so, I kind of just threw myself into it and realized that I actually liked it, you know, which I think is [laugh] hard for some people to understand. But like, I’ve actually always found it really interesting how these large systems work, and how people build in a variety of ways based on their needs, and, you know, just the dramatic change that we’ve seen in this industry over the past ten years. So, you know, I’ve really been doing that ever since.Corey: There’s a lot to be said for journalism in the space. And I know a lot of tech companies are starting to… well, that’s starting. This is, I guess, a six-year-old phenomenon, at least. But a lot of these small companies were built, and well, we’re just going to not talk to the press because we’ve had bad experiences doing that before, so we’re just going to show instead of tell. And that works to a point, but then you hit a certain point of scale where you’re a multi-trillion dollar company and, “We don’t talk to the press,” no longer becomes tenable. With success comes increased scrutiny, and deservedly so. I feel that there’s a certain lack of awareness of that fact in the tech industry versus other large industries that have come before.Tom: I think it’s always important to remember how, like, new a lot of this really still is, you know, when compared to, like, other American industries and businesses. Like, tech as a discipline, you know, it’s only really in the last ten years that it’s been elevated to the extent of, like, sports, or, like, a top-tier news category. And so, I think a lot of people who make those decisions, you know, grew up in a different environment where, you know, you didn’t really want to talk about what you were doing because you were worried about competitive things or you were just worried—you wanted to have a ground-up story. And like, yeah, the world is very different now. And I think that, you know, a lot of companies are starting to get that and starting to change the way they think about it.I mean, I also would argue that I think a lot of enterprise tech companies see better value in running ads alongside golf tournaments than actually talking to people about what they really do because I think a lot of them don’t really want people to understand [laugh] what they do. They want them to think that they’re, you know, the wizard behind the curtain, solving all your digital transformation needs and not actually get into the details of that.Corey: I used to think that I was, as an engineer, much smarter than any of the marketers who were doing these things that obviously make no sense. Like, why would you have a company’s logo in an airport for an enterprise software ad, but no URL or way to go buy something? Aren’t those people foolish? Yeah, it turns out no. People are not just-fell-off-the-turnip-truck level of sophistication.It’s a brand awareness story where you wind up going in and pitching to the board of some big company someday and they already know who you are. That’s the value of brand awareness, as I’ve learned the fun way because I accidentally became something of a marketer. I have this platform—Tom: [crosstalk 00:04:46], Corey—Corey: In the newsletters, but—Tom: Come one. You’re totally a brand. You’re a brand.Corey: Oh, absolutely. And breakfast cereal.Tom: [laugh].Corey: But I was surprised to realize that people not only cared about what I had to say but would pay me cash money in order to have their product mentioned in the thing that I do. And, “Can you give me money? Of course you can give me money.” But it was purely accidental along the way. So, I have to ask, given that you seem to be a fan of, you know, not starving to death, why would you start a media company in 2023?Tom: Uh, well I needed to do something, Corey. You know, like [laugh] [crosstalk 00:05:22]—Corey: You had a bright career in corporate communications if you want to go over to the dark side. Like, “I’m tired of talking to the audience about truth, I’d rather spin things now because I know how the story gets told.”Tom: I mean, that may come down the road for me at some point, but I wasn’t quite ready for that just yet. I have really felt very strongly for a long time that this particular corner of the world needs better journalism. I just, I feel like a lot of what is served up to the people who have to make decisions about this incredibly complicated part of the world, you know, it’s either really, really product-oriented, like, “So-and-so introduced the new thing today. It costs this much and it does these three things that they told us under embargo,” you know, or you get, like, real surface-level coverage from, like, the big financial business publications, you know, who understand the importance of things like cloud and things like enterprise software, but haven’t really invested the time to understand the technological complexities behind it and how, you know, easy narratives don’t necessarily, you know, play in this world.So, there’s a middle ground there that I think we at Protocol Enterprise found pretty fertile. And, you know, I think that, for this, for Runtime, you know, I’m really just continuing to carry that work forward and to give people content they need to make decisions about using technology in their businesses that business people can understand without an engineering degree, but that engineers will take a look at it and they’ll go, “You know what? He did that right. He did his homework, he got the details right.” And I think that’s rare, unfortunately, and then that’s a gap I hope to fill.Corey: Something that really struck me as being aligned with how I tend to view things is—to be clear, our timing is a little weird because to my understanding, the inaugural issue is going out later today after we record—Tom: That’s correct.Corey: But that would have already happened and have landed in the industry by the time people listen to this. So, I’m really hoping, first off, that the first issue isn’t horrifying to a point where, “Oh God, distance myself from it. What have I done?” But you’ve been in this industry enough that I doubt that’s going to be how you play it. But I am curious to know how it winds up finding its voice over the coming weeks and months. Even when you’ve done this before, as you have I think that every publication starts to have a different area of focus, a different audience, and focus on different aspects of this, which is great because I don’t want to see the same take from fifteen different journalist publications.Tom: Totally. I mean, you know, I think a lot of what Protocol Enterprise was, was my voice and, you know, how I thought about this industry and wanted to bring it forward. And so, I think that, you know, off the bat, a lot of what Runtime is will be similar to that. But to your point, I think everything changes. The market changes, what people want changes, I mean, like, look, just the last six months, the rise of all this generative AI discussion has dramatically changed a lot of what software—you know, how it’s discussed and how it’s thought about, and those are things that, you know, six months ago, we were talking about, maybe, here and there, but we certainly weren’t talking about them to degree than we are now.So like, those changes will happen over the coming months. And you know, you just have to sort of keep up with them and make sure—my job is to make sure I am talking to the right people who can put those things into context for the people who need to understand them in order to make their own decisions. You know, I mean, I think we talk a lot about the top-tier decision makers, you know, of companies who need information, but I think there’s, like, a whole other, I don’t want to call them an underclass, but like, you know, there’s a lot of other people within companies who advise those people and who genuinely need help trying to understand the pace and the degree to which things have changed and whether or not it’s worth it for them to invest, you know, hundreds of thousands, if not millions, of dollars in some of these new technologies. So, you know, that’s kind of the voice I want to bring forward is to represent the buyer, to represent the person who has to make sense of all this and decide whether or not, you know, the sparkly magic beans coming down from the cloud providers and others are really what it’s cracked up to be.Corey: The thing that really throws me is that when I started talking to you and other journalists where you speak generally to a tech-savvy audience, but for whatever reason, that audience and you by extension are not as deeply involved in every nuance of the AWS ecosystem or the cloud computing ecosystem as I am. So, I can complain for five minutes straight to you about the Managed NAT Gateways and their pricing and then you’ll finally say, “Yeah, I don’t know what any of the words Managed, NAT, or Gateway mean in this context. Can you distill that down for me?” It’s, “Oh, right. Talking about what I mean in a way that someone who isn’t me with my experience can understand it.” I mean, that is such a foreign concept to so many engineers that speaking clearly about what they mean is now being called prompt engineering, instead of, “Describe what you want in plain English.”Tom: Yeah. I think that’s a lot of what I hope to accomplish, actually, is to be able to talk to really smart engineers who are really driving this industry forward from their contributions and be able to articulate, like, what it is that they’re concerned about, like, what it is that they think is exciting, and to put that into context for people who, you know, who don’t know what a gateway is, let alone, like, any of [laugh] those other words you used. So, you know, like, I think there’s a real opportunity to do that and that’s the kind of thing I get excited about.Corey: I am curious, given that you are just launching at this point, and you have the express intention of being sponsor-supported, as opposed to a subscriber-driven model, which I’ve thought about a lot over the past, however many years you want to wind up describing I’ve been doing this. The problem that I’ve got here is that I have always found that whenever I’m doing something that aligns with making money and taking a sponsor message and putting it out to the world, how do I keep that from informing the coverage? And I’ve had to go a fair bit out of my way to avoid that. For example, this podcast is going to have ads inserted into it. I don’t know what they are, I don’t know who these companies are, and that only gets done after I’ve recorded this episode, so I’m not being restrained by, “Ohh, have to say something nice about Company X because they’re sponsoring this episode.” It stays away.Conversely, if I want to criticize Company X, I don’t feel that I can’t do that because well, they are paying the bills around here. You’re still in a very early stage where it is you, primarily. How are you avoiding that, I guess, sense of vendor capture?Tom: You have to be very intentional about it from day one. You have to make it clear when you’re talking to sponsors from the business side where the lines are drawn. And you have to, I think from the editorial side, just be fearless and be willing to speak the truth. And if you get negative reaction from sponsors over something you’ve said, they were never going to be a good long-term partner for you anyway. And I’ve seen that over the years.Like, companies that get annoyed about coverage because they’re sponsors are insecure companies. It’s almost a tell, you know, like when you attempt to put pressure on editorial organizations because you’re a sponsor and you don’t like the way that they’re covering something [laugh], it’s a deep, deep tell about the state of your business and how you see it. So, like for me, those are almost like signals to use and then go deeper, you know? And then, you know, I do think that there are enough companies that feel strongly about wanting to support the kind of work that I do without impugning the way I think about it, or the way I write about it. Because I mean, like, there’s just no other way to do what I do without pulling punches.And I think you would agree, you know, in terms of what you do, like, the voice that you have, the authenticity that you have, is your selling point. And if you compromise that, people know. It’s pretty obvious when you are bending your coverage to suit your sponsors. And there’s examples of it every day in enterprise tech coverage. And you know, I feel like my track record speaks for itself on that.Corey: I would agree. I don’t like everything you write. That’s kind of the point. I think that if you look at anyone who’s been even moderately prolific and you like everything that they’re writing, are they actually doing journalism or are they catering to your specific viewpoint? Now, that doesn’t mean that well, I don’t like this particular journalist. It’s, well, “Oh, because you don’t agree with what they say?” “No, because they’re editorially sloppy, they take shortcuts, and they apparently peddle misinformation gleefully.”Yeah, I don’t like a lot of that type of coverage. I’ve never seen that from you. And you’ve had takes I don’t agree with, you’ve had articles that I thought were misleading at times, but I’ve never gotten the sense at all that they were written in bad faith. And when I run into that, it often makes me question my own biases as well, which is sort of a good thing.Tom: I mean, it’s really tough because there are people out there in journalism and media who are operating in bad faith. Like, there’s just no… there’s no other way to dance around that. That is a fact of life in the 21st century. And I mean, all I can really do is do what I do every day and put it out there and, you know, let people judge it for what it is. And you know, like, I feel like, I have a pretty strong sense of what I will, you know, what I’ll cover and how I’ll cover it and where I’ll go with it, and I think that that sort of governs, you know, every editorial decision that I’ve ever made. For me, there’s just no other way to do it. And if I get to a point where I have to make those compromises in order to have a business, like, I’ll just go do something else. I don’t need this that much.Corey: When I was starting the Duckbill Group, one of the problems that I had was—it’s hard to start a company for a variety of reasons, but one that is not particularly sympathetic is that everything is hard when you’re just starting out. You don’t know where any business is going to come from if it ever does. And at any point, I looked around, and I have an engineering skill set and I live in San Francisco, and I look around and say, it’s Wednesday. I could have a job at a big tech company for hundreds of thousands of dollars a year by Friday if I just go out and say yes. And it’s resisting that siren call while building something myself that was really hard.You have that challenge as well, I’d have to imagine because there are always people that various companies are looking to build out their PR and corporate comms groups, and people who understand the industry and know how to tell a story, which you clearly qualify, are always in demand, regardless of the macroeconomic conditions. So, at any point, you have the sort of devil on your shoulder saying, it doesn’t need to be this hard. There’s an easier, more lucrative path instead of struggling to get something off the ground yourself. Do you find that that becomes a tempting thing that you want to give into, or is it, “Mmm, not today, Satan?”Tom: The latter. I mean, I’ve had offers from companies I respect and from people I would, you know, be happy to work with under other circumstances. But I mean, I sort of feel like I’m just wired this way. And then that’s, like, what I enjoy getting out of bed every day to do, is this. And, you know, like, it’s not to say that I couldn’t find, long-term, some kind of role inside one of those types of companies that you just mentioned, but I’m not ready for that yet.And, you know, I think I’d bring more value to the industry this way than I would jump in on some pre-IPO rocket ship kind of thing right now. I will say that, like, a lot of this business is a young person’s game, so like, that equation changes as you get older. I always tell everybody that, like, journalism over the last 20 years has been, like, one of the slowest-moving games of musical chairs that you’ll ever play. And, you know, I’ve [laugh] been pretty lucky over the past number of years to keep getting a chair, you know, in every single one of those downturns. But, you know, I’m not naive enough to think that my luck would  run out one day either. But I mean, if I build my own business, hopefully, I can control that.Corey: There are a lot of tech publications out there and I’m curious as to what direction you plan to take Runtime in, given that it is just you, and you presumably, you know, sleep sometimes, it’s probably not breaking news with the first take on absolutely everything, which just, frankly, sounds exhausting. One of the internal models we have here is the best take, not the first take. So, where does your coverage intend to start? Where does it intend to stop? And how fixed is that?Tom: Well, at the moment, you know, what we really want to do is tell the stories that the herd is not telling. And you know, we’re making a very deliberate decision to avoid a lot of the embargoed product training—I —I don’t know how many of your listeners actually know how the sausage is made, but like, so many PR departments and marketing departments in tech really like to tell news through these embargoed product announcement things. And they’ll email you a couple days ahead of time and they’ll say, “Hey, Tom, we’ve got a new thing coming up in our, whatever, cloud storage services area. You know, are you interested in learning more under embargo?” And then a lot of people just say, “Sure,” and take a briefing and write up a story.And like, there’s nothing inherently wrong with that. It is news and it is—if you think it’s interesting enough to bring out to people, like, great. There’s a lot of limitations to it, though, you know, in that you can’t really get context around that story because you sort of by definition, if you agree to not tell anybody about this thing that the company told you, you can’t go out and ask a third-party expert what they think about it. So, you know, I think that it’s a way to control the narrative without really getting the proper story out there. And the hook is that you’ll be first.And so, I think what we’re trying to do is to step away from that and to really tell more impactful stories that take more time to put together. And I mean, I’ve been on all sides of the news business and when you get on the hamster wheel, you really don’t have time to tell those stories because you’re too busy trying to deal with the output you’ve already committed to. And so, like, one thing that Runtime will be doing right off the bat is taking the time to do those stories to interview the people who don’t get talked to as much, who don’t have twenty-five PR people on staff to blast the world about their accomplishments, you know, to really go out and find the stories that aren’t being told, and to elevate the voices that aren’t being heard, and to shine a light on some of the, you know, more complex technological things that others simply don’t take the time to figure out.Corey: Well, do you have an intended publication schedule at this point or is it going to be when it makes sense? Because one of the things that drove me nuts that I would go back and change if I could is Last Week in AWS inherently has a timeliness to it and covers things over a certain timeframe as well. I don’t get to take two weeks off and pre-write this stuff.Tom: Yeah. So the primary vehicle right now is an email newsletter for Runtime and that’ll come out three times a week on Tuesdays, Thursdays, and Saturday mornings. You know, I’ll also be publishing stories alongside those newsletters. That will be a little more ad hoc. You know, I’d like to have that line up with the newsletters, but you know, sometimes that’s not, you know, a schedule you can really adhere to.But the newsletter is a three times a week operation at the moment and that, you know, is just basically based on—you know, at Protocol, we did five times a week with a staff of six. And that was a big effort. So, I decided that was probably not the best thing for me to tackle right off the bat here. So, one thing I really would like to do with Runtime is to get back to that place where there’s a staff, there’s beat reporters, there’s people who can really take the time to dig into these different areas, you know, across cloud infrastructure, AI, or security, or software development, you know, like, who can really, really plunge themselves into that, and then we can bring a broad product to the market. You know, it’ll take some time to get there, but that’s the goal.Corey: How do you intend to measure success? I mean, there’s obviously financial ways of doing it, but it’s also one of those areas of, like, one of the things that drove me nuts is that I’ll do something exhaustively researched that takes me forever to get out, and no one seems to notice or care. And then I’ll just slap something off eleventh hour, and it goes around the internet three times. And I always find that intensely frustrating. How do you measure whether you’ve succeeded or whether you failed?Tom: Well, I mean, welcome to the internet, Corey. That’s just how it works. I think I will be able to measure that, you know, by how sustainable of venture this is, and like, whether or not I can get back to that point where, you know, we can support a small team to do this because I, you know, I sort of feel that that’s the best—that’s really what this part of the world needs is that kind of broader coverage from subject matter experts who can really dive into things. I mean, I know a lot about a lot, but I can’t spend all my time talking to security people to really understand what’s happening in that market, and the same for any other, you know, one of these disciplines that we talk about. So, you know, if a year from now, I come on this show for the one-year anniversary of the launch, and we’ve got sustainable runway, we’ve got, you know, a few people on board, I’ll be thrilled. That’ll be great, you know?And like, one thing that I think will really be helpful, for me, at least in terms of determining how successful this is, is just how things travel, and not necessarily like traffic in terms of how things travel, I think that’s an easy trap to fall into, but whether or not you know, the stuff that we write about is circulating in the right places and also showing up in the coverage that some of those broader business financial publications actually wind up doing. You know, if you can show that, like, the work you’ve been doing is influencing the conversation of some of these topics on a broader national and global scene, then for me, that’ll be a home run.Corey: Taking a step back, what advice would you give someone who’s toying with the idea of entering the media space in this era, whether that be starting their own publication or becoming a journalist through more traditional means? Because as you said, you’ve been doing this for 20 years; you’ve seen a lot of change. How would you get started today?Tom: It was a lot easier. It was smaller. It was just a much smaller industry when I first started doing this, and… there wasn’t social media. The big challenge, I think, for a lot of people who are just starting now and trying to break through is just how many voices there are and, like, trying to get a foothold among a much, much bigger pond. Like, it was just a much [laugh] smaller pond when I started, and so you know, it was easier to stand out, I guess.I started in the trade magazine world; I started with IDG and I started—you know, which is a real, great bedrock system of knowledge for people to really get their footing in this industry on. And you know, you can count on many, many hands the number of people who started at IDG and have gone on to, you know, a very successful tech and media careers throughout. So, you know, for me, that was a big, that was a big thing. But that was a moment in time. And like, you know, the world now is so different.The only thing that has ever worked, though, is to just write, to just start, to just get out there and do what you’re doing and develop a voice and find a way to get it to the people who you want to read it. And you know, if you keep at it, you can start to break through. And, like, it’s a slog, I’m not going to pretend otherwise, but yeah, if it’s a career you really want to do, the best way to do it is just to start. And the nice thing about the modern era, actually is, like, there’s never been easier ways to get up and running. I mean you look at like things like Substack, or I’m using Ghost, you know, like, the tools are there in a way that they weren’t 20 years ago when I started.Corey: Step one: learn how to build a web server is no longer your thing. No, I think that that’s valuable. One of the things that I find at least is people are so focused on the nuts and bolts, the production quality. People reach out to me all the time and say, “What microphone should I get? What my audio setup should I use? What tools should I do for the rest of this?”And it’s, realize that it doesn’t matter how much you invest in production quality; if the content isn’t interesting and the story you have to tell doesn’t grip people, it doesn’t matter. No one cares. You have to get their attention first and then, then you can scale up on the production quality. I think I’m on generation six or so of my current AV setup. But that happened as a result of basically, more or less recording into a string can when I first started doing this stuff. Focus on the important part of the story, the differentiated parts.Tom: The best piece of advice, I got when starting Runtime was just to start. Like, don’t worry about building a perfect website, don’t worry about, you know, getting everything all dialed in exactly the way you want it. Just get out there and do the work that you’re doing. And it’s also a weird time right now, obviously, with the [laugh] the demise of Twitter as a vehicle for a lot of this stuff. Like, I think a lot of journalists are really having to figure out what their new social media distribution strategies are and I don’t think anybody’s really settled on anything definitive just yet.So, that’s going to be an interesting wrinkle over this year. And then I think, you know, there’s also still a lot of concern about the broader economy, you know, advertising is always one of those things that can be the first to go when businesses start to look at the bottom line a little bit more closely. But those things always come around, you know, and when the economy does start to get a little bit better, I think, you know, we’ve seen a little bit more, maybe [unintelligible 00:26:28] of the market over the last couple of weeks, you know, with some of the earnings results that we’ve seen. So, you know, like, I mean, those are famous last words, obviously, but I think that looking forward into the second half of the year, people are starting to get a little more confident.Corey: I sure hope so. I really want to thank you for being so generous with your time. If people want to learn more, and—as they should—subscribe to see how Runtime plays out, where can they find you?Tom: runtime.news.Corey: Excellent. We’ll, of course, put a link to that in the [show notes 00:26:54]. I’m really looking forward to getting the first issue in a few hours myself. Thanks again for your time. I really appreciate it.Tom: Thanks, Corey.Corey: Tom Krazit, Editor in Chief at Runtime. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling us that your company’s product is being dramatically misunderstood and to please issue a correction.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.