GOTO - The Brightest Minds in Tech

Software Engineering at Google • Titus Winters & Matt Kulukundis

• Titus Winters, Matt Kulukundis & GOTO • Season 2 • Episode 49

Share episode

This interview was recorded for the GOTO Book Club.
gotopia.tech/bookclub

Read the full transcription of the interview here

Titus Winters - Principal Software Engineer at Google and Co-Curator of "Software Engineering at Google"
Matt Kulukundis - Senior Staff Software Engineer at Google

DESCRIPTION
What’s the difference between programming and software engineering?
Join Titus Winters, co-curator of “Software Engineering at Google”, and Matt Kulukundis while they approach the lessons learned by software engineering teams at Google in establishing the right practices for writing sustainable code in a safe environment. Discover what Google is still trying to improve on and what software decisions are difficult to undo.

The interview is based on Titus' co-curated book "Software Engineering at Google"

RECOMMENDED BOOKS
Titus Winters, Tom Manshreck & Hyrum Wright • Software Engineering at Google
Forsgren, Humble & Kim • Accelerate: The Science of Lean Software and DevOps
George Fairbanks • Just Enough Software Architecture
Fred Brooks Jr. • The Mythical Man-Month
Kim Scott • Just Work
Douglas R. Hofstadter • Gödel, Escher, Bach
Douglas R. Hofstadter • I Am a Strange Loop
Alasdair MacIntyre • After Virtue
N. K. Jemisin • The Fifth Season
Becky Chambers • Wayfarers Series
Ken Liu • The Dandelion Dynasty
David Farley • Modern Software Engineering
Martin Kleppmann • Designing Data-Intensive Applications
Zhamak Dehghani • Data Mesh

Twitter
Instagram
LinkedIn
Facebook

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket: gotopia.tech

SUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted daily!

Intro

Matthew Fowles Kulukundis: All right. Well, Titus, it's weird for me to have a conversation with you like this without tiki drinks in hand but I guess we'll just run with it. I'm Matthew Fowles Kulukundis, I'm here to talk with Titus Winters about software engineering at Google. And I want to start with...just like the cover says, "Curated by Titus Winters Winters, Tom Manshreck, and Hyrum Wright," what does "curated by" really mean?

Titus Winters: We wanted to be as upfront as we could be about the fact that many of the chapters, most of the chapters are written by people whose names are not those three primaries. We were the project leads and did the lion's share of editing and feedback for authors and all of those things. We spent a lot of time on outlines and theses and trying to have a consistent voice across the text. But it would be false to claim that this was all our insight or all our input. We tried to come up with an interesting useful list of topics and then match, more experienced people to those topics and then filled in the gaps ourselves where we had to.

Build systems

Matthew Fowles Kulukundis: So which parts of it were when you got the chapters, were most greenfield to you, or where you were least familiar with the content of the chapters?

Titus Winters: I would say that for me, the most surprising place, I'll go with that at least, was the build systems chapter which I remember so vividly being the first time that there was a chapter outside of like, "We're going to have a project here." The chapter's not written by any of the three main authors, where we got a chapter that was, "Oh. Oh, that's how that works. Oh, I feel like I have learned so much all of a sudden." The writing quality and the technical depth there and the insight were just wonderful.

Matthew Fowles Kulukundis: What do you mean? Everyone knows that a build system is just a shell script as God and nature intended.

Titus Winters: So, part of the reason why that's one of my favorite chapters is that that's pretty much exactly where Eric starts with it, right? Starts with walking us through the sort of natural, "I got tired of this toil work progression that leads us to most build systems." And he starts with, …you invoke the compiler and you get tired of that and then you have this shell script, as you said, as God intended. Then we move up a level to something a little bit more, like make and what all of this is missing, and the thesis of the chapter that's just still one of my favorite things is the idea that the next important step follows the evolution of programming languages and what you want in your build system to be modeling a functional language. Specifically, because you need it to be hyper-aware of side effects and where there are state dependencies.

And yet, at the point where you have no unknown side effects, caching and parallelization and dependable builds are natural. What I have come to take out of all of that is if you ever have to run make clean or the equivalent for your, you know, your build system...

Matthew Fowles Kulukundis:Wait, which level? Mike Ruther said make claim, make disclaim and make real claim..

Titus Winters: Right. Like, if you have to do any of those things, your build system is fundamentally bad. And increasingly what I do, like...customer visits and talks in consulting and stuff. And they're like, "If you could fix one thing in, you know, tech or our technical systems, what piece of advice would you give?" You are not going to believe me but fundamentally, fix your build system. So much downstream productivity boon comes from...you don't have to question whether the build is good. You don't have to spend time doing a fresh build. The computer knows, like, what parts need updating.

Matthew Fowles Kulukundis: I find that interesting because of the way you described how you start with running events directly. Then you have a shell script that cobbles them together. I've been at companies where, like...okay. And then we wanted to add distributed things. So, you change your compiler from CC to CC and you're just familiar with the hacks. Like, deeper. At some point, you need to stop and re-envision this as a system. And I think there's something similar that happens with version control.

Titus Winters: Ok.

Matthew Fowles Kulukundis: I started with it like I had some files and I made some copies of some files. And then I have a dated ZIP file that I tarball periodically. And I just think the ecosystem is better...has better embraced the idea that your version control needs to be a file system, that your build needs to be a thought-out system.

Titus Winters: I think that that is correct. I think in the version control space, it was very obvious. We didn't have atomic commits for multiple files at once. And we ran into problems with those race conditions. With the build system and reproducibility and hermicity sort of properties. Somehow is less obvious that that is a thing that the build system can and should be doing. And I think part of that is our conceptualization of version control sort of always implied that you could commit multiple files at once. And then when it failed, that was just an expectation violation whereas the magical Shangri-La of your build system just works and it's cacheable and parallelizable and all this stuff, that doesn't feel like a thing that we assume out of the gate, although we really should. But I think that's what's held up the evolution gap between those two things.

Build systems dependencies

Matthew Fowles Kulukundis: There's a continual monkey knife fight between your build system and the language dependency graph of, like...I know that there's a whole chapter on dependencies that I promise someday I'll read. But can you speak about how and at which level dependencies and build systems interact?

Titus Winters: Well, if we accept the model that your build should be hermetic and should be reproducible, this is very good for the organization as a whole. Because you stop discussing, "Well, it worked on my machine." Like, there is a ground truth and also most people are going to experience the ground truth by default.

Matthew Fowles Kulukundis: For a moment let's pretend I'm as dumb as you think I am. Can you define hermeticity?

Titus Winters: Having no external dependencies that aren't tracked by the build system and your version control system. So, one of the ways that we violate that all the time now is in just the dev model for shared libraries for instance. Go to the Linux distribution model, right? If you have headers and .so's installed and the product that you're working on depends on those, right, you're deeply not hermetic, right? The particular patch version of those installed libraries might be different on your machine versus on the CI server versus on your officemate's machine.

Matthew Fowles Kulukundis: Right. This is the sort of thing we use Docker, virtual stuff for.

Titus Winters: Right. Exactly. Part of our sort of conceptualization and usage of most programming languages is where do you pick up sort of ambiently installed packages? And that is a distinct and direct tension with having more hermetic builds. It's a habit that I think we have to learn to break. Like, having developer packages installed on your machine is valuable for hobby stuff and when you're not coordinating with a lot of other people. But one of the things that we do have to sort of giving up is... it can't just be whatever you happen to have laying around. We need to have a bill of materials. We have to understand...the ground truth of what is part of the build.

Matthew Fowles Kulukundis: Right. You're espousing heresy that I should be able to update the OS on my dev machine without updating all of the versions associated with my build.

Titus Winters: Yes. This is a heresy that I would espouse, yeah.

Matthew Fowles Kulukundis: So, there's an interesting hermeticity, like, it's this sort of recurring theme that comes out of this and I think it plays interestingly in testing. And I was hoping to tease a little bit more testing about this and where hermeticity breaks down in testing.

Titus Winters: If you look at pretty much any published description of what are the properties that your tests should have, hermeticity shows up in this list, right? And this is largely because it's very easy to run into situations where only one person in the organization can run the tests at a time. After all, you have a dependence on some shared state, some...

Matthew Fowles Kulukundis: That's the QA engineer.

Titus Winters: ...database...right. And this is going to be a scaling problem. The more that we can get our especially unit tests to be more hermetic, the more they are not being...our results from those tests are not being affected by stuff outside of what we understand to be...being tested, right. The system under test is much more controlled. As you sort of expand out from the unit test environment into more of an integration test regime, then you have to draw different boundaries on what is the system under test, is the terminology we tend to use. And it...there will probably come a point in that scale where it can't be entirely hermetic and isn't valuable to be hermetic anymore, even. If you need, you know, a production ML model or something to get end-to-end results on is this feature working, having the system drawn with such tight boundaries that you don't have access to that model doesn't give you faith that your end-to-end test is working anymore, right.

And so increasingly, like, I hear arguments on both sides of we shouldn't have unit tests because they're gonna miss things. They don't have high enough fidelity. We should just rely on those integration tests and end-to-end tests. Or the end-to-end tests, the integration tests are too expensive. We can't possibly, like, use only that. And both of these arguments are true. Fundamentally, like, there needs to be a defense in depth and kind of a gradient in there. I like looking at it these days as what's the cheapest mechanism you can find to reliably prevent that class of bugs for whatever thing is bothering you at the moment.

Matthew Fowles Kulukundis: I sort of have a hot take about integration tests which are integration tests are the uncanny valley of tests. But they look like tests, they act like tests, but they're not and they just cause pain. So, you would know because I'm convinced of yes.

Titus Winters:In a perfect world, I think it's much more justifiable that we need unit tests. And I tend to recommend to groups, "Start with unit tests until they're not giving you enough faith, enough safety net, enough fidelity to predict your downstream problems." And then on the extreme other end of it for, like, release qualification and running canaries, like, I have a potential update to my release that I want to evaluate, right? That's effectively an end-to-end test in a slightly different coat. But in between, there are things that you can do that are much cheaper than that release qualification canary end-to-end sort of thing but are distinctly not as a hermetic and fast and single process as you would hope for from a unit test. And it varies from project to project and domain to domain, at what point do you need to reach for that, and how expensive are those things gonna be?

Matthew Fowles Kulukundis: I have a question on this a little bit because I have this, vague theory that everything that we get...we push on it only after we are already hit the point where we should've been pushing on it and we have to re-envision a thing as assist. Like, we did that with our build system. We did that with our unit tests. And at the moment, when people reach for integration tests, they would've been better off reaching for CI, for continuous...sorry, CD, continuous delivery, and continuous rollback sorts of things.

Titus Winters: I think that there are...I don't think that there's natural law here

Matthew Fowles Kulukundis: ...cars and aircraft and things that have to work.

Titus Winters: There's a talk that I caught internally and I don't think that there's an external version of it but the idea holds that the absolute highest value thing that you can do for proving that your release is good is canary. This is not shocking, right? This is the highest fidelity thing you can do.

Matthew Fowles Kulukundis: Only if you can understand the metric.

Titus Winters: Sure. But I would certainly argue that if you can achieve the strategic outcome of...I can run canary every few hours safely and it's not going to harm live users. I can't think of a better strategic footing to be on. And if you are in that state, then you could imagine maybe stepping away from your commitment to integration tests. But I suspect that even then you're gonna find classes of things that could've been caught earlier and cheaper. And if that is the case, then you still need integration tests.

The other place that integration tests had come up that is valuable is sort of the exploratory not CI use case. It's like, "I'm making the search change. I need to spin up a little, you know, indexing back end and a little front-end piece and a little other thing, right." If those can scale down so that you can just run them locally on your desktop and use that as the system under test while you poke and prod and, like, reboot your particular piece of it, that's a vastly better development experience than having to wait until the next scheduled integration, like, full system integration test.

Especially if those are being shared across the team and you're not going to get a hermetic result. There's the word again, hermetic results. Even if your idea of integration tests would be better off replaced strategically by continuous delivery and good canary and all of that stuff is true, which it might be, having the additional use case affordance of this improves developer experience, and gives you better cycle times on real seeming systems. Still integration testing.

Matthew Fowles Kulukundis:  I think you flirted into an interesting area there of performance, right? Let's put aside correctness and talk about, like, I'm an engineer whose job is to make it fly. How do I keep that loop tight enough but believable enough? And I think you're right there...I mean, something more like an integration test is...although, man, I love...in certain circumstances, I've had to bring up a cluster, even numbers have something and the odd numbers have something else if you do an AB test.

Titus Winters: Yes. I think that our whole approach to continuous delivery and continuous integration and experimentation and building local systems under test for manual testing and development work, I think is still a space where there's a lot of room for codifying a few good ideas that are out there. But I don't think that we're consistently all on the same page on that stuff.

What is Google trying to improve in engineering?

Matthew Fowles Kulukundis: So, as I sort of draw the technical side to a close for a moment...we'll probably come back to it but the book is software engineering at Google. Which of these things do you think Google stands to improve the most? Where is it furthest from the city on a hill?

Titus Winters: I think a lot of the culture chapters are a little bit aspirational.

Matthew Fowles Kulukundis: Not a bad culture. We're going to go to culture. First, though, the technical.

Titus Winters: You just want the technology. I think some of the ideas in the testing chapters are a little bit aspirational. But I think by far the biggest gap between this is what we think we understand and this is how stuff is currently actually operating is almost certainly the dependency management chapter.

Matthew Fowles Kulukundis: Interesting.

Titus Winters: That chapter gave me literal nightmares. I cannot express how challenging that particular piece was. And most of that boils down to there being just not great answers here. We have a lot of ideas of things that we don't think scale or that we don't think work well or that are just going to be high toil. But what we don't have is a proposal for what does work and is cheap and is intuitive and scales nicely.

We're starting to take some steps towards being a little better on some of that management front but fundamentally, the guidance that I have...I forget which of those chapters it is but it's largely...I prefer any number of version control problems over a single dependency management problem, right? If you have lost track of what is this depending on, what version is this dependent on, do I have choices in which version to use...we've lost coordination between these things.

It may not sound like that is a huge cost but especially when you start to iterate on that, that is just a deadly cost, right?

Matthew Fowles Kulukundis: I believe the answer there is continuous deployment. Like, I kinda think Google is emerging from, like...it's at the very beginning of the Renaissance. Like, you have some people, some projects that are doing well and you have some others that are just like, "My peasants can toil the fields."

Titus Winters: Yes. There is I think, a valid approach. I have increasingly been...like, I love the research that DORA does, the DevOps Research Association which feels a little bit more self-serving because we acquired them a while back. Sadly, I don't interact with them enough. They don't publish internally enough. I've seen more of their stuff externally than internally.

Matthew Fowles Kulukundis: So, this is a different one than the one that Sierra Jasmin wrote a chapter on?

Titus Winters: Yes. DORA wrote Accelerate and the State of DevOps reports. DORA's model has been externally focused surveys across the industry of teams looking for what are your technical practices, what are your outcomes. And so, DORA gives us statistical evidence. Sometimes causal, not just corelative practices, like, trunk-based development and focusing on your documentation quality lead to good technical outcomes.

And one of their major drumbeats for a long time has been continuous delivery. And it's, I think, specifically because when you are strategically capable of putting out a release on a couple of hours or daily sort of basis, that means that you have your automation system in hand, your tests are of sufficient quality and sufficiently fast to predict the quality of your release. The technical excellence level required to get there takes care of most of the quality problems.

And then in return, you get the ability to move fast, run experiments quickly, get feedback from actual users quickly, and resolve issues quickly. You stop having to have release branches and cherry-picks because just spin another one. Who cares, right? Like, all of this stuff just simplifies. And to your point of where we fail, I would be thrilled to figure out how to get team leads and directors and things all over the company to set continuous delivery targets. Because what we find very regularly unsurprisingly is the directors and people above aren't close enough to the code to know what are the things that are slowing us down tactically.

Matthew Fowles Kulukundis: Yes..

Titus Winters: They don't put out like, "Hey, team. I need you to go fix your flaky tests. Go update this dependency, whatever." Like, those things are not going to happen. And the TLs, the team leads, tend to be a little nervous because we don't entirely tell them what they have authority and control over when they take on the TL role. So, they focus on good feature stuff. It's important. That's how you bring value. But I think that maybe the missing piece is we get the directors to set continuous delivery targets, and quality targets. Then you get the results that they do care about without them having to know the details. And the team leads have the permission to tackle the tactical things that they know are bothering them. I think that would be a great result.

Matthew Fowles Kulukundis: I think that's fascinating. You mentioned documentation. And I'm going to take that as a moment to shift about...you know, some of CD...like, you mentioned a culture of excellence. I'm going to take that moment to shift to the topic of culture. Like all good engineers, I've accidentally skipped culture. And the first, I don't know, like five chapters of the book are just about these annoying, squishy culture things.

Titus Winters: We have known for at least 50 years that software engineering is about two things. It's time and people. There's a Dave Parnis quote that I refer to far too often. It's probably David Lorge Parnas, that software engineering is the multi-person construction of multi-version programs, right? Going back to the early '70s, we know that it's about time and people. It's obvious from the start that you can't ignore the programming aspect of all of this. And unfortunately, we train all of our undergrads effectively in individual programming tasks, right? To get a grade, you're taught individually, and to learn to program, you do that individually. Naturally, we're doing that. But that changes immediately upon actually joining the team. It's not an individual thing. It's a team sport. And nobody...I certainly wasn't told that and I was on uncertain terms when I joined any of the jobs that I've had.

When you recognize that..it is a productivity hindrance to be a jerk. It is a productivity boon in the long term to be helpful and mentor people and all of the above. I think those are essential.

Cheat sheet for building great teams

Matthew Fowles Kulukundis: So, I've been on a variety of teams. I've been on a handful of teams at Google, and I've been at some startups beforehand. At least one of those teams fall in your way, Titus, and it had just a palpably different culture than everywhere else I've been. Everyone was engaged and friendly and just, like, part of a team in a way that I've never had What's the cheat code? How do I build that?

Titus Winters: I don't have a cheat code for that. It does take time. It does take energy. It does take effort. It's an active process. One of the things...there's a handful of things that I think contribute to it. And there's certainly more on this in the leadership chapters in the book. One of the things that have worked for me is to be very upfront about no, I don't know. No, it's okay for me to be wrong. Like, I will be wrong on stuff and I rely on everyone...the more junior people included to tell me when I'm wrong or to, like, make me do the math.

When I was serving on the C++ standards committee, I was chairing the subcommittee for the design of the standard library. One of the things that I focused on there very aggressively was asking dumb questions. I refused to let the room start taking votes on design topics until I felt like everyone in the room had relearned the material sufficiently to take an informed vote.

And to do that, I had to ask a lot of dumb questions, some of which I was faking, some of which I wasn't because this is a very complicated domain, right? But when you model that it is okay to not know things and you see the flavor of those discussions to be kind of a teaching, educational bent, the individual discussion is probably going to get to a better result and you have invested in growth for everyone that is participating. That longer-term investment in growth through education and learning and discussion and it being okay to be wrong, that's essential, I think.

Matthew Fowles Kulukundis: There's something really interesting there because not only is it okay to be wrong but guarantees you will be wrong at some point or another. The thing that SRE has taught us in life is that 100% is not an SLO. Everything has an error function, including your correctness.

Titus Winters: Yes.

Matthew Fowles Kulukundis: And thus changes the necessary outcome of this, right? You need to be able to when you learn you were wrong, change.

Titus Winters: Yes. I think it's indicative of the major difference in understanding software engineering and project management that we've seen in the last, you know, 10, 20, 50 years. If you go back to Fred Brooks and the mythical management essays, he has this nonsense to us about the surgical team, right, where the whole notion of how you structure your teams and your projects are, well, you find the one dude that's, like, rockstar, amazing, 10X and you organize the whole rest of the team so that that person has most focused time, has the most focused time to, like, manage everything else. And that makes sense only in the domain where you're going to do exactly one project and you're trying to get that done as fast as possible. That's not how tech works, right? Like, your team is going to do multiple projects. People are going to shift around, right? It's a career event.

Matthew Fowles Kulukundis: Yes.

Titus Winters: It's the difference between prisoner's dilemma and iterating prisoner's dilemma.

Matthew Fowles Kulukundis: Moving software engineering from prisoner's dilemma to iterated prisoner's dilemma.

Titus Winters: But so, the model that I think is much better to espouse is it's not the best person should do this task. It's the most junior person that can do this with a moderate amount of oversight, right, because that's when it's going to get done and the team is going to grow. The people that might have been the only ones that could do it on paper have space to tackle the next impossible thing.

Matthew Fowles Kulukundis: I think this also came out of an era...like, back in the '60s or '70s, like, one person could hold most of the components of a system in their head. I always tell new people in teams that I'm on like, "In the past, one person could write software and work...we're out of that era." All the software that could feasibly be written by a single person has been written.

Titus Winters: Yes. Strongly agree.

Where can Google improve from a culture’s perspective?

Matthew Fowles Kulukundis: One of the things, we've talked about is culture a little bit but you started to answer that culture is one of the spots where Google has most improved. So, I want to go back. Where in this cultural world does Google have the most to improve?

Titus Winters: I think it's all over. There are pieces in the culture chapters that focus on just how to be on a team which is very near and dear to my heart. I care a lot about the gap between undergraduate education and then what you do as a new hire in the industry. And there's a lot that we're just not able to cover. I don't even blame the academic world. It's just not sensible to try to cover that material in a classroom setting. But there's a lot of table stakes sort of activity that is just missing from that. Your job needs to ask a lot of questions. I was doing a guest lecture for a software engineering class yesterday and this came up. And my point was as an undergrad, you spend a week or two on an assignment and if you're super stumped, you wait a day or two and you go to office hours and you get a question answered.

Most of the time your code is unwitnessed by anybody. And then when you join a big company, you're trying to be successful here, you are valuable specifically because you are fresh eyes, right?

Matthew Fowles Kulukundis: We call that the golden idiot, at the startup I worked at.

Titus Winters: You have a fresh perspective. You are not cursed by the curse of knowledge, right? And so, when the team has their playbook, their onboarding, how does the system work, right? You are the perfect test case for all of that. You need to get out of the habit of waiting until it's perfect before you get any feedback, everything that is causing you any stumbling is useful feedback to the rest of the team like, "Hey, this doesn't make sense yet."

Matthew Fowles Kulukundis: Yes.

Titus Winters: So even starting from how to be on a team, like, we could be clearer about all of that and I wish that our onboarding was updated along those lines a little bit.

How to grow as a software developer?

Matthew Fowles Kulukundis: All right. So, this next question's kind of a bank shot. It's just to watch your face as I do this. A thing you told me a long time ago was you have to put down the rubber ducky to learn to play the saxophone. Can you first describe what that's a reference to?

Titus Winters: I have no idea what you're talking about. This doesn't even sound like it's a phrase of mine.

Matthew Fowles Kulukundis: It is. And you've made me describe it. It's Bert or Ernie, I forget who, who wants to learn to play the saxophone but he refuses to put down his rubber duck so he can't do the fingering on the saxophone. And there's a whole number about it. I swear to God you were the person who told me this.

Titus Winters: It's vaguely plausible but I don't think so.

Matthew Fowles Kulukundis: Seach for it on YouTube. Anyway, it was a powerful allegory of, to grow into something harder, you have to put down the things that make you feel comfortable.

Titus Winters: Yes.

Matthew Fowles Kulukundis: And I feel like the chapter on leadership sort of tries to hit that with always be leaving but isn't quite evocative enough in doing it.

Titus Winters: I could agree with that. I think it is very easy to take a blanket statement like always leaving as just having an exit strategy, like, having a plan to leave before your mistakes catch up with you. And that shouldn't be the goal, right? Putting down the rubber ducky is kind of a better metaphor in that sense, Sesame Street unpacking notwithstanding.

You need to leave room for others to be growing. And to do that, you're probably going to have to go tackle something that you can't yet. And those exits are scary. But yeah.

Matthew Fowles Kulukundis: I think the antipattern I've seen from some leaders that I'm not going to name at this moment is that they try to...they leave the project but they try to puppet it from beyond the grave or beyond leaving it.

Titus Winters: Right, right.

Matthew Fowles Kulukundis: I always tell the people that if someone is making an incorrect choice, to overrule them, you need the cost of the mistake to be higher than the cost of the lessons and the blow to that person of being overruled.

Titus Winters: Yes. Absolutely. I have said many times I regard, "Because I said so," or, like, "I overrule you," or disagree and commit, those are failures in leadership in one way or another. I'm not going to argue that we have to have a completely consensus-driven decision-making culture, etc., right? There are times when you are the owner, authority, decisionmaker, etc., and having unanimity amongst the governed is not the right choice, right. That's not going to help. But if there is meaningful contention between, you as a team lead and someone that's designing a new system or implementing a new thing, just falling back on because I'm in charge, you need to do it this way, that's a total failure. And that's related to all of that specifically because if you've delegated, you better delegate, right? You have to trust that other people will be capable of doing the thing.

And if they're not capable of it because they're not...they're missing a piece, explain it. You should be able to lead them to that change in decision-making. But putting your foot down is occasionally necessary. It's a power that you want to keep in your back pocket but it should be avoided strongly.

Matthew Fowles Kulukundis: There is a mistake I've made where, "Oh, I have a junior engineer who's hungry to grow and I give them a project that's slightly too big. Then I look away and six months later, they're still just, like, drowning." Like, "Oh, God. What have I done?" How do you balance that tension?

Titus Winters: I think you need to...it goes back to the psychological safety pieces that we were talking about earlier. You need to make it clear that asking questions and asking for help and being wrong is acceptable and encouraged. If you have that as a basis, then it is much more palatable to do the leadership thing and check in like, "Hey, how is this going? Do you need anything? How can I help, etc., right?" Try to make that not an I'm looking over your shoulder but a resource to be relied on. And even if you just have to fall back to can you walk me through it, they rubber duck it...to use a different form of the rubber duck analogy.

Matthew Fowles Kulukundis: It's so versatile.

Titus Winters: Just...it's important. If you even just get them to explain what is going on, what is bothering them, what the current holdups are with enough depth, sometimes just delivering those updates triggers, "Oh, I need to be doing this instead."

What chapter is missing from the book?

Matthew Fowles Kulukundis: You've mentioned being wrong and correcting. Unfortunately, the book is now published book. It is a dead tree.

Titus Winters: True.

Matthew Fowles Kulukundis: Hopefully there'll be another version where you can correct some of the things and add or improve some other things. Like, what chapter do you think is missing entirely?

Titus Winters: The three that were proposed that were interesting that we never quite got traction on, there was at one point likely going to be a chapter on application frameworks and their role in productivity, efficiency, etc. It's a little bit of a fraught subject.

Matthew Fowles Kulukundis: When you say application frameworks, do you mean, Ruby on Rails level of scaffolding around your project?

Titus Winters: Yeah, effectively. Like, For the frameworks where most of the serving machinery is sort of handled centrally and behind the scenes, you don't need to touch all of the fiddly bits. You could but you probably shouldn't. And the inversion of control is kind of an important part of that, right? You're not controlling main. You are defining a handler for some incoming RPC request or web request or whatever...whatnot. and those increasingly I see as a very useful mechanism to take away the sort of fiddly control that most teams probably don't have the expertise to be working with and certainly aren't gonna keep actively maintained.

Matthew Fowles Kulukundis: This is the popcorn button.

Titus Winters: This is...yes, this is the popcorn button to go to...

Matthew Fowles Kulukundis: Now that I've said that, can you expand on what it means?

Titus Winters: Yeah. There's a talk I did at CPP Con last year on configuration and extensibility and how that runs up against maintenance concerns. And in there we talk about configuration based on intent is very maintainable and very powerful. If you say, "Here is my system. I need to prioritize keeping memory cost down." That is a much easier thing to work with over a longer period than saying how big is this buffer. I don't know if you chose that buffer size intentionally or just randomly.

Titus Winters: Yes. One of the chapters that got cut was the application framework thing. And I think that there's a lot of insight to be extracted there. I think that we could go back and revitalize that. There was an attempt to have a chapter on issue tracking and it turned out we couldn't find anyone to say anything intelligent about it.

Matthew Fowles Kulukundis: That's because all of them are bad. Issue tracking is, like, Wikis. It's just where information goes to die.

Titus Winters: That's currently...and I believe that there are people that have a philosophy around this that is useful and actionable and makes sense. And none of them wanted to write a chapter.

Matthew Fowles Kulukundis: I just don't believe you.

Titus Winters: I might be optimistic but yeah. And then the last...there was some discussion about design but I think fundamentally design is such a huge topic that it probably deserves to be its book. I am not volunteering for that.

Matthew Fowles Kulukundis: I've been told that I have pretty good design intuition and I've tried to articulate it and I just can't, right? I end up chatting with someone earlier today being like, "Man, I wish I didn't sound like such a bad person." And I was like, "Oh, you should decouple these concerns except when I tell you that you should couple them."

Titus Winters: I have similarly questioned whether InDesign can be taught but recently I've been spending more sort of classroom-style time with George Fairbanks who has books on designing just enough software design, just enough software architecture. I'm missing the title all of a sudden. There are aspects of this that I think are very teachable. And there are concepts not about how to make an individual design decision but how to think about making design decisions that I think are transmissible.

Matthew Fowles Kulukundis: Yes

Matthew Fowles Kulukundis: One of the big lessons I've learned in the design of my career from... even with Sanjay has been very much, like, if presented two choices and one of them is very easy to undo and one of them is very hard to undo, make them easy to undo choice.

Titus Winters: Yes, yes, yes. In chapter 1 we talk about sustainability. Over the expected lifespan of your code, you need to be capable of changing everything important that comes along. You don't have to do it but you do need to be capable. And when we make decisions that can't be undone, oh, dear Lord, that is the stressor.

What decisions are hard to undo?

Matthew Fowles Kulukundis: So, what decisions can't be undone? Or are disproportionally expensive to undo?

Titus Winters: Don't ship things without version numbers. It's a design thing, it's a maintenance thing. Like, just put a version number on it. Life is so much better.

Matthew Fowles Kulukundis: I'm the lead Protobuf which for people who don't know is the serialization format that everything at Google's kept in. When the world gets stored on disk, your life just gets so many orders of magnitude harder. Even across the network adds one layer of pain. Disks are forever.

Titus Winters: Unbounded time skew. That's a terrifying thought. There's an old file storage format that our team used to maintain. I think we've both wandered off a little bit. Where one of the scariest things that I've seen is a comment from circa 2008 I think that's doing checksum validation. This is just a serialization format. It's not a big deal. But there is the thing in there that says, "If the checksum validates or it's this weird bit pattern, then accept the checksum." And it has a comment on it saying, "For four days back in 2005 or whatever...whenever it was. There was a bug and we were always writing this bit pattern and we have no idea if all of those records have been, like, expunged." So that extra branch will be there forever.

Matthew Fowles Kulukundis: Why? Can't we add monitoring for that branch path?

Titus Winters: We could. It's probably not worth it. The maintenance cost and the, you know...it is technically feasible to undo this. It's really hard.

Matthew Fowles Kulukundis: Fair enough. So, rubber ducks have come up a fair amount. And it seems like you have the chance for the rubber duck to be the logo. Why a flamingo?

Titus Winters: Amusingly, the behavior these days...I think are right...just their editors pick a thing. I had no notion that flamingo was going to be it until I saw a completed book cover. I think Tom Manshreck might have had some advanced warning. He was working more closely with the publisher. But as near as I can tell, we had little to no input on the choice of animal.

Matthew Fowles Kulukundis: So I also have to ask, Titus Winters, Tom Manshreck, Hyrum, do you just pick the curators for the name impact? Like, those are some hefty names.

Titus Winters: Our parents, thank you for the feedback, I guess.

Matthew Fowles Kulukundis: There's no, like, Rob Smith on here.

Titus Winters: No. No. This is a bunch of fairly recognizable names. We all cut our teeth on C++ core libraries early days. Like, in 2011 we were all working together. We've spread out over the years but one of the major reasons that I was willing to take the spicy approach that, hey, I think we're qualified to write this, was specifically that we had seen everyone else's code by maintaining very low-level stuff and having to refactor everyone else's code as we were trying to clean things up.

Matthew Fowles Kulukundis: I wish we had more time actually because refactoring in LSC is the part that I find most fascinating in this book and I think is the part that is most foreign to other people.

Titus Winters: That is certainly the feedback that we've gotten is the...a lot of it is, "Hey, that's a lot of overhead. And hey, it sure does sound nice that you have all of these tests but that seems like a lot of investment." And then you go to the chapter on large-scale changes in particular and you're like, "Oh, wait, what? You can do what now?"

Matthew Fowles Kulukundis: Yes..

Titus Winters: And that is just such a mind-blowing...like, oh, all of the rest of this serves for that end.

Matthew Fowles Kulukundis: Right.

Titus Winters: We need the capability of making changes to everything. And fundamentally, most of the rest of the book is building the capacity to do that.

Matthew Fowles Kulukundis: That's where I came up under you very heavily in the LSC space and to a large extent. It's just the water we swim in. I've almost forgotten to think about it. But I see echoes of this in rough editions or whatever they're calling it. I want to live in a world where open-source things also do this.

Titus Winters: That is a tall order.

Matthew Fowles Kulukundis: Can you get off of your laurels and do it for me?

Titus Winters: No. And for several reasons. One, there's no coordination of course. And so open source is a loose federation of projects. But I also discussed trying to build large-scale changes for the Python packages on GitHub or some such. So, it was probably three or four years ago. And my questions to the person that was trying to do this were of the form, like, is there enough build system consistency, is there enough test consistency to know that, like, your patches apply cleanly and, the code is still working? And they said, "No, no, no. You don't need that. You just apply the change. You send the pull request and you apply the change. Most of the packages are broken ahead anyway." And I was like, "Okay. Yeah, I have no idea how to start such a task if that's the environment that we're living in."

Matthew Fowles Kulukundis: You could take a page out of my book. Like, the migration across Google to just throw the changes out there, and whatever sticks, sticks. And you just keep throwing it until you've got coverage.

Titus Winters: Right. But, amazingly, you made that work.

Matthew Fowles Kulukundis: So, given that the flamingo is an accident, given flamingo is not a thing you control, I guess the infinity sign on it is not this deep seeded homage to the future in change?

Titus Winters: Nope. It's certainly not—just a happy accident.

Matthew Fowles Kulukundis: I was hoping it's like the arrow in the FedEx logo where, like, some 12th-dimensional genius put that in there to subconsciously influence everyone.

Titus Winters: Nope. Nope.

3 Book recommendations

Matthew Fowles Kulukundis: All right. Well, I'm gonna close this interview by stealing Ezra Klein's final question because why would I come up with my ideas when I can steal other people's? So, what are three books that have influenced the way you think? And you can't name your own.

Titus Winters:  I think looking at my bookshelf when you foreshadowed that you would probably ask this, I... the first thing that I would surface is "Just Work" by Kim Scott which is probably the biggest shift in being able to explain my thoughts and position on equity and inclusion and the value of diversity and, like, please just be cool, be nice. Don't, you know, harass people. We just want to get the work done. And when those things go wrong, what do you do, right? And I thought that that was great. I don't know if it influenced the way that I think but I...it influenced a lot of what I think about. "Accelerate" by Gene Kim and Nicole Forsgren, the DORA research.

Matthew Fowles Kulukundis: Yes, you mentioned them.

Titus Winters: It's five years old at this point. I think that the studies have only made those results more extreme but as a citation, boy, I go to that all the time. That's just completely valuable. And then I guess philosophically, there's some Douglas Hofstadter in my background, some Gödel, Escher, Bach. But maybe more specifically his book on the translation I think kind of...the meaning of meaning is big.

Matthew Fowles Kulukundis: In the space of philosophy you should check out Alasdair MacIntyre's "After Virtue". I think it interacts tightly with culture and how the necessity of communication forces morality.

Titus Winters: All right.

Matthew Fowles Kulukundis: And I love to squeeze it in because I've personally gotten a lot of fiction recommendations from you that I've enjoyed. Can you name a fiction book?

Titus Winters: "The Fifth Season" by NK Jemisin, the whole "Dandelion Dynasty" series from Ken Liu is just rocking my world. I'm finishing book four right now. And I'm a big fan of...let's go with the "Wayfarers" series. It starts with "The Long Way to A Small, Angry Planet." I love the expansion of sci-fi into being, like, more humanist. It's just like feelings and people are good.

Outro

Matthew Fowles Kulukundis: Awesome. Well, thanks. This has been fun. I had fun.