Reminiscing: the retreat to comforting work. @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.

Posts from this week:

- Reminiscing: the retreat to comforting work.
- Fewer, happier incident heroes.

Reminiscing: the retreat to comforting work.

In Work on what matters, I wrote about Hunter Walk’s idea of snacking: doing work that is easy to complete but low impact. The best story of my own snacking behaviors comes from my time at Stripe. I was focused on revamping the engineering organization’s approach to operating reliable software, and decided that it might also make sense to start an internal book club. It was, dear reader, not the right time to start a book club. Once you start looking for this behavior, it is everywhere, including on your weekly calendar. Snacking isn’t necessarily bad, a certain amount gives you energy to redeploy against more impactful tasks, but you do have to be careful to avoid overindulging.

Beyond snacking, which can be valuable when it helps you manage your energy levels, there is a similar pattern that happens when a business or individual goes through a difficult moment: under pressure, most people retreat to their area of highest perceived historical impact, even if it isn’t very relevant to today’s problems. Let’s call this reminiscing. When you see your very capable CEO start to micromanage words in a launch email, or your talented CTO start to give style feedback in code reviews, take it for what it’s worth: they’re reminiscing. If you spend the time to dig deeper, they’re almost certainly panicking about something entirely unrelated.

Some real examples from my experience (don’t try too hard to connect them to individuals, I can quickly think of multiple examples for each of these):

• “The systems I architected never had significant reliability issues.” Senior-most engineering leader drives top-down rearchitecture of the company’s software, often anchored in the design problems from their last in-the-details technical role rather than current needs (e.g. a Grand Migration). They’d be better served by addressing the cultural or skill-gaps culminating in the reliability problems instead of trying to solve it themselves
• “We need to take more risks in our work.” Founders feel trapped by slog of meeting financial projections, and want to reorganize company efforts towards increased innovation without connecting dots to how it will meet the financial projections. Typically this is a throwback to an earlier company phase that is a poor fit for the current phase
• “The team I hired was much stronger than this team.” After years of absence, a founder starts revamping the performance or hiring process to address a perceived gap in hiring but without the context of why the process has evolved the way that he has. They’d be better served by holding their managers accountable or empowering their People team

Each of these examples is tightly ingrained into the person’s identity about why they’re someone successful. You can help them recognize the misalignment with today’s needs, but real progress on this issue depends on their own self-awareness. It usually won’t go quickly, but it always gets resolved faster than you might expect, typically through growing self-awareness but rarely by abrupt departures.

Over the past few years, I’ve gotten much better at being mindful of my own snacking inclination. That urge to reorganize our engineering wiki when work gets difficult? Even without knowing the other work I could be doing, it’s easy to identify: that’s snacking, for sure. Reminiscing is much harder for me to identify in a vacuum, it is valuable work, and it’s sometimes very impactful, difficult work, it simply isn’t particularly valuable for you personally to be doing right now.

To catch my own reminiscing, I find I really need to write out my weekly priorities, and look at the ones that keep slipping. Why am I avoiding that work, and is it more important that what I’m doing instead? If a given task slips more than two weeks, then usually I’m avoiding it, and it’s time to figure out where I’m spending time reminiscing.

Fewer, happier incident heroes.

My wife was kind enough to review a piece I’m writing about incident response, and brought up a related topic that I couldn’t fit into that article but is infinitely fascinating: how should companies respond to a small group of engineers who become critical responders in almost all incidents?

This happens at many companies, usually along the lines of: a few long-tenured engineers, who happen to have the explicit access credentials to all key systems and implicit permission to use them, help respond to almost all incidents. Over time, these folks become increasingly load bearing, as few others acquire the knowledge, and access credentials, to respond when they’re not available. Fast forward to the future, and one of these key responders leaves the companies, which creates more load on the remaining key responders. More and more depart, and eventually the company has a painful era of relearning how to effectively respond to incidents.

Having seen this happen a lot, in most cases you can solve this through:

• You can usually immediately identify the individuals filling this role without doing any extended analysis
• Ask the frequent responders to be less responsive when they’re not explicitly on-call. They can still watch, but wait at least ten or twenty minutes before responding as long as an incident isn’t impacting users too severely
• Ask the folks on-call to be more studied in when they escalate to the key responders
• As part of incident response where key responders are pulled in, ask them to document how they responded and train the broader on-call rotation with that documentation
• For any tool or system where documentation can’t close the gap after a couple of training sessions, prioritize designing something that can be mitigated in simpler ways
• This behavior will spread, particularly with a subset of new hires who will replicate the behavior (show up in every incident) without the key responders’ knowledge to be valuable when they respond. These folks create a lot of noise in their emulation attempt, and you should give them feedback immediately. If you don’t it’s very hard to make progress on this isue
• Promote and reward senior folks who run on-call rotations that succeed without their presence. Stop rewarding senior folks who have to personally show up to lead incident response
• Educate other leaders within your company to similarly reward folks for building resilient on-call rotations rather than heroics

Nine times out of ten, that should be able to resolve the issues.

Sometimes you’ll run into a values misalignment with other leaders, and in that case progress is going to be slow and painful. The very easy but very painful path is to just wait until a few key responders quit, which will prove your point. The harder but less painful path is to build data to support your case. For example, automatically record who is active in which incidents (e.g. logging incident response channels in Slack/Teams/whatever), correlate that with user impact / mean-time-to-mitigation / etc, and build a case that overreliance is making you less effective when they aren’t present, e.g. this approach is generating uncompensated corporate risk. It’s slow, but the data will always teach you something interesting, although it’ll rarely teach you what you initially anticipate.0

That's all for now! Hope to hear your thoughts on Twitter at @lethain!

This email was sent to you why did I get this?    unsubscribe from this list    update subscription preferences Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA