Physics and perception. @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.

Posts from this week:

- Physics and perception.

Physics and perception.

At one point in 2019, several parts of Stripe’s engineering organization were going through a polite civil war. The conflict was driven by one group’s belief that Java should replace Ruby. Java would, they posited, address the ongoing challenge of delivering a quality platform in the face of both a rapidly growing business and a rapidly growing engineering organization. The other group believed Stripe’s problems were driven by a product domain with high essential complexity and numerous, demanding external partners ranging from users to financial institutions to governments; switching programming languages wouldn’t address any of those issues. I co-wrote the internal version of Magnitudes of exploration in an attempt to find a useful framework for navigating that debate, but nonetheless the two groups struggled to make much progress in understanding one another.

I was reminded of those discussions while reading Steven Sinofsky’s Hardcore Software’s chapter on Innovation versus Shipping: The Cairo Project:

Landing on my desk early in 1993 was the first of many drafts of Cairo plans and documents. Cairo took the maturity of the NT product process—heavy on documentation and architectural planning—and amped it up. Like a well-oiled machine, the Cairo team was in short order producing reams of documents assembled into three-inch binders detailing all the initiatives of the product. Whenever I would meet with people from Cairo, they would exude confidence in planning and their processes. … While any observer should have rightfully taken the abundance of documentation and confidence of the team as a positive sign, the lack of working code and ever-expanding product definition seemed to set off some minor alarms, especially with the Apps person in me. While the Cairo product had the appearance of the NT project in documentation, it seemed to lack the daily rigorous builds, ongoing performance and benchmarking, and quality and compatibility testing. There was a more insidious dynamic, and one that would prove a caution to many future products across the company but operating systems in particular.

The simple narrative regarding both the Cairo development and Java migration is that there’s a group doing the “right” thing, and another group doing the “wrong” thing. The Cairo team was shipping vaporware. The Java team was incorrectly diagnosing the underlying problems. These sorts of descriptions are comforting because they create the familiar narrative structure of “good” in conflict with “evil.” Unfortunately, I’ve never found these sorts of narratives very useful for understanding what causes a conflict, and they’re worse than useless at actually resolving conflicts.

What I have found useful is studying what each faction knows that the other doesn’t, and trying to understand those gaps deeply enough to find a solution. Sometimes I summarize this as " solving for both physics and perception."

Solving for perception

Sinofsky’s represents Cairo as an impossibly broad project that didn’t ship, but he also explains why it picked up so many features:

Cairo tended to take this as a challenge to incorporate more and more capabilities. New things that would come along would be quickly added to the list of potential features in the product. Worse, something that BillG might see conceptually related, like an application from a third party for searching across all the files on your hard disk, might become a competitive feature to Cairo. Or more commonly “Can’t Cairo just do this with a little extra work?” and then that little extra work was part of the revised product plans.

It wasn’t ill-intentioned, rather they simply wanted to live up to their CEO’s expectations. They wanted to be perceived as succeeding within their company’s value system, because they correctly understood that their project would be canceled otherwise.

Many incoming leaders find themselves immediately stuck in similar circumstances. They’ve just joined and don’t understand the domain or team very well, but are being told they need to immediately make progress on a series of problems that have foiled the company’s efforts thus far. They know they need to appear to be doing something valuable, so they do anything that might look like progress. It’s particularly common for leaders to begin a Grand Migration at that moment, which they hope will solve the problems at hand, but no matter what will be perceived as a brave, audacious initiative.

This isn’t a problem unique to executives or product engineers, I frequently see platform teams make the same mistake when they undertake large-scale migrations. Many platform migrations are structured as an organizational program where a platform team tells product teams they need to complete a certain task (e.g. “move to our monorepo”) by a certain date, along with tracking dashboards that inform executives which teams have or haven’t completed their tasks. This does a great job of applying pressure to the underlying teams, and a good job of managing perceptions by appearing to push hard, but these migrations often fail because there’s little emphasis on the underlying ergonomics of the migration itself. If you tell teams they are failing if they miss a date, they will try to hit the date; if it’s hard, they’ll still fail. Platform teams in that case often blame the product teams for not prioritizing their initiative, when instead the platform teams should have the self-awareness to recognize that they made things difficult by not simplifying the underlying physics for the product teams they asked to migrate.

There’s nothing wrong about solving for perception, and indeed it’s a necessary scale to be an effective leader. Rather the lesson here is that most meaningful projects require solving for both perception and physics.

Solving for physics

When I joined Stripe, one of the first projects I wanted to take on was migrating to Kubernetes and away from hand-rolled tooling for managing VMs directly. This was heavily influenced by what I had learned migrating Uber from a monolithic Python application to polygot applications in a polyrepo. After a few months of trying to build alignment within engineering, I postponed the Kubernetes migration for a few years because I couldn’t convince them it solved a pressing problem. (I did come back to it, and it was a success when I did.) I could have forced the team to work on that project, but it goes against my instincts: generally when engineers push back on leadership ideas, there’s a good reason for doing so.

Similarly, my initial push at Stripe was not toward the Ruby typing work that became Sorbet, but rather to design an incremental migration towards an existing statically-typed language such as Java or Go. The argument I got back was that this was impractical because it required too large a migration effort, and that Facebook’s Hack had already proven out the viability of moving from PHP to a PHP-like typed language. I took my time to understand the pushback, and over time shifted my thinking to focus instead on sequencing these efforts: even if we wanted to move to a different language, first we needed to improve the architecture to support migrating modules, and that effort would benefit from typing Ruby.

I was fortunate in these cases, because there were few perceptions that I needed to solve for, and I was able to mostly focus on the physics. Indeed, the opportunity to focus on physics is one of the undervalued advantages of working within infrastructure engineering. You’ll rarely be lucky enough in senior leadership roles to focus on the physics.

For example, when I joined Carta, there was pressure across the industry and internally to increase our investment into using LLMs. Most engineers were quite skeptical of the opportunity to use LLMs, so if I’d listened exclusively to the physics, I would have probably ignored the pressure to adopt. However, that would have led me astray in two ways. First, I would have seriously damaged the wider executive team’s belief in my ability to incorporate new ideas. Second, physics are anchored on how we understand the world today, and LLMs are a place where things are evolving quickly. Our approach to using LLMs in our product is better than anything we would have gotten to by only solving for physics. (And vastly better than we’d have come up with if we’d only solved for perception.)

I think the LLM example is instructive because it violates the expectation that “physics” are real and “perceptions” are false. It can go both ways, depending on the circumstances. As soon as you get complacent about your perspective representing reality, you’ll quickly be disabused of that notion.

Balancing physics and perception

Effective leaders meld perception and physics into approaches that solve both. This is hard to do, takes a lot of energy, and when done well often doesn’t even look like you’re doing that much. Many leaders try to solve both, but eventually give in to the siren’s song of applying perception pressure without a point of view on how that pressure should be channeled into a physical plan. Applying pressure without a plan is the same issue as the infrastructure migration, where you can certainly create accountability, but it’s pretty likely to fail.

Pressure without a plan is appropriate at some level of seniority, and it’s important to understand within a given organization where responsibility lies for appending a plan to the pressure. In a small startup (10s of people), that’s probably the founders. In a medium-sized company (100s of people), that’s likely the executive team. As the company grows, more and more of the plan will be devised further from the physics, but you always have to decide where planning should start.

There is always a point where an organization will simply give up on planning and allow the pressure to cascade undeterred. In a high-functioning organization, that pressure point is quite high. In lower-functioning organizations, it will occur frequently even if there’s little pressure.

If you can reduce pressure too little, you can also reduce pressure too much. One of my biggest regrets from my time at Stripe is that I allowed too little pressure to hit my organization, which over time created a values oasis that operated with a clear plan but also limited pressure. When I left, the pressure regulator came off, and my organization had a rough patch learning to operate in the new circumstances.

Altogether, this balance is difficult to maintain. I’m still getting better at it slowly over time, learning mostly from mistakes. As a final thought here, respecting physics doesn’t necessarily mean doing what engineers want you to do: those who speak for physics aren’t necessarily right. Instead, it’s making a deliberate, calculated tradeoff between the two that’s appropriate to the circumstances. Sometime’s that courageously pushing back on an impossible timeline, sometimes it’s firing a leader who insists change is impossible.

That's all for now! Hope to hear your thoughts on Twitter at @lethain!

This email was sent to you why did I get this?    unsubscribe from this list    update subscription preferences Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA