Should you prioritize infrastructure costs? @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.

Posts from this week:

- Should you prioritize infrastructure costs?
- What is innovation?

Should you prioritize infrastructure costs?

This is an exerpt from Infrastructure Engineer’s section on efficiency.

Before diving into the mechanics of managing infrastructure costs, the first question to answer is whether it’s a valuable use of organizational time to make your current infrastructure spend more efficient. How you think about this will vary a bit depending on whether your company is early-stage, prioritizing growth or profitability.

Early-Stage

Generally speaking, very early-stage companies shouldn’t spend much time thinking about infrastructure costs. You should instead be focused on finding product-market fit for your first product.

Here are two checks you can run to determine if it’s worth reducing your infrastructure costs:

• If you were to reduce your infrastructure costs to $0, and it still doesn’t increase your runway by at least two months, then it’s not worth focusing on
• If you’re spending less than $2,000/month per employee on infrastructure costs, then it’s probably not a significant priority because your headcount spend will be so much higher

If you’re not violating either of those checks, then keep on ignoring infrastructure spend. If you are exceeding one, and infrastructure costs are a significant part of your overall burn, then invest a sprint into reducing spend, and then stop ignoring it once these checks resume passing.

The one notable exception is if you’re building a low-margin product or product where cost efficiency is a pillar of your long-term strategy. For example, if you’re operating a metrics collection and dashboarding product like Datadog, then efficiency probably is worth considering earlier than usual.

Growth

When you’re prioritizing growth, the primary focus of the engineering organization in a technology company is creating, operating and advancing the products that support the business. Managing costs is important, but even immaculate cost management won’t make your company a success if enough energy isn’t being invested in your product.

The fundamental question to ask is whether infrastructure’s share cost of goods sold (COGS) are increasing as a percentage of revenue? (The simplest way to think COGS is all your non-headcount costs, although a slightly better definition would be all costs to operate

Start answering this question by plotting revenue and infrastructure costs on a chart to get a sense of how these two numbers are moving. Although logarithmic scales often generate more confusion than they’re worth, in this case it’s usually the only way to see both lines closely enough to understand their slopes within a single chart. You particularly want to understand if either line has experienced an inflection over the past few quarters. If costs have started accelerating without corresponding acceleration of revenue, that’s worth digging into.

Once you’ve looked at the two lines independently to understand their movement, simplify your first chart into a chart showing infrastructure costs as a percentage of revenue. This chart hides some detail but is easier to parse for folks further away from the details. As long as the ratio is going down and you’re company is focused on growth, then this data should be sufficient to justify your current level of investment into efficiency: if growth is key, and infrastructure costs are not getting in the way, why should you slow down growth to reduce them?

Profitability

Even the best business lines stop growing at some point. Facebook is one of the most valuable businesses in the world, but even they at some point ran out of new users to attract to their platform. Once growth slows, a business naturally starts focusing more on costs, including infrastructure spend.

In those scenarios, the easiest approach is to work with the business to align on two numbers:

• Dollars spent on infrastructure overhead per engineer: this includes things like development environments, testing tools, and so on. Determine your starting point by bucketing vendors and non-production infrastructure costs into a chart and plotting them over time divided by headcount. Pick a reasonable point on that line as your target. Refine it by reaching out to industry peers to get a sense of how this number compares to theirs (be sure to pick industry peers in companies that are currently focused on profitability, otherwise their answers won’t be very helpful to you)
• Infrastructure dollars spent per N product operations served: anchoring on cost of operating the product. This will vary a bit depending on your product or business, but it might be “$1.00 in infrastructure costs to powering every 10,000 searches”, “$2.50 for every 10,000 payments processed”, or “$3.00 for every 10,000 trips scheduled”

In both, the key thing is moving away from anchoring on a percentage of revenue and instead setting a target against the fundamental operations that you support. Thinking of costs as a percentage of revenue works well when you’re growing, but is too abstract and hides too many details once you’re focused on reducing costs.

If you find yourself exceeding those targets, then it’s time to dive into reducing them.

What is innovation?

At a recent Engineering Q&A session, I spoke a bit about the idea of an innovation budget. We’ll accomplish more in the long-term if we protect some space for innovation in addition to maintaining our focus on immediate goals. That sounds great, someone noted in chat, but what did I even mean when I kept talking about innovation?

There are a lot of ways to think about innovation, but for me I always start with the idea of a hill climbing algorithm. You’re at a point in a two dimensional space, each point in that space has a value, you want to be at the point with the highest value, but you can only see the values of adjacent points. The simplest solution is to look at the surrounding points, and move to the one with the highest value. Do this over and over, and you’ll successfully climb the closest hill. That’s delightful, but at the top a new problem arises: you can’t climb any higher, but there are probably higher points out there somewhere, you just have no idea where.

Innovation is figuring out what to do when you’re at the top of a hill.

Well, that’s the simplified, inspirational version. A more useful version recognizes that moving between some points is more expensive than others. When you’re climbing a shallow slope, it’s pretty easy to keep going: all you need is a good pair of hiking boots. As the hill gets steeper, each step takes more energy, and you might need rope or climbing shoes.

This brings us to the two fundamental innovation strategies:

1. Explore around you for new hills to climb
2. Optimize how you climb your current hill

Explore around you for new hills

Clayton Christensen’s The Innovator’s Dilemma tries to explain why innovative companies rarely have a second, equally innovative act. The thesis is that most businesses are structured such that they prioritize investing into existing, successful businesses at the expense of novel, unproven endeavors. The problem isn’t that these successful companies are poorly run, but rather it’s being well run that prevents companies from further innovation.

Mihaly Csikszentmihalyi’s Creativity looks at innovation through a different lens. Two key components of innovations are, one, combining learnings across multiple fields (e.g. physics and chemistry) and, two, unstructured time for thinking. Few modern workplaces create space for either of those components, let alone both.

That said, innovation clearly is possible within the modern workplace, and it’s worth thinking about our best tools for enabling innovation:

• Customer-orientation: how can we create more long-term value by prioritizing customer need over short-term financial outcome?
• Data-driven thinking: how do we use data to drive objective learning about our area and users?
• Product-thinking: how can we enable innovation by aiming and incentivizing teams properly? (Particularly love Melissa Perri’s Escaping The Build Trap on this topic.)
• User-research and design thinking: how do we understand and operate from real customer behavior and needs rather than from our assumptions about their behavior and needs?

That said, the tool I’ve found most effective for enabling exploration is a simple investment thesis. Agree as a leadership team that 20% of each team’s time should be prioritized against innovation, and make sure at least one executive is measuring and fighting to preserve that 20% as you grow.

(20% is, of course, just a made up number.)

Optimize how you climb your current hill

In Jeff Besos’ 2011 letter to Amazon shareholders, he talked about driving innovation by eliminating gatekeepers:

I am emphasizing the self-service nature of these platforms because it’s important for a reason I think is somewhat non-obvious: even well-meaning gatekeepers slow innovation. When a platform is self-service, even the improbable ideas can get tried, because there’s no expert gatekeeper ready to say “that will never work!” And guess what – many of those improbable ideas do work, and society is the beneficiary of that diversity.

Broadening the idea, there are huge opportunities for innovation that are expensive to access but not inherently expensive. A/B tests are valuable because they allow you to know with confidence that one approach outperforms another, but they’re at least equally valuable because they derisk execution. With an effective experimentation platform you can safely allow teams to launch ideas that you’re not sure will succeed. By reducing risk, the untenable becomes acceptable.

There are even cases where process prevents obviously good ideas from moving forward. For example, when I joined Uber it was a multi-day task to provision a new service in our service oriented architecture. This meant that many folks who wanted new services simply weren’t able to get them provisioned. The move to self-service pulled the infrastructure team out of the decision loop: if a team believed a new service was necessary to make progress, they could take on the provisioning cost even if the infrastructure team had other, higher priorities.

This model is focused on maintaining innovation within an existing business rather than the creative innovation fostered within a new business. There’s a lot more to say about innovation beyond this one mental model, but hopefully it provides some food for thought.

That's all for now! Hope to hear your thoughts on Twitter at @lethain!

This email was sent to you why did I get this?    unsubscribe from this list    update subscription preferences Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA