Mailbag: What isn't measurable? To hire as exec or not? @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.


Posts from this week:

- Mailbag: What isn't measurable? To hire as exec or not?


Mailbag: What isn't measurable? To hire as exec or not?

This is a mailbag of several different questions I’ve been asked over the past few weeks.

What isn’t measurable?

A friend of mine is interviewing for engineering leadership roles and brought an interesting question into our group chat from one of their interviews, “What are outcomes that you can’t measure with metrics?” For the record, I’m skeptical this is a good interview question, my most generous read is that it’s trying to detect pragmatic, experienced leadership because experienced leaders inevitably should have encountered things that matter but are hard to measure. I’d also presume that the follow up question is pushing the candidate to offer an approach to trying to measure the immeasurable topic.

My general thinking rules here are:

  1. Some present values can be measured directly, e.g. uptime, time to close tickets over past week
  2. All present values can be measured indirectly through a proxy measurement (see also Forecasting synthetic metrics) , e.g. team morale can be approximated through employee retention, product quality can be approximated by number of support tickets
  3. You can forecast the future value of any present value, e.g. future time to close tickets, future employee retention

That said, if the goal of this kind of interview question is to have a discussion that simulates working together, then maybe a more interesting answer to this question is simply, “engineering velocity.” Almost every senior engineering leader is asked to measure engineering velocity on their team, and all answers to this question are deeply unsatisfying: features shipped, tickets closed, etc.

At Calm, I’ve focused on measuring engineering through the number of features shipped, maintaining an agreed upon target win/loss/neutral rate for experiments (e.g. roughly one-third in each bucket), and whether we made exactly one big technical investment per quarter. For many folks asking for an engineering velocity measurement, this will meet their criteria, but it’s really more of an investment thesis. Even if we shipped more features one quarter than another, I wouldn’t actually believe that our velocity had necessarily gone up, it’s more likely that the features themselves were smaller.

I generally think engineering velocity measures are only interesting for debugging a low velocity team, rather than particularly helpful in ensuring a team is high impact. What we really want to be measuring is “engineering impact.” I asked about measuring engineering impact on Twitter a while ago, and the answers were not too helpful. Generally folks recommend measuring engineering impact through the lens of business impact, which makes a lot of sense, but is a bit “measuring the output rather than the input” in my opinion. This certainly helps you grade the team’s impact, but it does less to help you become more impactful.

Should we hire this candidate as an executive?

A founder at an Series A startup asked me whether they should hire a product leader as an executive (e.g. VP Product) or should hire them into a less senior role (e.g. Director of Product). Interviewers were split on which was the right decision, but it was also a strong candidate that they didn’t want to lose.

Honestly, I don’t think I have the right answer here, but my default mental model here for hiring an executive at an early stage startup is roughly:

  1. Do they bring domain expertise to your field or are they uniquely skilled at an area that is essential for your success? Later stage executives need a broader skill set and consequently don’t have to go as deep on any given area (their teams can compensate for those gaps), but earlier stage executives need to bring some aspect of truly exceptional strength; hiring a generally good, later stage executive into an early stage role is rarely a good outcome for either the executive or the early-stage company
  2. Can they both do the team’s work themselves and scale to providing the structure/strategy for a team of folks to do the work beneath them in org structure? If not, they are likely to be the wrong person in the role if you are successful
  3. Have they managed and hired managers before? If not, they will have a challenging time growing their team if you are successful

If they meet those three criteria, then it makes sense for an early startup to hire them as an executive. If they don’t, I’d hire them in a less senior role to watch them work for a while. That said, the details will always depend on your specific company and the specific candidate. Most executives won’t land the transition from Series A to Series D, and a self-aware executive is self-aware of that! The risk of adding them as an executive is much lower if they’re able to have this kind of discussion with some humility.


That's all for now! Hope to hear your thoughts on Twitter at @lethain!


This email was sent to you
why did I get this?    unsubscribe from this list    update subscription preferences
Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA

Email Marketing Powered by Mailchimp

Key phrases

Older messages

Reminiscing: the retreat to comforting work. @ Irrational Exuberance

Wednesday, October 12, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Reminiscing: the retreat to

One-on-ones with executives. @ Irrational Exuberance

Wednesday, September 7, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - One-on-ones with executives. -

Bar raisers, hiring committees, and other complex ways to improve hiring quality. @ Irrational Exuberance

Wednesday, August 31, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Bar raisers, hiring committees,

How to estimate disk space. @ Irrational Exuberance

Thursday, July 14, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - How to estimate disk space. -

Downturn career decisions. @ Irrational Exuberance

Wednesday, July 6, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Downturn career decisions.

You Might Also Like

World Book Day Promo for Authors • Email Newsletter Included + Tweets + FB Posts

Friday, March 29, 2024

ContentMo's special promo 4/23/24 for World Book Day Join ContentMo's World Book Day Promotion #WorldBookDay is April 23rd each year. ContentMo is running a special promo on 4/23/24 for World

🎤 SWIPES Email (Friday March 29th, 2024)

Friday, March 29, 2024

The SWIPES Email ​ Edition: Friday, March 29th, 2024 ​An educational (and fun) email by Copywriting Course. Enjoy! ​ 🎤 Listen to this email here: ​ ​ ​ Swipe: This 1950's Schweppes Ad is still one

Taxing Time

Friday, March 29, 2024

Prepare your business to have a good relationship with the tax collector. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

👀 3 small biz ideas you’ve never thought about

Thursday, March 28, 2024

These unconventional Main Street biz's will raise your eyebrows... 3 small biz ideas you've probably never thought about Hey Contrarians, 🚨 Brace for impact 🚨 Our BigDeal podcast is coming to

3-2-1: On muddy puddles and leaky ceilings, the secret to productivity, and how to spoil a great relationship

Thursday, March 28, 2024

3 ideas, 2 quotes, and 1 question to consider this week. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

24 Hours Left: Grab Your Discounted CEX Ticket Now!

Thursday, March 28, 2024

Last call to save on your ticket - spring pricing goes away at the end of March ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Every cold outreach intro:

Thursday, March 28, 2024

I'm so impressed by what you are doing ‌ ‌ ‌ We have 4 updates for you this week: 1. On Peep's Mind Every cold outreach starts with a variation of "I'm so impressed by what you're

Here's What You Missed

Thursday, March 28, 2024

Last week, David hosted a live Q&A about all things writing and Write of Passage. Here are ten of the best questions we covered. Write of Passage logo transparent-1 The Write of Passage Bootcamp is

Do you know where your burnout is hiding?

Thursday, March 28, 2024

When “outer work” meets “inner work” ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Beard Dough

Thursday, March 28, 2024

When Russians paid the government for the right to have facial hair ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌