Mailbag: What isn't measurable? To hire as exec or not? @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.


Posts from this week:

- Mailbag: What isn't measurable? To hire as exec or not?


Mailbag: What isn't measurable? To hire as exec or not?

This is a mailbag of several different questions I’ve been asked over the past few weeks.

What isn’t measurable?

A friend of mine is interviewing for engineering leadership roles and brought an interesting question into our group chat from one of their interviews, “What are outcomes that you can’t measure with metrics?” For the record, I’m skeptical this is a good interview question, my most generous read is that it’s trying to detect pragmatic, experienced leadership because experienced leaders inevitably should have encountered things that matter but are hard to measure. I’d also presume that the follow up question is pushing the candidate to offer an approach to trying to measure the immeasurable topic.

My general thinking rules here are:

  1. Some present values can be measured directly, e.g. uptime, time to close tickets over past week
  2. All present values can be measured indirectly through a proxy measurement (see also Forecasting synthetic metrics) , e.g. team morale can be approximated through employee retention, product quality can be approximated by number of support tickets
  3. You can forecast the future value of any present value, e.g. future time to close tickets, future employee retention

That said, if the goal of this kind of interview question is to have a discussion that simulates working together, then maybe a more interesting answer to this question is simply, “engineering velocity.” Almost every senior engineering leader is asked to measure engineering velocity on their team, and all answers to this question are deeply unsatisfying: features shipped, tickets closed, etc.

At Calm, I’ve focused on measuring engineering through the number of features shipped, maintaining an agreed upon target win/loss/neutral rate for experiments (e.g. roughly one-third in each bucket), and whether we made exactly one big technical investment per quarter. For many folks asking for an engineering velocity measurement, this will meet their criteria, but it’s really more of an investment thesis. Even if we shipped more features one quarter than another, I wouldn’t actually believe that our velocity had necessarily gone up, it’s more likely that the features themselves were smaller.

I generally think engineering velocity measures are only interesting for debugging a low velocity team, rather than particularly helpful in ensuring a team is high impact. What we really want to be measuring is “engineering impact.” I asked about measuring engineering impact on Twitter a while ago, and the answers were not too helpful. Generally folks recommend measuring engineering impact through the lens of business impact, which makes a lot of sense, but is a bit “measuring the output rather than the input” in my opinion. This certainly helps you grade the team’s impact, but it does less to help you become more impactful.

Should we hire this candidate as an executive?

A founder at an Series A startup asked me whether they should hire a product leader as an executive (e.g. VP Product) or should hire them into a less senior role (e.g. Director of Product). Interviewers were split on which was the right decision, but it was also a strong candidate that they didn’t want to lose.

Honestly, I don’t think I have the right answer here, but my default mental model here for hiring an executive at an early stage startup is roughly:

  1. Do they bring domain expertise to your field or are they uniquely skilled at an area that is essential for your success? Later stage executives need a broader skill set and consequently don’t have to go as deep on any given area (their teams can compensate for those gaps), but earlier stage executives need to bring some aspect of truly exceptional strength; hiring a generally good, later stage executive into an early stage role is rarely a good outcome for either the executive or the early-stage company
  2. Can they both do the team’s work themselves and scale to providing the structure/strategy for a team of folks to do the work beneath them in org structure? If not, they are likely to be the wrong person in the role if you are successful
  3. Have they managed and hired managers before? If not, they will have a challenging time growing their team if you are successful

If they meet those three criteria, then it makes sense for an early startup to hire them as an executive. If they don’t, I’d hire them in a less senior role to watch them work for a while. That said, the details will always depend on your specific company and the specific candidate. Most executives won’t land the transition from Series A to Series D, and a self-aware executive is self-aware of that! The risk of adding them as an executive is much lower if they’re able to have this kind of discussion with some humility.


That's all for now! Hope to hear your thoughts on Twitter at @lethain!


This email was sent to you
why did I get this?    unsubscribe from this list    update subscription preferences
Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA

Email Marketing Powered by Mailchimp

Key phrases

Older messages

Reminiscing: the retreat to comforting work. @ Irrational Exuberance

Wednesday, October 12, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Reminiscing: the retreat to

One-on-ones with executives. @ Irrational Exuberance

Wednesday, September 7, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - One-on-ones with executives. -

Bar raisers, hiring committees, and other complex ways to improve hiring quality. @ Irrational Exuberance

Wednesday, August 31, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Bar raisers, hiring committees,

How to estimate disk space. @ Irrational Exuberance

Thursday, July 14, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - How to estimate disk space. -

Downturn career decisions. @ Irrational Exuberance

Wednesday, July 6, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Downturn career decisions.

Morfternight #62: ChatGPT is everywhere.

Monday, December 5, 2022

The one that was co-written by a human and a computer. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Food for Agile Thought #371: User Story Guide, Not all Planning Is Helpful, ROI of Product Management

Sunday, December 4, 2022

Also: Product Culture at Etsy, Agile Fluency Model®, Working Backward, Feature Team to Product Team ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

The teammate “sweet spot”

Sunday, December 4, 2022

Feeling lucky to have found it multiple times… ‌ ‌ ‌ Hey! Brian Casel here. As a reminder, you're subscribed to this here newsletter where I share my notes and conversations about products,

Set your future self up for a win

Sunday, December 4, 2022

With our free Annual Review Workshop ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

FTI #337: Be careful.

Sunday, December 4, 2022

​ ​ ​ If you look for a reason why it can't work, you'll find it. If you look for a reason not to trust someone, you'll find it. If you look for reason you're not ready, you'll find

"Don't settle": Bloom interviews Nabeeda Bakali

Sunday, December 4, 2022

The Bloom Issue #107, Dec 4 ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Daily Boost: December 4, 2022

Sunday, December 4, 2022

​ ​ ​ Something to ponder: Just like gardening, websites depend on good "soil" to thrive. With all other things being equal, good soil equals success and poor soil results in failure.

073 Two arguments made by people who don't understand epistemology

Sunday, December 4, 2022

"Because it has always been done that way" and "Because I say so" ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Poison Peanuts.

Saturday, December 3, 2022

What gopher hunting can teach us about patience. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Are you at war (with yourself?)

Saturday, December 3, 2022

What to do about these broken agreements ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌