Mailbag: What isn't measurable? To hire as exec or not? @ Irrational Exuberance
Hi folks,
This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.
Posts from this week:
-
Mailbag: What isn't measurable? To hire as exec or not?
Mailbag: What isn't measurable? To hire as exec or not?
This is a mailbag of several different questions I’ve been asked over the past few weeks.
What isn’t measurable?
A friend of mine is interviewing for engineering leadership roles and brought an interesting question into our group chat from one of their interviews, “What are outcomes that you can’t measure with metrics?” For the record, I’m skeptical this is a good interview question, my most generous read is that it’s trying to detect pragmatic, experienced leadership because experienced leaders inevitably should have encountered things that matter but are hard to measure. I’d also presume that the follow up question is pushing the candidate to offer an approach to trying to measure the immeasurable topic.
My general thinking rules here are:
- Some present values can be measured directly, e.g. uptime, time to close tickets over past week
- All present values can be measured indirectly through a proxy measurement (see also Forecasting synthetic metrics) , e.g. team morale can be approximated through employee retention, product quality can be approximated by number of support tickets
- You can forecast the future value of any present value, e.g. future time to close tickets, future employee retention
That said, if the goal of this kind of interview question is to have a discussion that simulates working together, then maybe a more interesting answer to this question is simply, “engineering velocity.” Almost every senior engineering leader is asked to measure engineering velocity on their team, and all answers to this question are deeply unsatisfying: features shipped, tickets closed, etc.
At Calm, I’ve focused on measuring engineering through the number of features shipped, maintaining an agreed upon target win/loss/neutral rate for experiments (e.g. roughly one-third in each bucket), and whether we made exactly one big technical investment per quarter. For many folks asking for an engineering velocity measurement, this will meet their criteria, but it’s really more of an investment thesis. Even if we shipped more features one quarter than another, I wouldn’t actually believe that our velocity had necessarily gone up, it’s more likely that the features themselves were smaller.
I generally think engineering velocity measures are only interesting for debugging a low velocity team, rather than particularly helpful in ensuring a team is high impact. What we really want to be measuring is “engineering impact.” I asked about measuring engineering impact on Twitter a while ago, and the answers were not too helpful. Generally folks recommend measuring engineering impact through the lens of business impact, which makes a lot of sense, but is a bit “measuring the output rather than the input” in my opinion. This certainly helps you grade the team’s impact, but it does less to help you become more impactful.
Should we hire this candidate as an executive?
A founder at an Series A startup asked me whether they should hire a product leader as an executive (e.g. VP Product) or should hire them into a less senior role (e.g. Director of Product). Interviewers were split on which was the right decision, but it was also a strong candidate that they didn’t want to lose.
Honestly, I don’t think I have the right answer here, but my default mental model here for hiring an executive at an early stage startup is roughly:
- Do they bring domain expertise to your field or are they uniquely skilled at an area that is essential for your success? Later stage executives need a broader skill set and consequently don’t have to go as deep on any given area (their teams can compensate for those gaps), but earlier stage executives need to bring some aspect of truly exceptional strength; hiring a generally good, later stage executive into an early stage role is rarely a good outcome for either the executive or the early-stage company
- Can they both do the team’s work themselves and scale to providing the structure/strategy for a team of folks to do the work beneath them in org structure? If not, they are likely to be the wrong person in the role if you are successful
- Have they managed and hired managers before? If not, they will have a challenging time growing their team if you are successful
If they meet those three criteria, then it makes sense for an early startup to hire them as an executive. If they don’t, I’d hire them in a less senior role to watch them work for a while. That said, the details will always depend on your specific company and the specific candidate. Most executives won’t land the transition from Series A to Series D, and a self-aware executive is self-aware of that! The risk of adding them as an executive is much lower if they’re able to have this kind of discussion with some humility.
That's all for now! Hope to hear your thoughts on Twitter at @lethain!
|
Older messages
Reminiscing: the retreat to comforting work. @ Irrational Exuberance
Wednesday, October 12, 2022
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Reminiscing: the retreat to
One-on-ones with executives. @ Irrational Exuberance
Wednesday, September 7, 2022
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - One-on-ones with executives. -
Bar raisers, hiring committees, and other complex ways to improve hiring quality. @ Irrational Exuberance
Wednesday, August 31, 2022
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Bar raisers, hiring committees,
How to estimate disk space. @ Irrational Exuberance
Thursday, July 14, 2022
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - How to estimate disk space. -
Downturn career decisions. @ Irrational Exuberance
Wednesday, July 6, 2022
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Downturn career decisions.
You Might Also Like
Wardley mapping the LLM ecosystem. @ Irrational Exuberance
Wednesday, December 25, 2024
Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Wardley mapping the LLM ecosystem
🤝 Where the Magic Happens as a Biz Buyer
Tuesday, December 24, 2024
For Goodness' Sake, Know Why You're Doing It Biz Buyers, If you're new here, welcome to Main Street Minute — where we share some of the best tips, tools, and ideas from inside our Main
• New Year's Day Newsletter Promo for Authors ● Reserve Your Spot
Tuesday, December 24, 2024
Book Your Spot Now in Our New Year's Day Holiday Books Email Newsletter ! Book Your Spot in Our New Year's Day Email Newsletter Enable Images
24 SEO Tips for 2025 (Free PDF Inside)🎄
Tuesday, December 24, 2024
New year, new strategy – your guide to 2025 SEO success is here! Hi Reader, Merry Christmas! 🎅✨ I hope you're enjoying a cozy holiday season. As we look toward 2025, it's the perfect time to
Why The American Military Pretends to Track Santa Claus
Tuesday, December 24, 2024
They know when he's been sleeping, they know when he's awake.
• Affordable Kindle Unlimited Promos for Authors •
Tuesday, December 24, 2024
Affordable KU Book Promos "I'm amazed in this day and age there are still people around who treat you so kindly and go the extra mile when you need assistance. If you have any qualms about
Level up like you are using cheat codes 🎓
Monday, December 23, 2024
Become CXL certified ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Reminder - Our best price yet💥
Monday, December 23, 2024
Get 13 months for $99 per month ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Jólabókaflóð
Monday, December 23, 2024
It's Icelandic and it's awesome.
Kindle Unlimited & Paperback: YA & Literary Fiction for All Ages ~ Doll Face by Dianne Stadhams
Monday, December 23, 2024
Tiilly's view of complex issues goes viral. Welcome to ContentMo's Book of the Day "What a refreshing view of the world! Stadhams has crafted a plucky, resourceful heroine in Tilly