Bar raisers, hiring committees, and other complex ways to improve hiring quality. @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.


Posts from this week:

- Bar raisers, hiring committees, and other complex ways to improve hiring quality.


Bar raisers, hiring committees, and other complex ways to improve hiring quality.

When Uber Engineering reached 800 engineers, engineering was divided across roughly five engineering directors. Most engineering and process issues were resolved locally within these five organizations. This worked well for the most part, but meant there was little consistency within our core processes. Hiring was particularly fragmented, with every team’s manager responsible for developing their own interviews, questions and structure.

However, no matter which team initially hired an engineer, almost every team ended up working with that engineer, and over time some teams became increasingly frustrated by other teams’ perceived lax hiring standards. Eventually, three directors became so dissatisfied with a fourth’s hiring standards that they successfully argued for rolling out a “bar raiser” program, modeled after Amazon’s. In that program, at least one well-qualified interviewer outside the hiring organization would weigh in on every hire. In theory, the bar raiser was the final decider on whether a candidate could receive an offer, ensuring each hire met at least two organizations’ hiring standards. As a member of the initial bar raiser rollout, I can vouch that it addressed many of these issues, although it struggled to deal with hiring managers who escalated any bar raiser decision that contradicted their own.

When I later joined Stripe, I’d internalized a core lesson about bar raising programs: they are easy to silently subvert unless they have a great deal of ongoing, empowered organizational support. So I pushed back on introducing bar raising, and was glad to throw support behind instituting a hiring committee. Wouldn’t you know it, hiring committees introduce some challenges of their own.

From here: deciding if you should introduce a program to improve hiring quality, running a bar raising program, operating hiring committees, and finally my preferred approach of using structured approval.

Should you prioritize hiring quality?

A big part of running a successful engineering organization is avoiding as much process as you can, but no more. Three helpful questions to answer when you’re considering if you need to improve hiring quality are:

  1. Have you already created a structured hiring process and trained your team on using it? If not, go do that first.
  2. Do you have a general concern about hiring quality, or is it a narrow problem related to a small number of individuals? If it’s just a few folks, introduce a performance management system first, and use it to manage those individuals up or out.
  3. Are you still concerned about the quality of your hiring after ramping up on structured hiring and performance management systems?? Alright, let’s introduce a program to improve hiring quality!

Generally, you’ll find these processes are most valuable during periods of rapid hiring and especially when many hiring managers are new to your company. If you believe it’s necessary to introduce these practices during a period of low hiring or long-tenured hiring managers, my initial guess is that you have a performance problem within your team that needs to be addressed directly, and that introducing a process is simply a form of postponing dealing with that issue.

Bar raisers

Since Amazon introduced its bar raiser program, it’s been tweaked and copied by many companies. As mentioned in the opening paragraphs, Uber also adopted the bar raiser program in ~2015, which worked as follows: \

  1. Interview loops were generally designed by each teams’ engineering manager
  2. However, all loops must include a bar raiser as an interviewer. The bar raiser must come from a different Director’s reporting chain than the loop’s hiring engineering manager
  3. Bar raisers gave an interview of their choice, rather than an interview assigned by the loop’s hiring engineering manager
  4. Bar raisers were selected for strong interviewing skills, evidence of experience interviewing at Uber specifically, and were approved by the team leading the bar raiser program. Bar raisers went through a brief training program before starting bar raiser interviews

Introducing bar raisers was a simple change to the interviewing process itself, but nonetheless meaningfully changed Uber’s overall interviewing outcomes. The biggest operational challenge was at times invisible, and came down to resolving conflict between bar raisers, hiring managers, and hiring managers’ reporting chains. Without executive sponsorship supporting bar raisers’ decisions, the role’s impact is easily undermined by frequent overrulings from the hiring manager’s reporting chain.

Pros

  • Decentralized, scales effectively as number of interviews scales
  • Avoids slowing down process by introducing an additional approval step
  • Multiple potential bar raisers for any given interview loop mean that hiring quality remains relatively high even if a percentage of bar raisers are poorly calibrated

Cons

  • Susceptible to overriding escalations from hiring management chain, undermining impact and morale of bar raisers
  • Unclear incentives for the bar raisers themselves, given the work is often not valued in promotion process
  • Decentralized day-to-day operation can create ambiguous role for executive sponsor, which often translates into disengaged rather than accountable sponsorship

Inspection questions

  • What are your offer-extend and offer-accept rates?
  • How often do bar raisers change the outcome of an interview?
  • How many bar raisers do you have?
  • How many bar raisers have you added/lost in the past quarter?
  • What is the interview load on bar raisers?
  • How does the executive sponsor know if structure needs to be adjusted?

Hiring committees

Through 2016, engineering hiring at Stripe followed the “pool hiring” model: a candidate would enter the process, get interviewed by engineers and one manager drawn from across the organization, and then the manager and recruiter would synthesize the feedback discussion into a hiring decision during the interview debrief (for reasons lost to history, candidate debrief meetings were internally referred to as the “trope”). The “pool hiring” bit is that the interviewers, including the manager, involved in hiring generally did not come from the team the candidate was likely to join.

As we started hiring more senior candidates, we moved away from pool hiring, as senior candidates have a higher expectation of learning about the team they’d join. It’s also remarkably tricky as a manager to pitch a candidate about a team that you only vaguely know about; candidates would often end up wanting to join the interviewing manager’s team rather than the headcount optimal destination.

The switch from pool hiring to team-specific hiring introduced some challenges of its own, particularly a loss of consistency in the hiring process. When the same team and same manager interview for all their team’s roles, you start to see teams’ hiring processes diverge. To address that divergence, we introduced a hiring committee, I believe sometime in 2018 with around ~800 engineers in the company.

One version of the hiring committee process worked as follows (I say “one version” because the details changed a bunch over time, although the core remained fairly constant):

  1. Candidates interview for specific teams’ roles, and are interviewed by the team and hiring manager they would work with
  2. Hiring Manager makes a tentative hiring decision and writes up a hiring proposal for Hiring Committee
  3. Hiring Committee would review hiring proposals in a standing 30 minute meeting every other day and approve/reject/send them back to the hiring manager for more detail
  4. Recruiter and Hiring Manager would coordinate follow up steps (either pitching the candidate, sending them back to Hiring Committee, or informing them they weren’t moving forward)

This process worked well for us, although some managers were frustrated that it slowed down the hiring process by one to two days (and sometimes more if the hiring manager was slow to prepare the hiring proposal). It was also periodically challenged by poorly calibrated members joining the hiring committee, which required ongoing attention to address.

Pros

  • Higher consistency of hiring decisions due to smaller number of committee members
  • Hiring committee members learn from and calibrate with each other quickly
  • Generates a centralized, clean data source to improve organizational hiring

Cons

  • Introduces, at minimum, an additional 1-2 day delay in hiring process
  • Significant time load on committee members, which may not be valued as high impact work
  • Hiring outcomes are disproportionately impacted with only a few miscalibrated committee members

Inspection questions

  • What are your offer-extend and offer-accept rates?
  • How quickly are hiring proposals approved/rejected?
  • What percentage of hiring proposals are approved, rejected, or returned for additional information?
  • What is the tenure of managers writing packets that are rejected or returned for additional information?
  • Are there managers whose proposals are always approved? Can you streamline the process to accelerate that?
  • Are there managers whose proposals are rarely approved? Can you provide improved education and training for those managers?
  • How does the executive sponsor know if structure needs to be adjusted?

Structured approval

It’s easy to argue that every hiring process should already have explicit accountable parties for each step, but my lived experience is that many hiring processes are ambiguous, uncertain, and unclear – fixing that is much easier than rolling out a hiring committee or bar raising. I’ve seen great success with a well-structured hiring process with clear accountable parties, which I’m labeling structured approval.

How we’ve generally run this process in Calm engineering is:

  1. Recruiter is responsible for sourcing, deciding on advancing forward to onsite, scheduling interviews
  2. Hiring Manager makes initial decision to make candidate an offer after gathering feedback on candidate performance, and hosting a hiring debrief
  3. Recruiter creates a private Slack channel including Hiring Manager, Recruiting Manager, Head of Engineering, and managers (if any) in reporting chain between Hiring Manager and Head of Engineering
  4. Hiring Manager writes a 1-page offer rationale, and shares it in the private Slack channel
  5. Recruiter makes initial compensation proposal based on compensation bands, and shares in Slack channel
  6. Head of Engineering has 24 working hours to review and approve the offer and the compensation proposal. If they miss the window, the offer and proposal are approved by default. They may send either component back for a second iteration to collect more feedback or modify the offer. Approval and coordination is done within Slack channel
  7. Recruiter extends offer, coordinates closes, and archives Slack channel on final outcome
  8. Head of Engineering meets with head of engineering recruiting on weekly basis to debug hiring process and work through any specific issues that have come up

At each stage we have an explicit accountable party, the Slack channel creates visibility into the process for those involved in a given candidate, and we meet frequently to debug and improve the underlying structured process. That’s not to say that this specific process will be ideal for you, that will vary on your team, hiring rate, and so on. However, I’m certain there is a version of this process that will work effectively for you, even at a significant scale (e.g. it’s easy to imagine a transition point after which only Staff-plus and Director-plus offers escalate to Head of Engineering, and other offers can be made closer to the hiring team).

Pros

  • Low coordination cost
  • Avoids diffuse accountability across many parties
  • Role of executive sponsor is clear, unambiguous, and fails loudly

Cons

  • No inherent quality control mechanics, outcomes are wholly dependent on individuals involved
  • Many specific structures are slow, often introducing 1+ day delay in hiring
  • Requires care to avoid degrading into exception-centric approach

Inspection questions

  • What are your offer-extend and offer-accept rates?
  • How long does it take new managers and recruiters to ramp up on process?
  • How long does approval take? Drilling in, what parts of approval are slow? Are they slow for everyone, or just folks ramping up?
  • How many rounds of back-and-forth does it take to align on a compensation proposal?
  • How does the executive sponsor know if structure needs to be adjusted?

Recommendation

Having experienced companies using all of these mechanisms, my default advice to folks on the topic of increasing the quality of organizational hiring is:

  • Introduced structured approval when you introduce your second hiring manager in a function (e.g. second engineering manager)
  • Wait until organization trust grows sufficiently weak that there’s significant skepticism about hiring quality across teams, then introduce a hiring committee
  • Avoid introducing bar raising unless you have a clear thesis on why the other approaches won’t work out

As with all organizational processes, good process is evolved, not designed: be thoughtful about your rollout, and continue paying attention while it transitions from concept to practice.

You’ll note that I don’t recommend bar raising in any scenario. It’s true, I really have not found it particularly effective, but I’m not arguing that it can’t be effective for some, just that I’ve never seen it work well in practice despite seeing three places attempt to adopt it, so I don’t recommend it. Conversely, I’ve found hiring committees to work reasonably well, failing only when they are not maintained, so I do recommend them if you have an engaged executive sponsor.

As with all things, your mileage may well vary.


That's all for now! Hope to hear your thoughts on Twitter at @lethain!


This email was sent to you
why did I get this?    unsubscribe from this list    update subscription preferences
Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA

Email Marketing Powered by Mailchimp

Older messages

How to estimate disk space. @ Irrational Exuberance

Thursday, July 14, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - How to estimate disk space. -

Downturn career decisions. @ Irrational Exuberance

Wednesday, July 6, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Downturn career decisions.

STAR method for interview questions. @ Irrational Exuberance

Wednesday, June 22, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - STAR method for interview

Career checkup template. @ Irrational Exuberance

Wednesday, June 8, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Career checkup template. Career

Plagarism, idea theft, and writing online. @ Irrational Exuberance

Wednesday, May 25, 2022

Hi folks, This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email. Posts from this week: - Plagarism, idea theft, and

Autumn, clocks, and chocolate

Sunday, September 25, 2022

A monotechnic issue, in a way ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Food for Agile Thought #361: Useful Agile Metrics, The Stakeholder Whisperer, Successful Transformations

Sunday, September 25, 2022

Also: Spotify Model, Kodal's Demise, VSM, Changing How You Build, Level 1 Goals ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

063 Being mad: a sign of genius?

Sunday, September 25, 2022

Isn't it ironic that the last name of the “Mad Scientist” is literally synonymous to the word Genius? ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Global menstrual research course, Philanthrocapitalism, Associate editor at Feminism In India

Sunday, September 25, 2022

The Bloom Issue #97, Sept 25 ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

FTI #327: Even the losers get lucky sometimes

Sunday, September 25, 2022

​ ​ ​ Tom Petty refused to sell seats in the first two rows of his shows. Instead, before the show he'd go find the people sitting in the back two rows and move them to the front. "I figure if

What’s the (actual) benefit?

Sunday, September 25, 2022

It's probably not what users actually love most. ‌ ‌ ‌ Hey! Brian Casel here. As a reminder, you're subscribed to this here newsletter where I share my notes and conversations about products,

Time management doesn’t work. Try this instead.

Saturday, September 24, 2022

We are all blessed with this self-renewing resource ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Did that hurt?

Saturday, September 24, 2022

We can endure all kinds of pain. It's shame that eats men whole. ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Friday Finds (Formula 1, Advanced Writers, Steve Jobs, Hollywood)

Friday, September 23, 2022

Read in your browser here. Hi friends, Monday is the last day to sign up for our next Write of Passage cohort, beginning on October 5th. This week, somebody on Twitter asked a delightful question:

Here's what you missed...

Friday, September 23, 2022

​ ​ ​ Looks like you may have missed last week's FOR THE INTERESTED newsletter so I wanted to send it to you again in case it got lost in your inbox. Have a great weekend! Josh *** You know that