Bar raisers, hiring committees, and other complex ways to improve hiring quality. @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.

Posts from this week:

- Bar raisers, hiring committees, and other complex ways to improve hiring quality.

Bar raisers, hiring committees, and other complex ways to improve hiring quality.

When Uber Engineering reached 800 engineers, engineering was divided across roughly five engineering directors. Most engineering and process issues were resolved locally within these five organizations. This worked well for the most part, but meant there was little consistency within our core processes. Hiring was particularly fragmented, with every team’s manager responsible for developing their own interviews, questions and structure.

However, no matter which team initially hired an engineer, almost every team ended up working with that engineer, and over time some teams became increasingly frustrated by other teams’ perceived lax hiring standards. Eventually, three directors became so dissatisfied with a fourth’s hiring standards that they successfully argued for rolling out a “bar raiser” program, modeled after Amazon’s. In that program, at least one well-qualified interviewer outside the hiring organization would weigh in on every hire. In theory, the bar raiser was the final decider on whether a candidate could receive an offer, ensuring each hire met at least two organizations’ hiring standards. As a member of the initial bar raiser rollout, I can vouch that it addressed many of these issues, although it struggled to deal with hiring managers who escalated any bar raiser decision that contradicted their own.

When I later joined Stripe, I’d internalized a core lesson about bar raising programs: they are easy to silently subvert unless they have a great deal of ongoing, empowered organizational support. So I pushed back on introducing bar raising, and was glad to throw support behind instituting a hiring committee. Wouldn’t you know it, hiring committees introduce some challenges of their own.

From here: deciding if you should introduce a program to improve hiring quality, running a bar raising program, operating hiring committees, and finally my preferred approach of using structured approval.

Should you prioritize hiring quality?

A big part of running a successful engineering organization is avoiding as much process as you can, but no more. Three helpful questions to answer when you’re considering if you need to improve hiring quality are:

1. Have you already created a structured hiring process and trained your team on using it? If not, go do that first.
2. Do you have a general concern about hiring quality, or is it a narrow problem related to a small number of individuals? If it’s just a few folks, introduce a performance management system first, and use it to manage those individuals up or out.
3. Are you still concerned about the quality of your hiring after ramping up on structured hiring and performance management systems?? Alright, let’s introduce a program to improve hiring quality!

Generally, you’ll find these processes are most valuable during periods of rapid hiring and especially when many hiring managers are new to your company. If you believe it’s necessary to introduce these practices during a period of low hiring or long-tenured hiring managers, my initial guess is that you have a performance problem within your team that needs to be addressed directly, and that introducing a process is simply a form of postponing dealing with that issue.

Bar raisers

Since Amazon introduced its bar raiser program, it’s been tweaked and copied by many companies. As mentioned in the opening paragraphs, Uber also adopted the bar raiser program in ~2015, which worked as follows: \

1. Interview loops were generally designed by each teams’ engineering manager
2. However, all loops must include a bar raiser as an interviewer. The bar raiser must come from a different Director’s reporting chain than the loop’s hiring engineering manager
3. Bar raisers gave an interview of their choice, rather than an interview assigned by the loop’s hiring engineering manager
4. Bar raisers were selected for strong interviewing skills, evidence of experience interviewing at Uber specifically, and were approved by the team leading the bar raiser program. Bar raisers went through a brief training program before starting bar raiser interviews

Introducing bar raisers was a simple change to the interviewing process itself, but nonetheless meaningfully changed Uber’s overall interviewing outcomes. The biggest operational challenge was at times invisible, and came down to resolving conflict between bar raisers, hiring managers, and hiring managers’ reporting chains. Without executive sponsorship supporting bar raisers’ decisions, the role’s impact is easily undermined by frequent overrulings from the hiring manager’s reporting chain.

Pros

• Decentralized, scales effectively as number of interviews scales
• Avoids slowing down process by introducing an additional approval step
• Multiple potential bar raisers for any given interview loop mean that hiring quality remains relatively high even if a percentage of bar raisers are poorly calibrated

Cons

• Susceptible to overriding escalations from hiring management chain, undermining impact and morale of bar raisers
• Unclear incentives for the bar raisers themselves, given the work is often not valued in promotion process
• Decentralized day-to-day operation can create ambiguous role for executive sponsor, which often translates into disengaged rather than accountable sponsorship

Inspection questions

• What are your offer-extend and offer-accept rates?
• How often do bar raisers change the outcome of an interview?
• How many bar raisers do you have?
• How many bar raisers have you added/lost in the past quarter?
• What is the interview load on bar raisers?
• How does the executive sponsor know if structure needs to be adjusted?

Hiring committees

Through 2016, engineering hiring at Stripe followed the “pool hiring” model: a candidate would enter the process, get interviewed by engineers and one manager drawn from across the organization, and then the manager and recruiter would synthesize the feedback discussion into a hiring decision during the interview debrief (for reasons lost to history, candidate debrief meetings were internally referred to as the “trope”). The “pool hiring” bit is that the interviewers, including the manager, involved in hiring generally did not come from the team the candidate was likely to join.

As we started hiring more senior candidates, we moved away from pool hiring, as senior candidates have a higher expectation of learning about the team they’d join. It’s also remarkably tricky as a manager to pitch a candidate about a team that you only vaguely know about; candidates would often end up wanting to join the interviewing manager’s team rather than the headcount optimal destination.

The switch from pool hiring to team-specific hiring introduced some challenges of its own, particularly a loss of consistency in the hiring process. When the same team and same manager interview for all their team’s roles, you start to see teams’ hiring processes diverge. To address that divergence, we introduced a hiring committee, I believe sometime in 2018 with around ~800 engineers in the company.

One version of the hiring committee process worked as follows (I say “one version” because the details changed a bunch over time, although the core remained fairly constant):

1. Candidates interview for specific teams’ roles, and are interviewed by the team and hiring manager they would work with
2. Hiring Manager makes a tentative hiring decision and writes up a hiring proposal for Hiring Committee
3. Hiring Committee would review hiring proposals in a standing 30 minute meeting every other day and approve/reject/send them back to the hiring manager for more detail
4. Recruiter and Hiring Manager would coordinate follow up steps (either pitching the candidate, sending them back to Hiring Committee, or informing them they weren’t moving forward)

This process worked well for us, although some managers were frustrated that it slowed down the hiring process by one to two days (and sometimes more if the hiring manager was slow to prepare the hiring proposal). It was also periodically challenged by poorly calibrated members joining the hiring committee, which required ongoing attention to address.

Pros

• Higher consistency of hiring decisions due to smaller number of committee members
• Hiring committee members learn from and calibrate with each other quickly
• Generates a centralized, clean data source to improve organizational hiring

Cons

• Introduces, at minimum, an additional 1-2 day delay in hiring process
• Significant time load on committee members, which may not be valued as high impact work
• Hiring outcomes are disproportionately impacted with only a few miscalibrated committee members

Inspection questions

• What are your offer-extend and offer-accept rates?
• How quickly are hiring proposals approved/rejected?
• What percentage of hiring proposals are approved, rejected, or returned for additional information?
• What is the tenure of managers writing packets that are rejected or returned for additional information?
• Are there managers whose proposals are always approved? Can you streamline the process to accelerate that?
• Are there managers whose proposals are rarely approved? Can you provide improved education and training for those managers?
• How does the executive sponsor know if structure needs to be adjusted?

Structured approval

It’s easy to argue that every hiring process should already have explicit accountable parties for each step, but my lived experience is that many hiring processes are ambiguous, uncertain, and unclear – fixing that is much easier than rolling out a hiring committee or bar raising. I’ve seen great success with a well-structured hiring process with clear accountable parties, which I’m labeling structured approval.

How we’ve generally run this process in Calm engineering is:

1. Recruiter is responsible for sourcing, deciding on advancing forward to onsite, scheduling interviews
2. Hiring Manager makes initial decision to make candidate an offer after gathering feedback on candidate performance, and hosting a hiring debrief
3. Recruiter creates a private Slack channel including Hiring Manager, Recruiting Manager, Head of Engineering, and managers (if any) in reporting chain between Hiring Manager and Head of Engineering
4. Hiring Manager writes a 1-page offer rationale, and shares it in the private Slack channel
5. Recruiter makes initial compensation proposal based on compensation bands, and shares in Slack channel
6. Head of Engineering has 24 working hours to review and approve the offer and the compensation proposal. If they miss the window, the offer and proposal are approved by default. They may send either component back for a second iteration to collect more feedback or modify the offer. Approval and coordination is done within Slack channel
7. Recruiter extends offer, coordinates closes, and archives Slack channel on final outcome
8. Head of Engineering meets with head of engineering recruiting on weekly basis to debug hiring process and work through any specific issues that have come up

At each stage we have an explicit accountable party, the Slack channel creates visibility into the process for those involved in a given candidate, and we meet frequently to debug and improve the underlying structured process. That’s not to say that this specific process will be ideal for you, that will vary on your team, hiring rate, and so on. However, I’m certain there is a version of this process that will work effectively for you, even at a significant scale (e.g. it’s easy to imagine a transition point after which only Staff-plus and Director-plus offers escalate to Head of Engineering, and other offers can be made closer to the hiring team).

Pros

• Low coordination cost
• Avoids diffuse accountability across many parties
• Role of executive sponsor is clear, unambiguous, and fails loudly

Cons

• No inherent quality control mechanics, outcomes are wholly dependent on individuals involved
• Many specific structures are slow, often introducing 1+ day delay in hiring
• Requires care to avoid degrading into exception-centric approach

Inspection questions

• What are your offer-extend and offer-accept rates?
• How long does it take new managers and recruiters to ramp up on process?
• How long does approval take? Drilling in, what parts of approval are slow? Are they slow for everyone, or just folks ramping up?
• How many rounds of back-and-forth does it take to align on a compensation proposal?
• How does the executive sponsor know if structure needs to be adjusted?

Recommendation

Having experienced companies using all of these mechanisms, my default advice to folks on the topic of increasing the quality of organizational hiring is:

• Introduced structured approval when you introduce your second hiring manager in a function (e.g. second engineering manager)
• Wait until organization trust grows sufficiently weak that there’s significant skepticism about hiring quality across teams, then introduce a hiring committee
• Avoid introducing bar raising unless you have a clear thesis on why the other approaches won’t work out

As with all organizational processes, good process is evolved, not designed: be thoughtful about your rollout, and continue paying attention while it transitions from concept to practice.

You’ll note that I don’t recommend bar raising in any scenario. It’s true, I really have not found it particularly effective, but I’m not arguing that it can’t be effective for some, just that I’ve never seen it work well in practice despite seeing three places attempt to adopt it, so I don’t recommend it. Conversely, I’ve found hiring committees to work reasonably well, failing only when they are not maintained, so I do recommend them if you have an engaged executive sponsor.

As with all things, your mileage may well vary.

That's all for now! Hope to hear your thoughts on Twitter at @lethain!

This email was sent to you why did I get this?    unsubscribe from this list    update subscription preferences Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA