👋 This is Gergely with a bonus 🔒 subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at big tech and high-growth startups through the lens of engineering managers and senior engineers.

This issue is a follow-up to my two September posts about the insane tech hiring market (part one, part two). Follow-up articles add additional data points to the original piece.

The market for tech professionals with experience - and especially software engineers - continues to be on fire. This newsletter issue is especially relevant for hiring managers, and those budgeting for next year.

This issue covers:

• Compensation increases done off-cycle with over 30 data points from various companies.

• A data engineer shortage, how companies are responding to this, and the opportunities this opens for engineers

• Facebook’s hiring crisis, and how this will impact the tech market.

• Remote work options are starting to dominate job postings.

• Low-cost regions: are they becoming too expensive?

• Advice to hiring managers on tackling all the above challenges.

Newsletter update: paying subscribers no longer see job adverts. The Pragmatic Engineer Job board is an experiment I ran for a month in this newsletter, offering a way for companies hiring to reach more potential candidates. Job adverts are visible on the bottom of The Pragmatic Engineer Blog, and in newsletters issues sent to free subscribers.

Effective with this issue, job ads are no longer shown to paid subscribers. This change is for two reasons:

1. The newsletter is funded by subscribers who get independent, high-quality, ad-free and sponsor-free content. Job ads are still ads: thus they have to go.

2. A growing number of companies purchase group newsletter subscriptions to engineering managers and senior engineers on their team. They do this as investment in their people: and not with the goal of finding that next job through the newsletter. Removing ads aligns these incentives as well.

A data engineer shortage?

Data usage is exploding, and companies need to make more use of their large datasets than ever. Talking with hiring managers, the past 18 months has been a turning point for many organizations, where they are doubling down on their ability to extract real-time insights from their large data sets.

“Data engineer” is a recent and pretty vague term that each company defines differently. It usually means one of these two:

1. A software engineer who picked up some data technology. 

2. Data scientist/analyst who learned how to productionize code. 

What makes hiring for data engineers challenging is the many languages, technologies and different types of data work different organizations have. Here are a few examples of what different companies expect from someone with the title “Data engineer”:

• Java, Snowflake SQL pipelines, and SAS/Apex stack

• Scala, zio+Flink, Beam, Python, DBT

• Scala, Spark, Hadoop, BigQuery, Kafka, Airflow

• Spark, Databricks, Flink, Presto, Hadoop (DataOps)

• Spark, Python, distributed systems, big-O understanding & map/reduce know-how

• Flink, Athena, Java, Python, ML tools

• Spark, Python, Scala, Kafka (streaming), Airflow (scheduling), Redis (fast access)

• Airflow, Snowflake, Redshift, DBT

There seems to be a severe data engineering shortage, based on over a dozen of hiring managers I’ve talked with. All of them have confirmed that they are struggling to hire for these positions, or to grow their data engineering teams.

An engineering manager working at a large company going through digital transformation shares:

“We are struggling to hire data engineers. We cannot find the people on the market with the skills we need. For now, we’re forced to share the time of available data engineers between multiple teams.

I’ve seen the lack of skills, first-hand as I am an interviewer. Many candidates show up have poor technical skills. For example, a candidate embedded a python script into a word document and could not open it during the interview...”

A hiring manager in Russia paying above the market says:

“Finding data engineers is extremely hard. There are an order of magnitude fewer candidates than backend engineers. Any new opening takes months to fill.”

Another hiring manager highlights what they struggle with:

“It is hard for us to find someone that understands Spark on a deep level, has experience with our tooling or equivalents and has strong engineering fundamentals. So we usually end up biased towards people with strong engineering fundamentals: they’ll learn the rest.”

Data engineers on the market share how each company is hiring for something different:

“Even now, some companies I interview with want to hire data engineers to help them scale their production infrastructure. Instead of what I’m looking for, as a data engineer, which is piping data into and out of a data warehouse, or productionalizing ML models.”

So what does a “regular” data engineer look like, from the eyes of data engineers interviewing on the market? Here are a few quotes people with the title Data Engineer shared:

“I say a data engineer is a data-savvy developer who can integrate data sources at scale Spark, most likely). They can also  and implement canned data science "stuff" ... maintain and scale data pipelines, ensure their reliability and availability, plus build and maintain internal APIs to access the data.”

“There are a few things almost all companies I interview ask for. This is: be competent at SQL have an overall understanding of systems design. Experience with big data is preferable but not usually required. Experience with streaming data pipelines was also the same.”

“Almost every company I interviewed with was either using Spark or interested in using Spark. Some were leveraging Databricks, others were managing things in house.”

“Technology-wise across companies there’s quite a mix of just the JVM technologies, SQL pipelines like Snowflake, proprietary SAS/Apex stacks and various Python tech like DBT. And then you have the Spark/Databricks shops.”

Many companies are responding with in-house data engineer training. IKEA has started a Junior Data Engineer accelerator program. Acamai does in-house training for software engineers, as do several other companies. A quote from a data engineering manager:

Data engineering is a career path already open to many software engineers - though few are aware that it’s a viable path, with large demand. I’ve talked with several software engineers who have applied for these roles, and were surprised to get good offers at well-known companies.

Because there is such a shortage, and because the “data” and “tools” part is easy enough to pick up, companies are usually open to hiring software engineers with backend experience, and then support them to pick up the toolset and data know-how.

It’s not just experienced engineers: a Lambda School graduate shared a thread on Twitter on how she pivoted to data engineering, after realizing that she’s getting far more success with this specialization than from most others. She now works as a data engineer:

EvidenceN @EvidenceNMedia

When I applied for data science jobs I got CRICKETS. No response. When I applied for data engineering jobs even the ones I don't qualify for, I got INTERVIEWS. So, I figured either there is high demand for data engineers or there is less competition for Data Eng roles.

April 20th 2021

11 Retweets230 Likes

Having a solid data engineering team and strategy can turn many businesses around, and accelerate even more. I talked with an engineer working at a private equity firm who specialize in business turnarounds. Their strategy? They buy businesses that struggle, extract data from them, then use this data to turn fix up the company and scale them - and have a high success rate, and solid profits.

I personally see the demand for data engineering keeping to grow in a similar fashion. Big Tech will keep investing, many scaleups build data engineering as part of their DNA, and even more traditional companies will hire data teams. It’s a more attractive option than getting bought by private equity firms who then do the same.

Advice for hiring managers: if you need to hire data engineers, prepare that the market is challenging. It might make sense to think about internal training, or hiring engineers motivated to transition into data roles.

Off-cycle compensation adjustments at companies

I wrote the article The Perfect Storm Causing an Insane Tech Hiring Market in September, when I suggested lobbying for one-off compensation increases to get ahead of attrition, responding to this major market shift:

“Lobby for out-of-cycle increases and spot bonuses. Most nimble companies are giving raises to all their staff. You’ll need to do it at the beginning of the year anyway, but if you wait until then, you will have lost people who would have stayed otherwise. Assume the compensation data you buy is outdated and that the market could be up 15-30% on last year”

The data is now in: many companies have, indeed taken this drastic step to adjust compensation early. I talked with more than two dozen hiring managers and employees at different companies across the globe who have all done off-cycle adjustments. Data points:

Europe

• 30% increase for all Data Engineers at a publicly traded company with a large office in Portugal. Other engineering expected to follow: however, Data Engineering attrition was too high to not act.

• 25% base increase for engineers at a Series B startup based in Prague, Czechia. This company saw iOS and Android native engineers being poached from them.

• 25% increase for engineers in the UK and Germany at a Series B startup with an US HQ

• 20-25% base increase for managers at a Series C FinTech in London. 

• 20-25% base increase for engineers at a Series A startup with most engineers based in Hungary.

• 20% base increase for a Seed startup in London

• 20% increase for “high impact loss” tech employees at an EU tech company with offices in Germany and Spain.

• FreeNow: 15-25% raise for tech managers at FreeNow in Germany. Engineers expected to get a raise later.

• 15-20% increase for senior engineers at a sports tech company in London

• 15% increase for a seed-stage startup with remote employees across Europe.

• GitLab: 15% base increase for engineers in the Security business unit at GitLab. The security org re-benchmarked their salary against the market and did an out-of-cycle increase for everybody currently working there.

• 15% increase or £10K increase (whichever higher) at a FinTech scaleup in London ONLY IF people commit to coming to the office 4 days/week

• eBay: 15% base increase at an eBay subsidiary in one of the countries in Europe in Aug 2021. 

• 15% base increase for all remote developers working in countries like Romania, Serbia by a London-based startup

• 14% raise for engineers at a bank subsidiary effective 1 Nov. This was after management told engineers in June there will be no raises until Apr 2022, answering questions. 25 engineers, mostly seniors (20% of staff) left over the summer and fall, most quoting unhappiness with pay, promoting this emergency measure.

• 10-20% increase for Tech, R&D+Ops employees in a Series B startup in Israel

• Booking.com: 10% base increase for everyone in engineering at Booking.com in the Netherlands

  • “So Booking decided two days ago to randomly give a flat 10% raise to all Software Developers, SREs, EMs, Production Engineers, Data Scientists in Analytics and security engineers. However, ML Engineers,Product and Design were left out of this raise.”

  • “The main reason is high attrition in these roles and very hard to close offers on candidates and many many open positions. The secondary reason was that the Pulse survey results were low.”

  • “Now a senior developer at Booking.com, in Amsterdam, is making 100-130k base, 20% bonus, $40-$60k RSUS refreshers.”

  • ML Engineering however has been reported to have received out-of-cycle raises earlier in Q1/2021

• Adyen: 5-20% base increase for everyone at Adyen in the Netherlands July-Aug 2021.

  • Engineers with less experience closer to 15-20%

  • Senior engineers closer to 5%

  • “Most juniors got a 10-20% bump”

• 5-20% increase for a Series D FinTech in London:

  • Mid-level: 5-10%

  • Senior: 10-15%

  • Staff: 15+20%

• 5% spot increase in a Series C company with EU engineers (UK, Finland & other offices)

• Coolblue: 4-10% increase at Coolblue in the Netherlands, based on seniority

  • Juniors: €1,608/year increase (€125/month + 8% holiday)

  • Seniors: €6,480/year increase (€500/month + 8% holiday)

  • Principals: €9,720/year increase (€750/month + 8% holiday)

• 2% spot increase at DMG Media for all staff in the UK (Daily Mail, Metro etc)

US & Canada

• Accenture: 15-25% base salary increase at Accenture in parts of the US, in certain groups in June this year. Part of marketplace adjustment increase.

• 15-20% increase for remote engineers at a Series B startup across the US (Midwest, Texas etc) 

• 10-15% increase for all engineers at a Series B company with most employees in NYC

• 10% increase at bootstrapped Aha.io, a SaaS company with around 200 employees, mostly US. Details in a pubic post.

• 5-10% spot raise + doubling profit sharing at a non-VC funded company in the Midwest (<100 people)

  • “I was already fairly compensated, in my opinion, but the reason given was the current state of the market. Over the summer we had a couple people leave, and compensation did play a role.”

• Ubisoft: 5% minimum raise at Ubisoft, Canadian studios

• 5% spot raise for tech at a publicly traded FinTech company

• Employees being pulled up to the bottom of increased bands: common at many tech companies

Global

• $100K-150K one-off stock grant (vesting over 4 years) for Staff+ levels for both existing employees and new offers at a recently IPO’d tech company

• 15-25% raise + equity bump at a Series F company with US, EU, Asia offices. “I received a 20% salary raise and about a 12% bump in options. My teammate received a salary bump of about 17%.”.

• New Relic: 8% base increase. They predicted attrition if they did not act. In the past years, this increase was 2%.

• 7% global raise for remote engineers at a Series B company with US and EU remote engineers. Another 10-15% raise planned based on attrition risk and location. 

• Amazon: $3,500 spot-bonus: several employees were granted 1 RSU (trading at $3,500), vesting immediately. Targeting those at high attrition risk.

• ServiceNow: 4-10% increase. IC2s got 6% increase

• AMEX: 3% base increase on top of the regular annual increases, globally.

• Policy to allow counter offers up to +30% without heavyweight process at several mid-size companies

• Red Hat to hire fewer senior engineers to control costs (source)

  • “An internal email sent on Wednesday by Timothy Cramer, SVP of software engineering, to Red Hat managers directs hiring requisitions to be made at a lower level of seniority than usual.”

  • "All new plan reqs should be opened at a level below senior (e.g., Associate Software Engineer or Software Engineer)," the message says.

  • "While this change allows us to use our budget more effectively, it also helps us balance the organization as we have many engineers with senior titles. We recognize that this will mean we need to plan for training and mentoring, promotions, and internal mobility as well, and we are here to support you in that."

Asia & Australia

• 60% increase for people planning to resign, plus 10% retention bonus for 3 years in Bangalore, India at a non-tech company facing high attrition

• PhonePe: 40% increase for all tech staff at PhonePe, a payments company in India

• Atlassian: 20-25% base + more equity for certain people. “Was done in a fairly indiscriminate way”

• 15% base increase for engineers working remotely in India from by an EU-based startup

• $1,500 (100K INR) retention bonus for India engineers at a Series B company in exchange to agree to stay until 2023. Paid out in two installments in 2022.

Latin America

• 50% raise in Chile for software developers working remote at a company to fight attrition. They now make $100K: this was $65K beforehand.

• Amazon Brazil: 10% increase for all engineers

Africa

• 20% increase for remote senior engineers in Cape Town, South Africa at a Series B US company

Counteroffers

• 100% increase on a counteroffer in India, at a Series B startup to retain (and match new salary)

• 80% raise via increased stock for a staff engineer who received an even higher offer from a pre-IPO company desperate to recruit them, very specifically due to specialization

• 40% raise via base salary + increased equity at a pre-IPO company in the Netherlands for a key tech lead, after they got an even higher offer from a FinTech startup

• 30% raise in counter offer for a staff engineer at a German car industry company, after receiving that much higher of an offer.

• 25% as a remote engineer working remote, for a US company:
  “I told my boss that there’s a big difference between what the market offers, and what I’m being paid”

Location-independent remote pay

• Etsy is moving to location-independent pay per country over the previous location-based one. Ranges are pegged to metro area locations This means in the US they’ll pay a single SF/NYC pay scale regardless of location, Canada pegged to e.g. Toronto/Vancouver rates.

• Reddit and Spotify have done the above earlier the year

• RevenueCat (Series A) is paying global location-independent salaries

• More well-funded seed & Series A & B startups are doing location-independent pay per country

No raises - and their impact

I’m assuming that the majority of companies have not done off-cycle adjustments. Here are a few companies where I have talked with managers on the impact that holding off on one-off adjustments has had

• Meta (Facebook), Google, Apple Uber, Lyft, Microsoft not doing out-of band adjustments

  • Impact: upleveling is far more common during interviews, in the opposite of what you’d expect based on The Seniority Rollercoaster

  • I have confirmed examples from the above companies on candidates being offered a higher level when those companies could not match compensation expectations

• Klarna, RedHat, Solarisbank (for years: leadership said they are considering)

  • Impact: attrition, very challenging to hire, heavy upleveling

• Dutch scaleup expected to IPO the next year where all engineers are issued equity (and value has gone up). Market salary is now well below what they could get elsewhere. CEO on all-hands: “We’re aware of the market but we don’t think we should be competing with companies that are more aggressive towards recruiting)”

  • Impact: none right now (stock increase + IPO coming up), likely increased attrition after IPO

• Series B US company: “Right now we are ignoring the fact that the hiring market is hot. Many good candidates we want just refuse to join after hearing our salaries. From what I see the company is now lowering the hiring bar to keep salaries the same. Also no stock refreshers.”

Advice for hiring managers: if your company has not yet made off-cycle compensation changes, you’ll likely have to make a case for a larger than usual raise for the new year. Start engaging your leadership now. The market has moved exceptionally the past 12 months: exceptional raises will be necessary to keep pace with it, and not risk attrition due to finances. Subscribers will find more advice in the next section.

Summary of the market changes since January

How has the tech market changed overall, and how is this change likely to continue? Let’s dive in...