More Follow-Up on the Tech Hiring Market
This is a free preview of a 🔓 paid subscribers-only post. If you’d like to read the full post, subscribe: More Follow-Up on the Tech Hiring MarketOff-cycle compensation increases, a data engineer shortage and low-cost regions becoming too expensive.👋 This is Gergely with a bonus 🔒 subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at big tech and high-growth startups through the lens of engineering managers and senior engineers. This issue is a follow-up to my two September posts about the insane tech hiring market (part one, part two). Follow-up articles add additional data points to the original piece. The market for tech professionals with experience - and especially software engineers - continues to be on fire. This newsletter issue is especially relevant for hiring managers, and those budgeting for next year. This issue covers:
Newsletter update: paying subscribers no longer see job adverts. The Pragmatic Engineer Job board is an experiment I ran for a month in this newsletter, offering a way for companies hiring to reach more potential candidates. Job adverts are visible on the bottom of The Pragmatic Engineer Blog, and in newsletters issues sent to free subscribers. Effective with this issue, job ads are no longer shown to paid subscribers. This change is for two reasons:
A data engineer shortage?Data usage is exploding, and companies need to make more use of their large datasets than ever. Talking with hiring managers, the past 18 months has been a turning point for many organizations, where they are doubling down on their ability to extract real-time insights from their large data sets. “Data engineer” is a recent and pretty vague term that each company defines differently. It usually means one of these two:
What makes hiring for data engineers challenging is the many languages, technologies and different types of data work different organizations have. Here are a few examples of what different companies expect from someone with the title “Data engineer”:
There seems to be a severe data engineering shortage, based on over a dozen of hiring managers I’ve talked with. All of them have confirmed that they are struggling to hire for these positions, or to grow their data engineering teams. An engineering manager working at a large company going through digital transformation shares:
A hiring manager in Russia paying above the market says:
Another hiring manager highlights what they struggle with:
Data engineers on the market share how each company is hiring for something different:
So what does a “regular” data engineer look like, from the eyes of data engineers interviewing on the market? Here are a few quotes people with the title Data Engineer shared: “I say a data engineer is a data-savvy developer who can integrate data sources at scale Spark, most likely). They can also and implement canned data science "stuff" ... maintain and scale data pipelines, ensure their reliability and availability, plus build and maintain internal APIs to access the data.” “There are a few things almost all companies I interview ask for. This is: be competent at SQL have an overall understanding of systems design. Experience with big data is preferable but not usually required. Experience with streaming data pipelines was also the same.” “Almost every company I interviewed with was either using Spark or interested in using Spark. Some were leveraging Databricks, others were managing things in house.” “Technology-wise across companies there’s quite a mix of just the JVM technologies, SQL pipelines like Snowflake, proprietary SAS/Apex stacks and various Python tech like DBT. And then you have the Spark/Databricks shops.” Many companies are responding with in-house data engineer training. IKEA has started a Junior Data Engineer accelerator program. Acamai does in-house training for software engineers, as do several other companies. A quote from a data engineering manager: Data engineering is a career path already open to many software engineers - though few are aware that it’s a viable path, with large demand. I’ve talked with several software engineers who have applied for these roles, and were surprised to get good offers at well-known companies. Because there is such a shortage, and because the “data” and “tools” part is easy enough to pick up, companies are usually open to hiring software engineers with backend experience, and then support them to pick up the toolset and data know-how. It’s not just experienced engineers: a Lambda School graduate shared a thread on Twitter on how she pivoted to data engineering, after realizing that she’s getting far more success with this specialization than from most others. She now works as a data engineer: When I applied for data science jobs
I got CRICKETS. No response.
When I applied for data engineering jobs even the ones I don't qualify for,
I got INTERVIEWS.
So, I figured either there is high demand for data engineers or there is less competition for Data Eng roles. Having a solid data engineering team and strategy can turn many businesses around, and accelerate even more. I talked with an engineer working at a private equity firm who specialize in business turnarounds. Their strategy? They buy businesses that struggle, extract data from them, then use this data to turn fix up the company and scale them - and have a high success rate, and solid profits. I personally see the demand for data engineering keeping to grow in a similar fashion. Big Tech will keep investing, many scaleups build data engineering as part of their DNA, and even more traditional companies will hire data teams. It’s a more attractive option than getting bought by private equity firms who then do the same. Advice for hiring managers: if you need to hire data engineers, prepare that the market is challenging. It might make sense to think about internal training, or hiring engineers motivated to transition into data roles. Off-cycle compensation adjustments at companiesI wrote the article The Perfect Storm Causing an Insane Tech Hiring Market in September, when I suggested lobbying for one-off compensation increases to get ahead of attrition, responding to this major market shift:
The data is now in: many companies have, indeed taken this drastic step to adjust compensation early. I talked with more than two dozen hiring managers and employees at different companies across the globe who have all done off-cycle adjustments. Data points: Europe
US & Canada
Global
Asia & Australia
Latin America
Africa
Counteroffers
Location-independent remote pay
No raises - and their impact I’m assuming that the majority of companies have not done off-cycle adjustments. Here are a few companies where I have talked with managers on the impact that holding off on one-off adjustments has had
Advice for hiring managers: if your company has not yet made off-cycle compensation changes, you’ll likely have to make a case for a larger than usual raise for the new year. Start engaging your leadership now. The market has moved exceptionally the past 12 months: exceptional raises will be necessary to keep pace with it, and not risk attrition due to finances. Subscribers will find more advice in the next section. Summary of the market changes since JanuaryHow has the tech market changed overall, and how is this change likely to continue? Let’s dive in... Subscribe to The Pragmatic Engineer to read the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
Older messages
Real-World Engineering Challenges Roundup
Friday, November 12, 2021
C++ code sharing, accessible colors, migration patterns and selling more milkshakes. Issue 2.
Engineering Career Paths at Big Tech and High-Growth Startups
Tuesday, November 2, 2021
Levels at big tech, the most common career paths, and what comes after making it to Staff
Incident Review and Postmortem Best Practices
Tuesday, October 19, 2021
A survey of how companies deal with incidents today, and a peek into the best practices of the future.
Real-World Engineering Challenges Roundup
Friday, October 8, 2021
Caching, multi-tenancy, E2E tests, and database cutovers. Issue 1.
How Big Tech Runs Tech Projects and the Curious Absence of Scrum
Tuesday, September 21, 2021
A survey of how tech projects run across the industry highlights Scrum being absent from Big Tech. Why is this, and are there takeaways others should take note of?
You Might Also Like
JSK Daily for May 2, 2024
Thursday, May 2, 2024
JSK Daily for May 2, 2024 View this email in your browser A community curated daily e-mail of JavaScript news What is Throttling in JavaScript? Explained with a Simple React Use Case Welcome back,
Issue 315 - Look ma, tight parallel park, no radar!
Thursday, May 2, 2024
View this email in your browser If you are just now finding out about Tesletter, you can subscribe here! If you already know Tesletter and want to support us, check out our Patreon page Issue 315 -
Full-Stack .NET Development, Creating Reactive Applications in .NET, More
Thursday, May 2, 2024
Home | News | How To | Webcasts | Whitepapers | Advertise .NET Insight May 2, 2024 THIS ISSUE SPONSORED BY: ■ How to Build Interactive Blazor Apps with WebAssembly ■ VSLive! Hands-On Virtual Training
Daily Coding Problem: Problem #1429 [Easy]
Thursday, May 2, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Sumo Logic. Given a array that's sorted but rotated at some unknown pivot, in which
Ranked | Which Country Has the Most Billionaires in 2024? 💰
Thursday, May 2, 2024
According to the annual Hurun Global Rich List, the US and China are home to nearly half of the world's 3279 billionaires in 2024. View Online | Subscribe Presented by: The economy is changing. Is
⚙️ Rovo
Thursday, May 2, 2024
Plus: Microsoft are (were?) terrified of Google's AI
Have VPN connection issues? This might be why
Thursday, May 2, 2024
DJI Power station; Studying with AI; Best gaming PCs -- ZDNET ZDNET Tech Today - US May 2, 2024 placeholder Having VPN connection issues? Microsoft warns the April 2024 Windows update is to blame If
Programmer Weekly - Issue 203
Thursday, May 2, 2024
View this email in your browser Programmer Weekly Welcome to issue 203 of Programmer Weekly. Let's get straight to the links this week. Quote of the Week "The hardest part of design is keeping
Python Weekly - Issue 648
Thursday, May 2, 2024
View this email in your browser Python Weekly Welcome to issue 648 of Python Weekly. Let's get straight to the links this week. News Fake job interviews target developers with new Python backdoor A
A new approach to access management for the way we work today
Thursday, May 2, 2024
Announcing 1Password® Extended Access Management ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏