Why We Will Never Have Enough Software Developers

Why We Will Never Have Enough Software Developers

Read on My Site

We will never have enough software developers.

Developers are dropping out of the profession in large numbers, despite efforts to grow the number of computer science graduates and software engineers.

Here's why.

Developer dropout is real

Software development has a serious retention problem:

  • At age 26, 59% of engineering and computer science grads work in occupations related to their field of study. By age 50, only 41% work in the same domain, meaning a full ~30% drop out of the field by mid-career
  • In contrast, engineering and computer science majors who join unrelated fields upon graduation retain at much higher rates, with only 10-15% switching out after the age of 26:

Engineers often leave engineering for non-STEM management roles. Graduation into management is not surprising. What's surprising is that these are non-STEM positions. Engineers swap technical roles for non-technical roles over time.

This phenomenon, which I'll call "developer dropout," is a real problem. What's behind it?

Out with the old skills, in with the new skills

Programming-related jobs have high rates of skill turnover. Over time, the types of skills required by companies hiring software developers change more rapidly than any other profession.

To demonstrate this, researchers analyzed job postings on more than 40,000 online job boards and company websites between 2007 and 2019, controlling for employer, location, and occupation. They defined "new" skills as those that were rare or non-existent in 2007 but prevalent in 2019 and "old" skills as those that were prevalent in 2007 but rare or extinct in 2019.

  • While only 30% of all job vacancies required at least one new skill by 2019, 47% of computer and mathematical jobs required at least one new skill (i.e. a skill that was not common back in 2007)
  • This compares to less than 20% of jobs in fields like education, law, and community and social services
  • In addition, 16% of jobs in computer and mathematical fields in 2007 required a skill that was obsolete by 2019 (i.e. a skill that was common in 2007 but relatively rare in 2019), more than double any other job category:

About a third of the change in required skills in computer-related occupations is due to specific new software:

  • The fastest-growing software skills between 2007 and 2019 include Python, R, and Apache Hadoop
  • Software that was popular in 2007 but effectively obsolete by 2019 includes QuarkXpress, ActionScript, Solaris, IBM Websphere, and Adobe Flash (ah, finally a name I recognize)

Data science, machine learning, and AI saw big increases among technology-intensive jobs as well. For example, the number of STEM-related jobs requiring skills in machine learning and AI grew more than 4x from 2007-2017, touching more than 15% of STEM jobs:

To better compare rates of skill change across occupations, the researchers came up with a measure of skill change that tracks the absolute growth or decline of various skills within each profession from 2007 to 2019. Occupations whose required skills change rapidly in prevalence among job postings receive a high score, while jobs whose skills do not change much receive a lower score:

  • Computer-related occupations receive the highest score by far, 4.8. Note that the mean and standard deviation of this measure are ~3 and ~1 respectively, so computer-related jobs are nearly two standard deviations away from the typical job in America
  • Meanwhile, jobs in education and and those involving manual labor have very low skill change scores, typically less than 2.

We can get even more granular and look at specific job roles. This level of detail makes the difference even more stark (only showing the fastest changing roles):

Web development has the highest rate of skill change among all jobs in the country. Next up are sales engineers, another often technical role. Database administrators, computer network architects, sysadmins, and application developers all make the top 10, and we see many other technical roles among the top 30. The mean and standard deviation are similar here, placing web development more than 3 standard deviations away from the typical job in America in terms of skill change over time.

Suffice to say, software development is a rapidly changing profession.

You might think, however, that skill change would eventually settle down as one becomes more experienced.

You'd be wrong. The skills for software engineering jobs change rapidly throughout the entire career lifecycle:

  • In entry-level roles in computer and engineering occupations all the way through those requiring 12+ years of experience, the proportion of job postings requiring at least one new skill in 2019 was effectively the same, 40-45%
  • In contrast, 29% of entry-level non-computing and engineering roles in 2019 required at least one new skill, but this proportion declines to 24% for jobs requiring more than four years of experience
This means that experienced STEM workers seeking employment in 2019 are often required to possess skills that were not required when they entered the labor market in 2007 or earlier.

Software engineers never escape the skill-change vortex, even many years into their careers. Experienced engineers must learn and adopt technologies that didn't even exist when they started out. Developers must constantly retool themselves, even well after their formal education ends.

Nothing's changed but my change

College majors associated with faster changing jobs pay more early on.

  • In professions with one standard deviation increased skill change, pay is~30% higher in the first few years of one's career
  • If we exclude both the fastest and slowest-changing fields (Engineering/Computer Science at the high end, Health/Education at the low end), the early earnings premium for faster-changing roles increases to ~60%:

Fast-changing fields pay better.

Notice however that the pay advantage declines over time. By the time one approaches the age of 50, the pay premium for working in rapidly changing fields falls dramatically to only 20-30% vs slower changing professions.

Here's another way to see the eroding pay advantage. The below chart simulates the earnings of the average worker by category of college degree from ages 23 to 50 in 2016 dollars.

  • Computer science and engineering grads start off with sizable advantage vs any other major
  • However, this premium falls over time as the earnings of CS and engineering graduates plateau over time while the earnings of their peers grow faster for longer
  • In fact, life and physical science graduates' earnings surpass their computer and engineering classmates by the age of 40:

Excluding business majors, the earnings premium of software engineering declines over time in both percentage and absolute dollar terms, to the point where engineers barely out-earn social science majors:

But the focus on college major is somewhat misleading. This phenomenon has less to do with one's field of study and more to do with choice of occupation.

To show this, researchers plotted the earnings premium of various categories workers relative to those with a non-Engineering/Computer Science major working in a non-Engineering/Computer Science job.

  • Workers who major in Engineering or Computer Science but work in unrelated fields actually see their earnings advantage compound over time, rather than decline
  • On the other hand, regardless of major, individuals who work in Engineering or Computer Science jobs see their earnings advantage erode over the years:
Declining relative returns is a feature of STEM jobs, not majors. The earnings premium for non-STEM majors in STEM occupations starts off near 40%, but declines to 20% within a decade. In contrast, the relative earnings advantage grows over time for computer science and engineering majors working in non-STEM occupations.

The profession of software development drives the declining earnings premium, not the college major.

In fact, computer science majors who work in non-CS fields experience the opposite dynamic of their non-developer peers — their relative earnings premium rises as they advance. A CS major who eschews the profession doesn't earn much more than otherwise similar non-CS majors early on, but eventually out-earns their peers by nearly 20%.

OK, that's enough about what is happening. Now let's see why it's happening.

Human capital depreciates too

Imagine a simple model where workers choose their profession in order to maximize income, which is a derivative of their own skill or human capital. Over time, workers gain new skills, while the value of their existing skills depreciates somewhat due to changing times.

Some workers, endowed with superior ability, learn faster than others, picking up skills at a quicker pace. Those workers will tend to sort into high-skilled, fast-changing professions initially, maximizing their early career earnings. Less impressive workers will sort into low-skilled, slower-changing professions.

In a world where human capital never depreciated, we could imagine that high-skilled individuals like software developers would maintain a relative human capital (and earnings) advantage over other professionals, leading to consistently increasing pay and a stable relative premium:

But, if human capital depreciates over time and that rate of depreciation is higher in rapidly-changing fields like software development, then developers' initial advantage would erode over time, narrowing the gap vs. non-developers:

This simple model helps explain what we see in the data — the software engineering earnings advantage disappears as the effective human capital gap narrows.

Applied majors such as computer science, engineering, and business teach vintage-specific skills that become less valuable as new skills are introduced to the workplace over time.

Specific skills in software development quickly become dated. Programming languages and development frameworks go out of style. Hadoop is hot one year, and it's old news the next. Like a fast, expensive car that quickly loses value as it's driven around town, the skills and human capital of software engineers fall apart without constant, expensive maintenance:

Intuitively, careers with high rates of obsolescence require workers to learn many new tasks each year, which diminishes learning gains and lowers the returns to experience.

Quick learners are fast dropouts

The hits don't stop there. Ironically, the quickest learners are also the quickest dropouts.

To understand why, think back to the model we just outlined. Quick learners accumulate human capital faster than their slower peers, which means they have the most to lose when certain skills or abilities fall out of favor. In fact, the return to being a fast learner is higher in jobs with low rates of skill change because the learnings can compound over time instead of dwindle in relevance.

High-ability workers are faster learners, in all jobs. However, the relative return to ability is higher in careers that change less, because learning gains accumulate.

Said another way, the opportunity cost of working in a rapidly-changing field is highest for the best learners. This creates immense pressure to drop out of software engineering and other fast-changing careers into more stable roles and industries.

The researchers show this by regressing STEM job status on a number of other variables, including an interaction between age and score on the Armed Forces Qualifying Test (AFQT), a common measure of cognitive ability. The coefficient comes out negative and statistically significant, implying that the relative probability of working in STEM at any given age declines with cognitive ability (column 1 and 2 below):

The results imply that a worker with cognitive ability one standard deviation above average is 4.9 percentage points more likely to work in STEM at age 23, but only 1.6 percentage points more likely to be working in a STEM job by age 34.

In fact, by age 40, the regression predicts workers with higher cognitive ability are less likely to work in STEM than those with lower cognitive ability.

Simply put, professionals with higher cognitive ability drop out of STEM careers earlier and faster:

... STEM majors with higher scores on the Armed Forces Qualifying Test (AFQT)—a widely used proxy for academic aptitude—leave STEM careers more often and at younger ages.

Implication: Growing the software engineering talent pool is harder than you think

Two points I want to drive home:

First — software development has a turnover problem.

Growing the supply of software developers is not trivial because the field already sees high levels of developer dropout and turnover, and this would only increase if the field were to grow larger. A larger software development labor pool would presumably drive down wages, encouraging even more developers to shift out of the profession, especially those past their career midpoints. On that flat part of the earnings curve, the incentive to remain a developer is weak at best.

Second — software development also has a selection problem.

The highest ability, fastest learners disproportionately leave the field over time. They have a multitude of other ways to profitably leverage their intellect and skills. Software development carries serious opportunity cost. Again, this is ironic because one would normally expect the best to stay and worst to leave, but that's not what we see in the data. The software development talent pool mix shifts toward lower cognitive ability as any given cohort matures.

Combined, these points suggest software development may be destined for perennial labor shortages unless the pace of change slows sufficiently.

... rapid technological change can lead to a short shelf life for technical skills. This tradeoff between technology-specific and general skills is an important consideration for policymakers and colleges seeking to educate the workers of today, while also building the skills of the next generation.

To conclude, I emphasize: highly skilled people prefer highly stable careers in the long-run. This lets their relative ability and human capital advantage compound over time. Rapid deterioration of skills continuously levels the playing field, preventing the best from separating themselves from the pack. In such a situation, it makes more sense to quit the race early than get caught in the pile up.

Thanks to David Deming, Kadeem Noray, the authors of the study from which much of this essay is derived.

Copyright © 2020 Who is Nnamdi?, All rights reserved.
You are receiving this email because you opted in via my website.

Our mailing address is:
Who is Nnamdi?
2200 Sand Hill Road
Menlo Park, CA 94025

Add us to your address book


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.

Email Marketing Powered by Mailchimp

Older messages

Enterprise Software Monetization is Fat-Tailed 🐋

Tuesday, October 6, 2020

Why the "average" customer is a meaningless concept Enterprise Software Monetization is Fat-Tailed Read on My Site Hey, Nnamdi here. In enterprise software, the "average" customer

Top Three Strategic Priorities of Developer Productivity Startups

Wednesday, August 19, 2020

What's top of mind for developer productivity leaders Top Three Strategic Priorities of Developer Productivity Startups Read More Hey, Nnamdi here. Developer productivity startups must adapt to an

Six Trends Shaping Developer Productivity

Thursday, July 16, 2020

We interviewed developer productivity leaders. Here's what they said. Six Trends Shaping Developer Productivity Read More Hey, Nnamdi here. Developer productivity is undergoing a tectonic shift.

Why Don't VCs Index Invest?

Wednesday, July 8, 2020

Math and data say early-stage VCs should index invest, and late-stage investors should stock pick. Yet they do the opposite. Why Don't VCs Index Invest? Read More Hey, Nnamdi here. Math and data

Do College Degrees Matter for Software Engineers? Maybe

Saturday, June 20, 2020

Do college-educated developers earn more? Yes, but less than you might think Hey, Nnamdi here. It's a never ending debate across Reddit, Hacker News, and Stack Overflow: do college degrees matter

You Might Also Like

The maturation of Munich

Friday, March 29, 2024

Does Europe need the "next Silicon Valley"? The frantic search for buried hydrogen and a big raise for industrial automation. View in browser Morgan Stanley flagship logo Good morning there,

Inside Grindr's plan to squeeze its users

Friday, March 29, 2024

The company's bet on AI boyfriends and other paywalled features is making some employees nervous Platformer Platformer Inside Grindr's plan to squeeze its users By Zoë Schiffer • 28 Mar 2024

Inside Grindr's plan to squeeze its users

Friday, March 29, 2024

The company's bet on AI boyfriends and other paywalled features is making some employees nervous Platformer Platformer Inside Grindr's plan to squeeze its users By Zoë Schiffer • 28 Mar 2024

SaaSHub Weekly - Mar 28

Thursday, March 28, 2024

SaaSHub Weekly - Mar 28 Featured and useful products Loach logo Loach Say goodbye to unachieved OKRs with Loach, an OKR management solution that empowers your team to seamlessly align their daily work

84 new Shopify apps for you 🌟

Thursday, March 28, 2024

New Shopify apps hand-picked for you 🙌 Week 12 Mar 18, 2024 - Mar 25, 2024 New Shopify apps hand-picked for you 🙌 What's New at Shopify? 🌱 New login requirements for the Order Status Page Changed ⸱

We've Been Tinkering

Thursday, March 28, 2024

Join Our Beta! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏

AMA with Chiedo John, Director of Growth @ GitHub

Thursday, March 28, 2024

Live on Wednesday, April 3rd, starting at 9 AM PST͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌ ͏‌

red hot startup intel

Thursday, March 28, 2024

FULL SWOT analysis on Ramp, Databricks, Cohere, Devoted Health, and Coalition inside Scouting Reports on 🔥 Startups Download Now Hi there, Psst. Want competitive intel on the top 5 industry leading

The Ultimate Amazon Guide: Strategies For Sellers In 2024

Thursday, March 28, 2024

Learn from Melisa Vong for FREE in the Start Your Ecommerce Business Summit ! Another day, another exciting opportunity to introduce you to one of our 20+ ecomm trailblazers in the Start Your Ecommerce

[SaaS Club] Overcoming Early Struggles to Bootstrap to $22M ARR

Thursday, March 28, 2024

Hey Reader Let's connect on LinkedIn! Follow me to stay in touch! Here's a quick round up of what's been going on at SaaS Club: 🎧 Podcast Hubstaff: Overcoming Early Struggles to Bootstrap