👋 Hey, I’m Lenny and welcome to a 🔒 subscriber-only edition 🔒 of my weekly newsletter. Each week I tackle reader questions about building product, driving growth, and accelerating your career.

P.S. Check out my recruiting service (helping you hire Sr. PMs and VPs of Product), Lennybot (an AI chatbot trained on my newsletter posts, podcast interviews, and more), and my swag store (great gifts for your favorite PM, or yourself!)

“How Duolingo reignited user growth,” by Jorge Mazal, is my #1 most popular post of all time. But the story didn’t end there. Growth at Duolingo has continued to go exponential, and VP of Engineering Sean Colombo has been there through it all.

In the past five years, Sean’s team has grown Duolingo’s daily active users from 5 million to nearly 30 million today (over 6x!), and the stock has nearly 3x’d in the past two years, largely thanks to methodical product iteration and experimentation. This re-acceleration is something you almost never see at a company that’s over a decade old, especially after a multi-year deceleration.

In this post, Sean shares five of his biggest growth lessons over these five years at Duolingo. If you’re looking for ideas and strategies to drive your own product’s growth, this post is for you.

For more from Sean, follow him on LinkedIn.

When we were bringing all our Growth teams under one org, I made an internal doc called “Colombo’s Musings on Growth” with some opinions on what I felt was driving our massive re-accelerated growth over the past few years and how we could keep up the momentum. The doc seemed to resonate well with my fellow Duos, and so this post is an attempt to share several of these musings with a larger audience—a continuation of Jorge’s post, in a way. It covers both strategic and tactical things that we learned during the growth we’ve sustained for over five years.

1. 📈 Maintain a sense of urgency to drive compound growth

One of the core things we’ve learned is that a sense of urgency leads to compounding growth.

To get the best long-term gains, you should always have a sense of urgency. The quicker you launch winning experiments, the quicker those changes impact your growth. Not only that, but these improvements compound!

At Duolingo, we are very fortunate to have strong word of mouth. Even with our extremely successful marketing (and partially because of it), about 90% of our DAU growth comes when new learners hear about us from friends, family, teachers, coworkers, and others. Because of this, DAUs benefit from some compound growth automatically. An even more pronounced compound growth effect is seen in retention metrics. 

An example

Say you have 100,000 DAUs and a retention rate of 80%, and for every five users, one of them tells a friend each day to download your app (and they do it). You get 20k new users per day just from word of mouth. In this equation, X_n is the number of DAUs today and X_n+1 is the number of DAUs tomorrow:

X_n+1 = (X_n * 80%) + (X_n * (⅕))

If you start with X_n = 100,000 DAUs, you’ll notice that the next day’s value (X_n+1) remains 100,000 DAUs—an equal number of users join and churn each day.

However, if we find an experiment that can improve retention by 1%, something interesting happens:

Day 0 (Sun.): 100,000 = X0
Day 1 (Mon.): (X0 * 81%) + (X0 * (⅕)) = 101,000 (which we will now call X1)
Day 2 (Tue.): (X1 * 81%) + (X1 * (⅕)) = 102,010 (this is X2, etc.)
Day 3 (Wed.): (X2 * 81%) + (X2 * (⅕)) = 103,030
Day 4 (Thu.): (X3 * 81%) + (X3 * (⅕)) = 104,060
Day 5 (Fri.): (X4 * 81%) + (X4 * (⅕)) = 105,100
Day 6 (Sat.): (X5 * 81%) + (X5 * (⅕)) = 106,151
Day 7 (Sun.): (X6 * 81%) + (X6 * (⅕)) = 107,213
Day 8 (Mon.): (X7 * 81%) + (X7 * (⅕)) = 108,285

You’ll notice that between Sunday and Monday the first week, we added 1,000 DAUs. One week later we added 1,072 DAUs that day because there were more users to spread the word.

Whatever experiment it was that resulted in this 1% gain to retention is already adding 72 more people each day just because it was launched one week prior. After two or three weeks, the effect continues to grow compared to that first day. At Duolingo we launch hundreds of experiments per quarter, so this type of impact can really add up.

OK, so compound growth is great, but how does this tie in with a sense of urgency at Duolingo? We run experiments on almost everything we do, and launching these experiments quickly has a meaningful impact on our long-term success.

Compound interest in action—in the real totally hypothetical world

Let’s take a completely hypothetical example from an app called Luodingo that had two major feature launches. 

Feature A they ported from iOS to Android quickly, immediately after they launched the win on iOS. Including development time, experiment rollout, and running the experiment for two weeks to get data, they ended up launching the port of Feature A 82 days later. 

Feature B had a similar impact in its initial experiment, but the team did two small iterations to polish it before they ported it to Android. The iterations were roughly neutral, but they made Feature B into something that the Luodingans were more proud of (and of course they hoped the results might also be positive). The Android port of Feature B was launched 184 days after the launch of Feature B on iOS.

Let’s see the impact here from waiting to port:

• 184 days to launch vs. 82 days, so 102 extra days that didn’t have the port live.

• The port showed a gain of roughly 21k DAUs per day over a two-week period.

• There were about 7.3 of these two-week periods in the 102-day gap between the two potential approaches.

• This means that roughly 7.3 * 21k, or 153.3k DAUs, were lost on an average day during the period. (Note that this is just roughly or directionally accurate; you can’t actually multiply experiment results by time completely linearly like this. Please forgive me, Data Platform and Data Science & Analytics teams! 🙏)

• At this totally hypothetical company, they tend to get one new user each day from word of mouth for every 50 DAUs, on average.

• During this period, that means they were missing out on an average of (153,000/50), or 3,000 new users, per day from word of mouth! This is on top of the average of 153.3k DAUs that they lost from not having Feature B live on Android during that time.

• The moral of the story is that the team probably worked really, really hard coming up with other experiments, writing specs, implementing, etc. to try to get gains equal to the amount of DAUs they could have gotten by just porting this win in a different order.

• I would clearly never have made this mistake and it’s obviously not a completely real-world example from a team I was responsible for. 0:-)

To ensure that we’re moving quickly and striving for growth without sacrificing quality, we use these guidelines that you can also apply at your org: 

1. Make a decision as soon as you’re confident you have the data to do so. At any given point, teams often have several experiments running at once while also working to ship upcoming features. Teams might let experiments sit on the back burner to “collect more data” and check in on them every now and then. Having the discipline to carefully and regularly examine experiments to know when you have sufficient data to launch them means that you’ll get the maximum impact as soon as possible.

2. Roll out experiments to as many learners (aka users) as you can, as quickly as you can. In order to have sufficient confidence in experiment results, we need a ton of users treated into experiments. Many experiments start with a very small number of people who are allowed to see the new experiment at the start, and, as I noted above, there is often no pressure to ramp this up while the team is working on the next big thing. But it’s important to increase the rollout as soon as you’re confident that a change is safe. The sooner you get the data that you need to make a decision about the experiment, the sooner you’ll launch it and start compounding the gains.

3. Don’t pull your punches. If you have a good new feature but there are some things you want to change about it, don’t just shut it down, make a tweak, and then run another experiment. If the imperfect version is strictly an improvement over the prior experience, launch it to new users (which starts the compound gains) and then make the improvements through iterative experiments after that. The amount of re-work saved by only launching afterward is likely very low, compared with the benefit of having the feature released. It also means that users on the various platforms will have a more consistent experience, which is valuable.

4. Consider return on investment when determining the order of experiments. There are many factors that team leads consider when prioritizing projects, but I’d argue that doing the most efficient, high-ROI projects first is the right path, because they will have a longer period of paying compound interest. When teams do quarterly planning, they often overlook order and focus on the list of things they want to accomplish in a three-month period. Order matters!

5. Launch on your biggest platforms first. If you have significantly more users on Android, launching experiments on Android first and then porting to iOS will give you more gains in the long run than if you launch on iOS first and then port to Android. If you launch to the bigger pool of users first, you have bigger compound gains than if you do it in the other order.

6. Port the wins. If your company doesn’t have to run experiments on all platforms at once, it’s probably faster to try things on one platform and then port to others—and this is the most common approach. Porting wins quickly is a huge opportunity to reap the benefits of compound growth. At Duolingo we launch around 50% of experiments. Experiments that are ports, however, are much closer to 100% launched. Nearly everything that wins on one platform wins on the other. Users aren’t that different between platforms.

Of course, many reasonable people (and companies) gather as much data as possible, wait to launch until things are perfect, and expose experiments to only a small number of users for a long time. And these companies can also succeed. However, our experience at Duolingo has shown me that our approach works very well over the long term. Keeping a sense of urgency leads to fantastic compound growth.

If the concept of urgency resonates with you and you want to take it to an entirely new level, it’s worth reading Frank Slootman’s book Amp It Up or his LinkedIn article to get a shorter version of his ideas.

2. ♟️ Identify your strategic advantage

Finding your strategic advantage can be really helpful in guiding your product strategy. At Duolingo, our strategic advantage is that our users want to build a habit.

Our CEO, Luis von Ahn, has said many times that our competition might actually be Instagram, TikTok, and other ways for people to just pass the time on their phones. While social networking and games have their own advantages, we have one that those other apps don’t: our learners want to build a habit on Duolingo. Nobody downloads Royal Match thinking, “I hope I keep up playing this game for hours each day.” But our users have a conscious desire to learn another language, so they’re a partner in our goal to retain them.

This strategic advantage plays out well in a number of ways. One especially important thing I’ve learned is that if we give users useful notifications (with an easy way to turn them off for those who don’t find them valuable), those tend to be well-received.

Please note that this doesn’t mean we should spam users. (See the section below on notifications and the golden goose.)

The best book I’ve read about strategy is Good Strategy/Bad Strategy by Richard Rumelt. Conveniently, the author was also on Lenny’s Podcast recently discussing strategy. Check it out!

While it’s definitely worth reading the whole book, a very short explanation of how to find your strategic advantage would be:

Identify a unique strength that your company/product has that aligns well with the current opportunities in your environment. Then leverage that strength in a way that others cannot easily replicate. Focus on what sets you apart and use it to create a sustainable competitive advantage.

Our learners wanting to build a habit gives us an advantage over other ways they could spend their time. We built on that advantage, and now we have a sustainable lead over language learning apps and other learning apps, too.

For us, identifying our strategic advantage pointed us toward iterating on notifications and the streak system (which helps build the habit that our users want to build). When you find your strategic advantage, it will point you in the direction of product changes that should have outsize results for your business. First, read the Rumelt book mentioned above, figure out your strategic advantage, and then use it to home in on actionable product changes.

3. Copy first, then innovate

🚲 Don’t reinvent the wheel when it comes to app mechanics

There are literally millions of mobile apps, and the biggest of them have billions of users. The larger ones have run A/B tests with a combined total of billions of treatments. Furthermore, the app marketplace itself could be seen as a genetic algorithm where winning concepts end up with more users, which means they inspire the next set of app developers, and so on.

This means that if you would like to use a concept that’s already widespread in the industry, your first approach should be to start with a system that closely resembles an existing successful one, but adapted to your app. A great example for Duolingo was our leaderboards. Many apps, especially casual mobile games, have excellent leaderboards. They’ve spent tons of time refining them, so we rightly assumed we could find a version of these that was effective for us, too. Our current leaderboard system was our fourth iteration, but it blew away our prior attempts.

We based it heavily on successful leaderboards in other games such as Gardenscapes, Golf Clash, Toon Blast, and more. The winning leaderboard for us (our fourth iteration) was an opt-out experience that put people in a new group of 30 each week, where a number of people get promoted or demoted to higher or lower leagues each week. The leagues automatically tuned people to be with others of a similar difficulty. Along with rewards for the top three places, this created several different interesting boundaries in the group of 30, where many users are always close to either gaining or losing something based on their position. 

It took a ton of great product and design work to adapt these ideas to fit in Duolingo. Our leaderboard system was the most complicated feature we’d added to Duolingo, but we had to make sure that people could figure it out without pop-up explainers. But it was worth the effort: D1 retention went up 1%, D7 retention went up 2%, and D14 retention went up 3%. Additionally, the time people spent learning increased by around 17%, which was an absolutely amazing result for us—typically if our experiments can move a major metric by 1%, that’s a really good outcome.

These sorts of lessons can be applied to many other very common game mechanics that work well in some apps and platforms and poorly in others. For example: achievements, quests, badges, cosmetics. I even think Duolingo could often do better at following our own advice.

Your first minimum viable product (MVP) version of an existing, widespread mechanic is not the time to get clever and innovative.

💡 When should we innovate?

There are two scenarios where I think you should try to be clever and start to push the envelope:

1. When you’re already at the cutting edge. Duolingo’s notifications are a good example of this scenario. We became well-known as leaders in notifications early on, and since then we’ve pushed boundaries by using a bandit algorithm, making UIs to let non-engineers run copy experiments, and creating a way to send millions of notifications within seconds of a button push, which have all been important wins. While there are still opportunities to pull in new things from the industry over time (like we did with adding emoji to notifications when others found it to be effective), it is important to continue pushing our advantage rather than waiting for others to do the innovation that we can copy. We’re on the cutting edge with our experimentation tooling, and our Experiments team is constantly trying things in our tooling where there isn’t a lot of prior art.

   One analogous example I like to keep in mind here is early Netflix. Netflix put Blockbuster out of business with their mail-in DVD system, but then Netflix became a pioneer in an even more advanced delivery system (streaming) and successfully put their own mail-in market out of business. By making sure they were the continued innovators on delivery of home movies and TV series, they maintained the dominant position even through a massive pivot in the industry.

The bandit algorithm

Our quirky notifications are a hugely important lever for us, and one of our most successful iterations has been to add a bandit algorithm, which automatically scores various notifications to see how effective they are at bringing people back to Duolingo to do a lesson shortly after they are received.

See this blog post and this academic paper to geek out on more details.

2. After launching a solid MVP based on best-in-class industry standards. An MVP of our leaderboard system still took us months to design and implement—largely because our app is clean and simple, and leaderboards were complex. After that, we still tried out other aspects of popular leaderboards that were too advanced to make it into the MVP. Once we got to a point where we had either implemented—or at least experimented with and rejected—the major features of other leaderboards, it made sense to start exploring our own independent ideas. (One of our more impactful innovations there has been how we group people into batches, but the technical details are a good topic for a completely different type of post.)

4. 🦢 Notifications are the goose that lays the golden eggs—don’t kill the goose

This won’t apply to all apps, but at Duolingo, notifications have been a source of continued small-to-medium wins year after year, with no obvious signs of slowing down.

1. Modifying existing notifications and even (rarely and carefully) adding new, useful notifications has been a nearly infinite source of gains. We’ve been improving notifications for years, and they still deliver consistent wins that affect a huge slice of the pie (see the next section for more on this). These experiments are golden eggs!

2. The notification channel itself is the goose. If we send notifications that users do not find interesting, that they don’t think are worth reading, or that cause them to unsubscribe, we will destroy the channel and kill the goose. There’s a story that we’ve heard from talking to people from Groupon: When they ran experiments that sent additional emails, every experiment showed positive metrics. So they sent more and more emails. Eventually, though, the channel started to rot and they had all sorts of problems—users would unsubscribe, emails would go to the spam folder and delivery rates dropped, and people just generally didn’t bother to read the emails anymore.

Some things we do at Duolingo to avoid killing the goose:

1. We set a very high bar for how successful an additional notification has to be. We’ve killed some notifications that were huge wins but weren’t big enough to justify the additional level of spamminess. Here are two concrete examples of when this happened and what we did instead:...