Top of the lyne - Growth ❤️ Experimentation
Growth ❤️ ExperimentationThe hypothesis-led experimentation engine that powers the modern growth teamGalileo made conclusions about the speed of falling bodies, Lavoisier derived his theories of conservation of mass, and Louis Pasteur fleshed out the germ theory of diseases all on the shoulders of the father of empiricism: Francis Bacon. Francis Bacon pioneered the quest for repeatable observations through experiments and laid the groundwork for the scientific method as we know it today. “The true method of experience first lights the candle (builds the hypothesis), and then by means of the candle shows the way”, writes Bacon in Novum Organum (New Instrument), “not bungling or erratic, and from it deducing axioms, and from established axioms again new experiments.” The Baconian method soon evolved to become a scientific substitute for the way truths were established in societies. Bacon’s empirical method dictated a description, classification, and acceptance of connected facts and replaced a prevailing reliance on fiction, guesses, and the citing of curiously absent authorities. Bacon, for instance, argued that healthy people should be exposed to outside influences like the cold, rain, and other sick people - and that the outcome of these experiments must be tested for repeatability before the truth could be known. Centuries have passed since Francis Bacon, and the last you heard of him was likely through this Reddit post, but the methods he championed have proved to be highly transferable. Today, the core driver of growth for the fastest-growing businesses in the world is that spirit: that of experimentation. 🌟 “Scientific method applied to KPIs.”Growth - A function that took birth in Facebook's hallowed Menlo Park office and later popularized by its success at Airbnb, Uber, Pinterest, and more, is now a staple at all your favorite technology companies. You might know us as people chasing answers for "How is our product growing?" or more specific ones like "How are we acquiring users?", "How can we retain them?", "How can we get them to pay us more?". Given that this is a growth newsletter, a lot of you are probably growth professionals yourself. Brian Balfour, Founder of Reforge and ex - VP Growth at Hubspot roots the need for a dedicated growth team in the following market changes:
Connecting users to the underlying value in a product (aka Distribution) has become a cross-functional effort that rapidly (but scientifically) skips between channels and tactics, solving customer problems by combining data and channel expertise.
This new approach to distribution and driving product adoption needed a new skill set and a specialized team - The Growth Team, equipped with a specialized Growth Process - one of Hypothesis led experimentation.
Growth teams,
Let's dig deeper into a day in the life of a Growth Experiment 🌞 🤳 But first, let me make a hypothesisHypothesis | pronounced: hʌɪˈpɒθɪsɪs/ | noun - "a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation." also - the cornerstone of scientific experimentation While ideas are plagued with biases, myths, and false facts, a hypothesis is clear, scientific, and testable.
Prioritization: Growth teams create multiple hypotheses that can solve varied problems and these are converted into individual tests prioritized by reach 🟢, impact 🟢, confidence 🟢 and effort 🔴 and are accordingly added to the experimentation roadmap. These experimentation roadmaps are managed in workspaces for scheduling, executing, and reporting ease. The chosen ones 🧙🏻♀️ enter the next phase - the testing stage! 👨🏻🔬 🪄 The potions and the spells
Growth experiment tests mimic software deployment processes like remote config and feature flags (we pinged our engineering team for this 🤭) that enable personalization and rolling releases. Control and variant versions (a modification, a new addition, omission from the normal, etc) are shown to a homogenous split of unsuspecting users and the hypothesis is validated by observing changes in the outcome metric between the two versions. A test could be any of the following: From Surveymonkey changing copy to increase adoption of their "Audience collector” feature
to Uber's full-fledged Uber-XP experimentation platform that runs 1000 experiments at any point in time The underlying recipe 👨🏻🍳, however, remains the same: 🛠 Identify the test method - Different experiments warrant different test methods. A/B tests (1 change), Multivariate tests (multiple changes), and Split URL testing (redirecting traffic to an entirely different page) are methods that can be used to test out different hypotheses. 👨🏻💻 Creating the control and the variant - Different versions are created either on the client-side (variants live and render on the user’s browser) or are built separately on the server-side. Config settings and feature flags ensure that the audience is split equally (or as prescribed) between the two versions. 📈 Metrics to be measured - Well-defined primary and secondary objective metrics and the necessary instrumentation to report them are set up.
👥 Segmentation and targeting - The audience for a test on certain occassions needs to be segmented basis basic user properties like - geography, device used, browser used, and sometimes with more complex product usage triggers. User segmentation can also be used to restrict the test to a small sample of users enabling multiple parallel tests that are independent of each other. 📊 Statistical significance - determining the minimum sample size required for accurate test results and for when a test can be considered complete. The statistical significance approach called the frequentist method( p-value & statistical power) or the Bayesian approach are used for the same. 🔍 Analysis and reporting - Dashboards, heatmaps, and other visualization tools for reading test results. While Uber and Spotify have built out internal tools and dashboards for running these experiments, many plug-and-play solutions exist that allow Growth teams to execute and analyze tests with minimal setup time. Last but not the least, let’s discuss who mans this engine… Growth hackers!? 🤨 😤 "No! We're not hackers"While our ancestry can be traced back to solo magicians who moved numbers with unconventional ‘hacks’, growth today is a team game. Organized in siloes or alternatively distributed across the product, design, engineering, and marketing verticals, growth teams are cross-functional machinery capable of managing, implementing, and analyzing an experimentation engine. It’s very important to note that successful growth teams are backed by an organization-wide buy-in to the experimentation way of life and are provided the required resources and autonomy to succeed. 🤝 Some housekeeping… Is your mailbox trying to keep our content away from you! What can you do about it? Mark this email as ‘not spam’ or move it from your promotions to the primary folder It’s very easy! Thanks again, and please tell a few friends if you feel like it. Originally published on the Toplyne blog If you liked this post from Top of the Lyne, why not share it? |
Key phrases
Older messages
Announcing our new integration with Uber
Friday, April 1, 2022
Toplyne now integrates with Uber (seriously)
Black Mirror's SaaS collab, Google Maps for Zelda, Super(loom)inal communication, and more...
Thursday, March 31, 2022
👋 Hey, Ruchin here! Welcome to this week's edition of the ✨Top of the Lyne✨ newsletter. Each week we publish the hottest news from the beautiful world of product-led growth! (PLG) The outcome we
Lusha and the science of super-targeted prospecting
Tuesday, March 29, 2022
The story of the sales-intelligence platform's hyper-growth to a $1.5B valuation in five years while building a community of 800k+
Meme led growth, NRR solves all problems, preaching to the choir, and more...
Thursday, March 24, 2022
👋 Hey, Ruchin here! Welcome to this week's edition of the ✨Top of the Lyne✨ newsletter. Each week we publish the hottest news from the beautiful world of product-led growth! (PLG) The outcome we
Clash of Clan's elixir and gold - Product Led Growth
Tuesday, March 22, 2022
What SaaS Product-led companies can learn from Clash of Clans
You Might Also Like
10words: Top picks from this week
Friday, April 19, 2024
Today's projects: Spheria • Narration Box • OnTheFly • AnyCheese • Dumbbe • HASTY • lockrScan • IdeaAize • Mailmo • ProSolvio • GigHunterBot • WeekToDo 10words Discover new apps and startups in 10
[Replay] Building a million dollar beauty brand
Friday, April 19, 2024
Free summit replay extended Hey - great news! We've extended the summit replay for 24 more hours. Watch it here free >> And, I want to highlight one of the inspiring founders, Alicia Scott.
Put it on paper
Friday, April 19, 2024
In case you missed it, here is the full recording of the 668 person workshop I co-hosted yesterday "how to build products of the future". It's completely free to watch here. Put it on
7 prompts for your B2B company’s next LinkedIn post
Friday, April 19, 2024
Plus, tips, news & Buffer updates for your social media journey Image Hey folks 👋🏾 It's been an interesting week in social, with the launch of TikTok Notes (their take on a photo + text
Challenges of Offering an API — The Bootstrapped Founder 313
Friday, April 19, 2024
When you sell access to an API, what should you keep in mind? How would you protect your data but still make it easy to use?
12 hours to go! 🐣
Friday, April 19, 2024
Catch the super early bird rates before they expire! View this email in your browser. 1200 x 380 (Desktop & Tablet) (8) Hey there, There are just 12 hours to go before Summit Summit ticket prices
Why I Switched My Blog to Ghost (and Why You Might Love It Too)
Friday, April 19, 2024
Tired of slow, clunky blogging platforms? I was too! Find out how Ghost changed my blogging life (and my company!), and why it might be the upgrade you need. Salehin Khan Salehin Khan Why I Switched My
Latvia's startup financing gets creative
Friday, April 19, 2024
We visit Riga to explore Latvia's tech scene, layoffs at Stability AI and why climate tech's latest phase will be built on debt. View in browser Notion flagship logo final Good morning there,
How Meta is paving the way for synthetic social networks
Friday, April 19, 2024
Five ways of thinking about Llama 3, its latest large language model Platformer Platformer How Meta is paving the way for synthetic social networks By Casey Newton • 18 Apr 2024 View in browser View in
SaaSHub Weekly - Apr 18
Thursday, April 18, 2024
SaaSHub Weekly - Apr 18 Featured and useful products Ant Design logo Ant Design An enterprise-class UI design language and React implementation with a set of high-quality React components, one of best