The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
Brought to you by Mixpanel—Event analytics that everyone can trust, use, and afford | Round—The private network built by tech leaders for tech leaders | Eppo—Run reliable, impactful experiments — Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon. He was also honored with a lifetime achievement award by the Experimentation Culture Awards in September 2020 and teaches a popular course on experimentation on Maven. In today’s podcast, we discuss: • How to foster a culture of experimentation • How to avoid common pitfalls and misconceptions when running experiments • His most surprising experiment results • The critical role of trust in running successful experiments • When not to A/B test something • Best practices for helping your tests run faster • The future of experimentation — Enroll in Ronny’s Maven class, Accelerating Innovation with A/B Testing, at https://bit.ly/ABClassLenny. Promo code “LENNYAB” will give $500 off the class for the first 10 people to use it. — Listen now on Apple, Spotify, Google, Overcast, and YouTube. Find the transcript for this episode and all past episodes at: https://www.lennyspodcast.com/episodes/. Today’s transcript will be live by 8 a.m. PT. Where to find Ronny Kohavi: • Twitter: https://twitter.com/ronnyk • LinkedIn: https://www.linkedin.com/in/ronnyk/ • Website: http://ai.stanford.edu/~ronnyk/ Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • Twitter: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ In this episode, we cover: (00:00) Ronny’s background (04:29) How one A/B test helped Bing increase revenue by 12% (09:00) What data says about opening new tabs (10:34) Small effort, huge gains vs. incremental improvements (13:16) Typical fail rates (15:28) UI resources (16:53) Institutional learning and the importance of documentation and sharing results (20:44) Testing incrementally and acting on high-risk, high-reward ideas (22:38) A failed experiment at Bing on integration with social apps (24:47) When not to A/B test something (27:59) Overall evaluation criterion (OEC) (32:41) Long-term experimentation vs. models (36:29) The problem with redesigns (39:31) How Ronny implemented testing at Microsoft (42:54) The stats on redesigns (45:38) Testing at Airbnb (48:06) Covid’s impact and why testing is more important during times of upheaval (50:06) Ronny’s book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (51:45) The importance of trust (55:25) Sample ratio mismatch and other signs your experiment is flawed (1:00:44) Twyman’s law (1:02:14) P-value (1:06:27) Getting started running experiments (1:07:43) How to shift the culture in an org to push for more testing (1:10:18) Building platforms (1:12:25) How to improve speed when running experiments (1:14:09) Lightning round Referenced: • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing: https://experimentguide.com/ • Seven rules of thumb for website experimenters: https://exp-platform.com/rules-of-thumb/ • GoodUI: https://goodui.org • Defaults for A/B testing: http://bit.ly/CH2022Kohavi • Ronny’s LinkedIn post about A/B testing for startups: https://www.linkedin.com/posts/ronnyk_abtesting-experimentguide-statisticalpower-activity-6982142843297423360-Bc2U • Sanchan Saxena on Lenny’s Podcast: https://www.lennyspodcast.com/sanchan-saxena-vp-of-product-at-coinbase-on-the-inside-story-of-how-airbnb-made-it-through-covid-what-he8217s-learned-from-brian-chesky-brian-armstrong-and-kevin-systrom-much-more/ • Optimizely: https://www.optimizely.com/ • Optimizely was statistically naive: https://analythical.com/blog/optimizely-got-me-fired • SRM: https://www.linkedin.com/posts/ronnyk_seat-belt-wikipedia-activity-6917959519310401536-jV97 • SRM checker: http://bit.ly/srmCheck • Twyman’s law: http://bit.ly/twymanLaw • “What’s a p-value” question: http://bit.ly/ABTestingIntuitionBusters • Fisher’s method: https://en.wikipedia.org/wiki/Fisher%27s_method • Evolving experimentation: https://exp-platform.com/Documents/2017-05%20ICSE2017_EvolutionOfExP.pdf • CUPED for variance reduction/increased sensitivity: http://bit.ly/expCUPED • Ronny’s recommended books: https://bit.ly/BestBooksRonnyk • Chernobyl on HBO: https://www.hbo.com/chernobyl • Blink cameras: https://blinkforhome.com/ • Narrative, not PowerPoint: https://exp-platform.com/narrative-not-powerpoint/ Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com. Lenny may be an investor in the companies discussed. |
Older messages
How Shopify builds product
Tuesday, July 25, 2023
VP of Product Glen Coates on how Shopify plans around yearly themes, organizes around jobs to be done, tracks using a homegrown tool, why they shifted away from a GM structure, and much more
The 10 traits of great PMs, how AI will impact your product, and Slack’s product development process | Noah Weiss …
Sunday, July 23, 2023
Listen now (86 min) | Brought to you by Sidebar—Catalyze your career with a Personal Board of Directors | Superhuman—The fastest email experience ever made | Vanta—Automate compliance. Simplify
How today’s top consumer brands measure marketing’s impact
Tuesday, July 18, 2023
Lessons from studying how 40+ brands measure their marketing impact, including McDonalds, H&M, TikTok, Amazon, Airbnb, and Uber
M&A, competition, pricing, and investing | Julia Schottenstein (dbt Labs)
Sunday, July 16, 2023
Listen now (61 min) | Brought to you by Vanta—Automate compliance. Simplify security | Superhuman—The fastest email experience ever made | AssemblyAI—Production-ready AI models to transcribe and
LinkedIn’s product evolution and the art of building complex systems | Hari Srinivasan (LinkedIn)
Sunday, July 16, 2023
Listen now (65 min) | Brought to you by Miro—A collaborative visual platform where your best work comes to life | Brave Search API—An independent, global search index you can use to power your search
You Might Also Like
Shiftx AI, View AI Ready Data as a Service (DaaS), and hanabi.rest
Friday, September 20, 2024
AI-based API building platform BetaList BetaList Daily Shiftx AI Improve your processes with AI View AI Ready Data as a Service (DaaS) Get Your Company's Data Ready for AI hanabi.rest AI-based API
Will We See a Better Exit Market Next Year?
Friday, September 20, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Will We See a Better Exit Market Next Year? The Fed cut rates
Tips to help make consistent content creation a habit
Friday, September 20, 2024
Plus tips, news & Buffer updates for your social media journey ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Issue #126: Building $1K-$10K MRR Micro SaaS Products around Automated Address Change Management, Rundown Creation…
Friday, September 20, 2024
Build Profitable SaaS products!! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
10words: Top picks from this week
Friday, September 20, 2024
Today's projects: Relay.app • memoiri • Strike Analytics • Notion Finance Tracker Template • OpenLang • Ruly • Ad Agency • Homer • Super Agent • PhotoFairy • Revnue • VideoTrim.app 10words Discover
🚨 48 hours left - are you in?
Friday, September 20, 2024
Don't get left behind - $100m lesson! Hey Friend , You've been waiting for the right moment to start your own premium ecommerce business… Well, this is it. In about 48 hours, the Start Your
The AI-Powered Solopreneur — The Bootstrapped Founder 347
Friday, September 20, 2024
AI is revolutionizing the way I work as a solopreneur. So why not share exactly how I use it?
Let's talk about E-S-G
Friday, September 20, 2024
Plus: The French billionaire upping his tech investments; latest deals View in browser Silo flagship Good morning there, Yesterday at one of London's central schmoozing spots a couple hundred
🗞 What's New: YouTube just launched Hype to help small creators get discovered
Thursday, September 19, 2024
Also: Running newsletter ads!
SaaSHub Weekly - Sep 19
Thursday, September 19, 2024
SaaSHub Weekly - Sep 19 Featured and useful products Schedul Threads logo Schedul Threads Boost your Threads following, reach monetization status fast and automate your Threads content publishing for