Listen now on Apple, Spotify, and YouTube.

Brought to you by:

• WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs
• Productboard—Make products that matter
• Wix Studio—The web creation platform built for agencies

—

Keith Coleman (VP of product) and Jay Baxter (founding ML engineer), the minds behind Community Notes, reveal how a small, scrappy team inside Twitter/X built the most trusted crowdsourced information system on the internet—one that’s changing the way we understand truth online. What you’ll learn:

1. How Community Notes actually works—a deep dive into the groundbreaking algorithm that rewards “bridging agreement” instead of majority rule

2. The seemingly crazy yet brilliant way this idea survived multiple CEO changes—from Jack to Parag to Elon

3. How this project started with a dumpster fire GIF (literally)—the untold backstory of its early launch

4. The secret to running ultra-fast, high-impact product teams—no OKRs, no Jira; just one Google Doc

5. What Meta’s adoption of Community Notes means for the future of online (mis)information—why this open source system is becoming the industry standard

Find the transcript at: https://www.lennysnewsletter.com/p/how-x-built-the-best-fact-checking-system-on-the-internet

Some takeaways:

1. Key principles that enabled Community Notes’ success:

   1. Universal access: Random selection of contributors rather than curated experts

   2. Complete transparency: Open source code and data

   3. Focus on bridging divides: Show notes that people across the political spectrum find helpful

   4. Prove it works: Demonstrate success at each stage before expanding

   5. Let the people decide: No “god mode” or company override of community decisions

2. The system’s effectiveness stems from focusing on bridging-based agreement:

   1. Notes only appear when rated helpful by people who typically disagree

   2. About 8% of proposed notes meet the quality threshold

   3. Notes can reduce engagement with misleading content by 50% to 60%

   4. Authors are 80% more likely to delete posts after receiving a note

3. Small, focused teams can have outsize impact when structured correctly:

   1. Single clear owner/decision-maker

   2. 100% dedicated team members

   3. Freedom from standard corporate processes

   4. Start with minimal necessary roles (ML, frontend, backend, design, research)

   5. Use lightweight tools (e.g. a single Google Doc vs. complex project management systems)

4. Building trust through radical transparency:

   1. All code and data is public and downloadable

   2. External researchers can replicate and verify results

   3. Community members can propose algorithm improvements

   4. Regular public updates on system changes

   5. No hidden controls or override mechanisms

Where to find Keith Coleman:

• X: https://x.com/kcoleman

• LinkedIn: https://www.linkedin.com/in/keith-coleman-19b12b46/

Where to find Jay Baxter:

• X: https://x.com/_jaybaxter_

• LinkedIn: https://www.linkedin.com/in/jaybaxter/

• Website: http://jaybaxter.net/

In this episode, we cover:

(00:00) Introduction to Community Notes

(06:56) How the “bridging-based” algorithm works

(13:33) The impact and scale of Community Notes

(17:24) Understanding the note publishing threshold

(21:32) Challenges and philosophies

(26:26) The effect of notes on re-sharing content

(29:41) Origin story

(35:46) Embracing small teams for big impact

(40:23) The thermal project approach

(47:47) Algorithm development and internal competitions

(50:34) An inside look at how the team operates

(58:56) Working with Elon

(01:05:30) Launching Birdwatch

(01:10:48) The core principles behind Community Notes

(01:26:15) Anonymity and pseudonymity in contributions

(01:32:17) Sustaining the project through leadership changes

(01:37:57) Future directions for Community Notes

(01:42:12) Final thoughts and optimism for the future

Referenced:

• Community Notes on X: https://x.com/CommunityNotes

• Sign up to be a Community Notes contributor: https://communitynotes.x.com/guide/en/contributing/signing-up

• The Making of Community Notes: https://asteriskmag.com/issues/08/the-making-of-community-notes

• “Readers added a Community Note to this Tweet”: https://x.com/HelpfulNotes/status/1718103364792205704

• Note-ranking algorithm: https://communitynotes.x.com/guide/en/under-the-hood/ranking-notes#matrix-factorization

• Study: Community Notes on X could be key to curbing misinformation: https://giesbusiness.illinois.edu/news/2024/11/18/study--community-notes-on-x-could-be-key-to-curbing-misinformation

• Study Finds X’s (Formerly Twitter’s) Community Notes Provide Accurate, Credible Answers to Vaccine Misinformation: https://qi.ucsd.edu/study-finds-xs-formerly-twitters-community-notes-provide-accurate-credible-answers-to-vaccine-misinformation/

• Did the Roll-Out of Community Notes Reduce Engagement with Misinformation on X/Twitter?: https://dl.acm.org/doi/10.1145/3686967

• Kayvon Beykpour on LinkedIn: https://www.linkedin.com/in/kayvz/

• Jack Dorsey on X: https://x.com/jack

• “Birdwatch gives me the creeps” tweet: https://x.com/elonmusk/status/1589454464611540992

• Blake Scholl on LinkedIn: https://www.linkedin.com/in/blakescholl/

• Creating Truthtelling Incentives with the Bayesian Truth Serum: https://www.eecs.harvard.edu/cs286r/courses/fall12/papers/DW08.pdf

• Asana: https://asana.com/

• Spaces: https://blog.x.com/en_us/topics/product/2021/spaces-is-here

• Amazon MTurk: https://www.mturk.com/

• Community notes on GitHub: https://github.com/twitter/communitynotes

• What do I think about Community Notes?: https://vitalik.eth.limo/general/2023/08/16/communitynotes.html

• X’s community-led approach: tackling inaccurate and misleading information: https://blog.x.com/en_us/topics/company/2023/xs-community-led-approach-tackling-inaccurate-and-misleading-information

• Linda Yaccarino on LinkedIn: https://www.linkedin.com/in/lindayaccarino/

• Messi-Ronaldo rivalry: https://en.wikipedia.org/wiki/Messi%E2%80%93Ronaldo_rivalry

• Supernotes paper: https://arxiv.org/pdf/2411.06116v1

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

Lenny may be an investor in the companies discussed.