How to Improve Your LLM : Combine Evaluations with Analytics
Tomasz TunguzVenture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. How to Improve Your LLM : Combine Evaluations with Analytics
The future of LLM evaluations resembles software testing more than benchmarks. Real-world testing looks like this, asking LLMs to produce Dad jokes like this zinger : I’m reading a book about gravity & it’s impossible to put down. Machine learning benchmarks like those published by Google for Gemini2 last week, or precision and recall for classifying dog & cat photos, or the BLEU score for measuring machine translation provide a high-level comparison of relative model performance. But this isn’t enough for a product team to be satisfied that their LLM-enabled product will perform well in the wild. LLMs are tricky. They don’t always provide the identical answer to the same or similar input. 1 can be greater than 4.. This is called non-determinism. How to solve this problem? To produce high quality LLM-products, teams will need to combine analytics with evaluation. Combining analytics with evaluation is the key to improving performance. Analytics surface the questions users ask when using the model. Those questions create the evaluations product teams use to determine performance. They gather additional data, retrain/fine-tune the model, & release it again. Today, evaluations are rule based or human-in-the-loop evaluations. But in the future, other models will judge the output to ensure consistency over time. And the iteration wheel improves ensuring that the Dad jokes from a model really are the best. |
Older messages
In Like a Lamb, Out Like a Lion
Wednesday, December 6, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. In Like a Lamb, Out Like a Lion Mongo announcing earnings
Top 10 Posts of 2023
Tuesday, December 5, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Top 10 Posts of 2023 These are the top 10 posts by engagement
Managing Data as Product : Office Hours with Philip Zelitchenko
Wednesday, November 29, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Managing Data as Product : Office Hours with Philip Zelitchenko
New Broadcast
Wednesday, November 22, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Title: The Evolution of Data Teams and Data Security Over the
Who Took a Bet on You?
Wednesday, November 22, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Who Took a Bet on You? I can remember at each career
You Might Also Like
Broken engine?
Saturday, September 7, 2024
Hey friends, I'm back from my two-week trip to the US. I've been meaning to write you an update about my experiences in the podcast industry, hanging out with my co-founder in Chicago, and
Hunch
Saturday, September 7, 2024
A note app for product people to help with the discovery process BetaList BetaList Daily Hunch A note app for product people to help with the discovery process Too much email in your life? Switch to
💄 Building a million-dollar beauty brand
Saturday, September 7, 2024
Alicia Scott's inspiring story Hey Friend , Does it ever feel like your entrepreneurial dreams are just out of reach? Maybe you feel like the problem you're solving is too niche. Or the market
3+ Years of daily posting on X: Top Lessons
Saturday, September 7, 2024
+ DNA School Invite
What’s 🔥 in Enterprise IT/VC #410
Saturday, September 7, 2024
Speed mode + also, is the great unbundling of SaaS powered by agentic workflows? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
🗞 What's New: Launch tips from Product Hunt's Head of Growth
Saturday, September 7, 2024
Also: Increasing your luck as a founder!
🚀 BlackSky To Reverse Split
Saturday, September 7, 2024
Plus Lynk Global delists from the Nasdaq, $ASTS mixed shelf offering, $PL and $SATX earnings, and more! The latest space investing news and updates. View this email in your browser The Space Scoop Week
my crazy idea…
Friday, September 6, 2024
Read time: 1 min. 18 sec. Today I want to tell you about something I've been working on. And to be honest… it might be the best thing I've ever built. Warning: Do not read this unless you'
Issue #125: Building $1K-$10K MRR Micro SaaS Products around Client Onboarding, AI Agent for online shopping, AI-b…
Friday, September 6, 2024
Build Profitable SaaS products!! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
How to write a blog post (according to the Buffer Content team)
Friday, September 6, 2024
Plus tips, news & Buffer updates for your social media journey ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏