How to Improve Your LLM : Combine Evaluations with Analytics
Tomasz TunguzVenture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. How to Improve Your LLM : Combine Evaluations with Analytics
The future of LLM evaluations resembles software testing more than benchmarks. Real-world testing looks like this, asking LLMs to produce Dad jokes like this zinger : I’m reading a book about gravity & it’s impossible to put down. Machine learning benchmarks like those published by Google for Gemini2 last week, or precision and recall for classifying dog & cat photos, or the BLEU score for measuring machine translation provide a high-level comparison of relative model performance. But this isn’t enough for a product team to be satisfied that their LLM-enabled product will perform well in the wild. LLMs are tricky. They don’t always provide the identical answer to the same or similar input. 1 can be greater than 4.. This is called non-determinism. How to solve this problem? To produce high quality LLM-products, teams will need to combine analytics with evaluation. Combining analytics with evaluation is the key to improving performance. Analytics surface the questions users ask when using the model. Those questions create the evaluations product teams use to determine performance. They gather additional data, retrain/fine-tune the model, & release it again. Today, evaluations are rule based or human-in-the-loop evaluations. But in the future, other models will judge the output to ensure consistency over time. And the iteration wheel improves ensuring that the Dad jokes from a model really are the best. |
Older messages
In Like a Lamb, Out Like a Lion
Wednesday, December 6, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. In Like a Lamb, Out Like a Lion Mongo announcing earnings
Top 10 Posts of 2023
Tuesday, December 5, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Top 10 Posts of 2023 These are the top 10 posts by engagement
Managing Data as Product : Office Hours with Philip Zelitchenko
Wednesday, November 29, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Managing Data as Product : Office Hours with Philip Zelitchenko
New Broadcast
Wednesday, November 22, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Title: The Evolution of Data Teams and Data Security Over the
Who Took a Bet on You?
Wednesday, November 22, 2023
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Who Took a Bet on You? I can remember at each career
You Might Also Like
Germany leapfrogs France in 2024
Monday, December 23, 2024
Plus: UK tech in review; 1Komma5's €150m pre-IPO raise View in browser Sponsor Card - Up Round-34 Good morning there, It's been a pretty good year when it comes to debt funding in Europe.
📸 How David Lieb Built Google Photos
Sunday, December 22, 2024
Dave's story is all about perseverance. From multiple pivots, to beating cancer, he's had quite the journey. This Week at YC December 22nd, 2024 ✨ The Latest Happy holidays from the YC team!
#211 | The Next Standard Oil, AI SDRs, Four Futures, & more
Sunday, December 22, 2024
Dec 22nd | The latest from IVP, Madrona, Khosla Ventures, Sapphire, USV, and others ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
This might be the simplest $12K/month app ever
Sunday, December 22, 2024
Starter Story Sunday Breakfast ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Behind the founder: Marc Benioff
Sunday, December 22, 2024
Salesforce CEO Marc Benioff on the beginners mind, AI, marketing, leadership, and life lessons from Steve Jobs ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
🎄 𝟖 𝐃𝐚𝐲𝐬 𝐎𝐧𝐥𝐲: 𝟖𝟓% 𝐎𝐅𝐅 + 𝟏 𝐅𝐑𝐄𝐄 𝐂𝐨𝐮𝐫𝐬𝐞!
Sunday, December 22, 2024
This 8-day holiday deal is the perfect gift for your 2025 self! fdrlogo Hey Friend , The holidays are here, and it's time to kickstart your journey toward an incredible 2025! Foundr's 8-Day
What’s 🔥 in Enterprise IT/VC #425
Saturday, December 21, 2024
Will business applications collapse in the agent era? Where does logic go + what are the "second order" effects when dozens or hundreds of agents are running amok at enterprises? ͏ ͏ ͏ ͏ ͏ ͏
🚀 The 2024 Space Stock Rally
Friday, December 20, 2024
Plus Synspective IPO's in Tokyo, $HON exploring strategic alternatives for Aerospace business, and more! The latest space investing news and updates. View this email in your browser The Space Scoop
🚨 Announcing: The inaugural “What’s in your stack?” survey
Friday, December 20, 2024
Tracking the most commonly used (and beloved) tools in tech ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
2025 Predictions
Friday, December 20, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. 2025 Predictions Every year I make a list of predictions &