Tomasz Tunguz - Agentic Systems' Sales Cycles
Tomasz TunguzVenture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Agentic Systems' Sales Cycles
As software startups begin to sell agentic systems, the procurement process will change. Unlike classical software, where the application either meets the criteria (price, integration into other software, particular features) or doesn’t, agentic systems operate on a performance continuum. Here’s a recent evaluation table for Codestral, Mistral’s open-source code generation AI. All of these benchmarks are machine-generated : HumanEval & HumanEvalFIM are not human testers - but open-source projects that evaluate AI code.1 This type of evaluation works well for broad sense of relative performance. But what if a business writes code in a particular language? Or with particular performance characteristics in mind? What if an AI-powered customer support agent needs to be able to manage very technical telecom queries? Or a marketing AI needs to be culturally sensitive to a particular region? The generic tests probably won’t work, which translates to slower sales cycles as prospective buyers understand the system’s performance in their own context. In addition, agentic systems in the future will operate for longer periods of time without human intervention. The greater the autonomy, the greater the potential for errors. Benchmarks may not be enough; buyers may want to see how the system performs in their own context over time. Startups - as they always do - will find ways to accelerate the evaluation. They might develop their own standards much the way that OpenAI has, or partner with third-parties to offer those third party evaluations for particular use-cases. Imagine a modern day Gartner for Agentic Systems, a company that maintains a diverse pool of human evaluators & computer scientists skilled in various the evaluation of agentic products. Alternatively, the most sophisticated organizations could create standards that then become broadly adopted. Banks could publish open-source standards for regulator-compliant customer support chatbots. This purchasing behavior does exist elsewhere. Backtesting is the norm in trading algorithms & marketing optimization. Within the most sophisticated security organizations, security labs exist to test machine learning-based security products and performance before deploying them. In certain cases, the business need will overwhelm the procurement process. This happens in classic software & it will happen with AI but it’s rarer. However the problem is solved, agentic systems will evolve the procurement process & startups will need to navigate it. 1 OpenAI created both of these tests to measure the accuracy of its code generation model & now it’s a standard for evaluating AI code generation models. |
Older messages
The Future of Blockchain Data : Our Investment in Allium
Thursday, July 18, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. The Future of Blockchain Data : Our Investment in Allium Large
Punctuated Equilibrium in AI : Is it Better to Be A First Mover or A Last Mover?
Monday, July 15, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Punctuated Equilibrium in AI : Is it Better to Be A First Mover
AI Agent Pricing
Friday, July 12, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. AI Agent Pricing In a world where AI agents are 2.5-3x as
AI Pricing Strategies for SaaS Companies Offering Copilots
Thursday, July 11, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. AI Pricing Strategies for SaaS Companies Offering Copilots
Select avg(Moby Dick) limit 2 sentences
Monday, July 8, 2024
Tomasz Tunguz Venture Capitalist If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here. Select avg(Moby Dick) limit 2 sentences The SQL statement
You Might Also Like
Broken engine?
Saturday, September 7, 2024
Hey friends, I'm back from my two-week trip to the US. I've been meaning to write you an update about my experiences in the podcast industry, hanging out with my co-founder in Chicago, and
Hunch
Saturday, September 7, 2024
A note app for product people to help with the discovery process BetaList BetaList Daily Hunch A note app for product people to help with the discovery process Too much email in your life? Switch to
💄 Building a million-dollar beauty brand
Saturday, September 7, 2024
Alicia Scott's inspiring story Hey Friend , Does it ever feel like your entrepreneurial dreams are just out of reach? Maybe you feel like the problem you're solving is too niche. Or the market
3+ Years of daily posting on X: Top Lessons
Saturday, September 7, 2024
+ DNA School Invite
What’s 🔥 in Enterprise IT/VC #410
Saturday, September 7, 2024
Speed mode + also, is the great unbundling of SaaS powered by agentic workflows? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
🗞 What's New: Launch tips from Product Hunt's Head of Growth
Saturday, September 7, 2024
Also: Increasing your luck as a founder!
🚀 BlackSky To Reverse Split
Saturday, September 7, 2024
Plus Lynk Global delists from the Nasdaq, $ASTS mixed shelf offering, $PL and $SATX earnings, and more! The latest space investing news and updates. View this email in your browser The Space Scoop Week
my crazy idea…
Friday, September 6, 2024
Read time: 1 min. 18 sec. Today I want to tell you about something I've been working on. And to be honest… it might be the best thing I've ever built. Warning: Do not read this unless you'
Issue #125: Building $1K-$10K MRR Micro SaaS Products around Client Onboarding, AI Agent for online shopping, AI-b…
Friday, September 6, 2024
Build Profitable SaaS products!! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
How to write a blog post (according to the Buffer Content team)
Friday, September 6, 2024
Plus tips, news & Buffer updates for your social media journey ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏