Tomasz Tunguz

If you were forwarded this newsletter, and you'd like to receive it in the future,

If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here.
How Much Does it Cost to Use an LLM?

How much does AI cost? It depends on a a few different dimensions, but for a typical use case where a user enters a query & expects about a 200 word response, the cost varies from $0.03 to $3.60.

Different models cost different amounts. Also, the size of the context window is an important factor.
The context window is a fancy way of saying the quantity of background information the user can provide to the AI.
When asking an AI to write a two-paragraph essay in the style of Hemingway, having a few sentences of his writing won’t help much. But perhaps uploading one-third of the Old Man & the Sea imbues Ernest’s style within the prose - no adverbs, simple direct sentences.

Longer tracts cost more. A 500 word essay totals $8.40 with the fanciest version of GPT-4 & $0.07 with Llama2, Facebook’s open-source model.
There is a 120x difference in cost between the top of the line AI & the free version.
Which is better? It depends on the use case & the amount of optimization one is willing to commit to the model.
These prices aren’t fixed. Earlier this week, OpenAI announced a 3x reduction in costs - a trend that will continue as engineers optimize the chips, the algorithms, & the systems for AI.
Relative to other infrastructure costs, these fees can be significant & understanding the gross-margin impact of this new layer of software will be essential, & will likely push startups from the more expensive larger models to more capital-efficient AI over time.

Tomasz Tunguz - How Much Does it Cost to Use an LLM?

​How Much Does it Cost to Use an LLM?​

Older messages

How Much Does it Cost to Use an LLM?