If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here.​

​How Much Does it Cost to Use an LLM?​

How much does AI cost? It depends on a a few different dimensions, but for a typical use case where a user enters a query & expects about a 200 word response, the cost varies from $0.03 to $3.60.

Different models cost different amounts. Also, the size of the context window is an important factor.

The context window is a fancy way of saying the quantity of background information the user can provide to the AI.

When asking an AI to write a two-paragraph essay in the style of Hemingway, having a few sentences of his writing won’t help much. But perhaps uploading one-third of the Old Man & the Sea imbues Ernest’s style within the prose - no adverbs, simple direct sentences.

Longer tracts cost more. A 500 word essay totals $8.40 with the fanciest version of GPT-4 & $0.07 with Llama2, Facebook’s open-source model.

There is a 120x difference in cost between the top of the line AI & the free version.

Which is better? It depends on the use case & the amount of optimization one is willing to commit to the model.

These prices aren’t fixed. Earlier this week, OpenAI announced a 3x reduction in costs - a trend that will continue as engineers optimize the chips, the algorithms, & the systems for AI.

Relative to other infrastructure costs, these fees can be significant & understanding the gross-margin impact of this new layer of software will be essential, & will likely push startups from the more expensive larger models to more capital-efficient AI over time.

​