If you were forwarded this newsletter, and you'd like to receive it in the future, subscribe here.​

​The Art of Product Management in the Fog of AI​

Product managers & designers working with AI face a unique challenge: designing a delightful product experience that cannot fully be predicted.

Traditionally, product development followed a linear path.

A PM defines the problem, a designer draws the solution, and the software teams code the product. The outcome was largely predictable, and the user experience was consistent.

However, with AI, the rules have changed. Non-deterministic ML models introduce uncertainty & chaotic behavior.

The same question asked four times produces different outputs. Asking the same question in different ways - even just an extra space in the question - elicits different results.

How does one design a product experience in the fog of AI?

The answer lies in embracing the unpredictable nature of AI and adapting your design approach. Here are a few strategies to consider:

1. Fast feedback loops : Great machine learning products elicit user feedback passively. Just click on the first result of a Google search and come back to the second one. That’s a great signal for Google to know that the first result is not optimal - without tying a word.
2. Evaluation : before products launch, it’s critical to run the machine learning systems through a battery of tests to understand in the most likely use cases, how the LLM will respond.
3. Over-measurement : It’s unclear what will matter in product experiences today, so measuring as much as possible in the user experience, whether it’s session times, conversation topic analysis, sentiment scores, or other numbers.
4. Couple with deterministic systems : Some startups are using large language models to suggest ideas that are evaluated with deterministic or classic machine learning systems. This design pattern can quash some of the chaotic and non-deterministic nature of LLMs.
5. Smaller models : smaller models that are tuned or optimized for use cases will produce narrower output, controlling the experience.

The goal is not to eliminate unpredictability altogether but to design a product that can adapt and learn alongside its users.

Just as much as the technology has changed products, our design processes must evolve as well.

​