💡 ML Concept of the Day: What is Constitutional AI?

In the previous issue of this series we explored the concept of reinforcement learning with human feedback(RLHF) as one of the fundamental techniques to fine-tune the instruction alignment problem with LLMs. RLHF has proven to be incredibly effective but also quite hard to implement as it requires a decent number of people involved and the process is largely inconsistent from one model to another. An interesting alternative to RLHF was proposed by Anthropic as the main technique powering its Claude model.

In the Constitutional AI approach, the AI training process encompasses two key phases:...