Introduction

Peter F. Drucker, in his famous book “The Effective Executive”, coined the phrase “What gets measured, gets improved”. This saying is as popular in the Machine Learning (ML) world as it is in the corporate world. For instance, to know that you need to work on the accuracy of the model, you first need to be rightly informed that the model's accuracy is not sufficient. And, to know that the accuracy is lacking, you need to measure it.

To improve the models, we need to gauge them across numerous facets. The monitoring metrics depend on the task at hand and overlap with those of conventional ML models. Depending on the task, you could still use metrics like the F1 score, accuracy, and precision to gauge the performance of LLMs but, in addition to these metrics, you will also need to take care of:

Failing to monitor LLMs could result in a tarnished reputation and might cause irrevocable damage to both the company using it and the company that made it. So, what should you know about monitoring large language (and traditional) models?

In today’s Token we cover:

Turns out things can get nasty really quick with LLMs, how can I start monitoring my models and infrastructures?

In the ML world, 100s of new tools emerge every week. Not all of them are going to be useful for your use case. Below, we have curated a list of open-source tools that solve some of the most pressing problems with LLM monitoring and observability:

Thank you for reading, please feel free to share with your friends and colleagues. In the next couple of weeks, we are announcing our referral program 🤍

Previously in the FM/LLM series: