machine learning in production

A complete guide to machine learning in production

How to maintain ML models once after you deploy them, and what exactly to prepare for? In this guide, we look into the key concepts that relate to production ML model operations.

Explore topics →

Model Monitoring

What metrics to use and how to design a model monitoring strategy.

Concept Drift

What concept drift is, why it matters, and how to detect and react to it.

Data Drift

What data drift is, why it matters, and how it differs from similar concepts.

OPEN-SOURCE ML OBSERVABILITY COURSE

Open-source ML observability course

Want to learn production ML monitoring? Sign up for our Open-source ML observability course for data scientists and ML engineers. It's free!

Save my seat →

View course notes →

Ranking and recommendations METRICS in machine learning

A complete guide to ranking quality metrics

How do you evaluate the quality of ranking and recommender systems? In this guide, we look into the key ranking metrics and explain them step by step.

Explore metrics →

Precision and Recall at K

How to use Precision and Recall at K to assess the performance of ranking and recommender systems.

Mean Average Precision (MAP)

How to compute MAP to evaluate the quality of ranking and recommender systems.

Normalized Discounted Cumulative Gain (NDCG)

How to use NDCG at K to measure the ranking quality for both binary and graded relevance scores.

ML MONITORING METRICS

Monitoring ML systems in production. Which metrics should you track?

"ML monitoring" can mean many things. Are you tracking service latency? Model accuracy? Data quality? This guide organizes everything one can look at in a single framework.

See the guide →

CLASSIFICATION METRICS in machine learning

A complete guide to classification metrics in machine learning

How to evaluate the quality of a classification model? In this guide, we break down different machine learning metrics for binary and multi-class problems.

Explore metrics →

Confusion Matrix

How to create a confusion matrix for binary and multi-class models.

Accuracy, Precision, Recall

The difference between metrics and how to choose the appropriate one.

Multi-class Precision and Recall

How to calculate multi-class metrics by class, micro- and macro-averaging.

ML MONITORING METRICS

Monitoring ML systems in production. Which metrics should you track?

"ML monitoring" can mean many things. Are you tracking service latency? Model accuracy? Data quality? This guide organizes everything one can look at in a single framework.

See the guide ⟶

Data drift detection

How to detect drift on large datasets? 5 methods compared

We ran an experiment to help build an intuition on how popular drift detection methods behave. In this guide, we share the key takeaways and the code to run the tests on your data.

See the guide ⟶

Data drift detection

Which test is the best? We compared 5 methods to detect data drift on large datasets

We ran an experiment to help build an intuition on how popular drift detection methods behave. In this guide, we share the key takeaways and the code to run the tests on your data.

See the guide →

Shift happens: we compared 5 methods to detect drift in ML embeddings

Monitoring embedding drift is relevant for the production use of LLM and NLP models. We ran experiments to compare 5 drift detection methods. Here is what we found.

See the guide →

The big book of ml monitoring

The Big Book of ML Monitoring

A comprehensive introduction to ML monitoring, including a framework to organize ML monitoring metrics and steps to take when dealing with data drift in production.

Get your PDF copy →

MLOps guides

A complete guide to machine learning in production

Model Monitoring

Concept Drift

Data Drift

Open-source ML observability course

A complete guide to ranking quality metrics

Precision and Recall at K

Mean Average Precision (MAP)

Normalized Discounted Cumulative Gain (NDCG)

Monitoring ML systems in production. Which metrics should you track?

A complete guide to classification metrics in machine learning

Confusion Matrix

Accuracy, Precision, Recall

Multi-class Precision and Recall

Monitoring ML systems in production. Which metrics should you track?

How to detect drift on large datasets? 5 methods compared

Which test is the best? We compared 5 methods to detect data drift on large datasets

Shift happens: we compared 5 methods to detect drift in ML embeddings

The Big Book of ML Monitoring