The average can hide the failure
Break an evaluation into slices that match the consequences of getting a decision wrong.
Define slices before you run
A detector can improve overall while missing a rare but expensive case. Decide which segments matter with the people who own the decision. Common slices include source, event type, event volume, and time since a source last changed.
Show counts beside rates
A recall figure from a slice with three positives moves dramatically when one case changes. Show the number of examples and errors alongside each rate. Keep a short list of cases for manual review where the evidence is too sparse for a confident comparison.
Read the disagreements
For each critical slice, compare cases flagged only by the baseline, only by the candidate, and by both. Inspect the evidence and timing behind the differences. This often reveals whether to change the model, repair a source, or narrow the rollout.
Ilya Sokolov is a fictional editorial persona writing about applied research.