Model monitoring

Model monitoring is the ongoing tracking of a deployed model's inputs, predictions and, where available, actual outcomes, to catch problems that only appear after launch. An offline evaluation only measures performance on historical data at one point in time; monitoring is what shows whether that performance is holding up as real-world data keeps changing.

Monitoring typically covers several layers: operational health, such as latency and error rates; input health, comparing incoming feature distributions against the training baseline to catch data drift; and outcome health, comparing predictions against actual results once they become available, to catch concept drift or a general accuracy decline. Some outcomes, such as whether a loan eventually defaulted, are only known weeks or months later, which delays that last layer of monitoring.

Model monitoring is a core part of MLOps and closely tied to model risk management in regulated industries such as banking and insurance, where a model's ongoing performance and fairness must be demonstrably tracked, not just validated once before launch. Without it, degradation is often only discovered when someone notices a business metric has quietly worsened, by which point the model may have been making poor decisions for a long time.