Glossary

LLM observability

Monitoring a language model's live behavior in production, tracking its inputs, outputs, latency, cost, and failure patterns.

Also called: AI observability

LLM observability is the practice of continuously monitoring a language model's behavior after it has been deployed: logging prompts and responses, tracking latency and token cost, and watching for shifts in output quality, safety, or user satisfaction as real traffic flows through the system.

It extends the older discipline of model monitoring to the specifics of generative systems, where "correct" output is harder to define than a single predicted number. In practice this means capturing structured signals, such as flagged hallucination rates or guardrails triggers, alongside the raw conversation logs, and often replaying failures to reproduce and diagnose them.

Observability differs from LLM evaluation in timing and purpose: evaluation happens before or between releases against curated test sets, while observability runs continuously against live traffic to catch problems evaluation did not anticipate, including data drift as user behavior or input patterns change. The main pitfalls are logging so much raw data that nothing gets reviewed, and treating dashboards of averages as sufficient when a small number of severe failures can matter more than the mean.

Last reviewed September 22, 2026

In the index now

Related terms

LLM evaluation Systematically testing a language model's outputs for quality, accuracy, safety, and consistency before and after deployment. Guardrails Rules, filters, or checks placed around a language model to keep its inputs and outputs within acceptable, safe bounds. Hallucination When a language model generates confident, fluent output that is factually wrong or unsupported by its source material. Large language model (LLM) A neural network trained on vast amounts of text to predict and generate language, the technology behind most modern AI assistants. Model monitoring Ongoing tracking of a deployed model's inputs, predictions and outcomes to catch accuracy loss, drift or failures after launch. Data drift A change in the statistical properties of input data over time that can silently degrade a deployed model's accuracy.

Related tools

Arize AI Hosted AI observability platform that monitors deployed ML models and LLM applications for drift, quality, and performance issues. Braintrust Evaluation-first platform for testing, scoring, and comparing LLM application changes, with production tracing built in. Comet Hosted experiment-tracking platform that also offers a separate LLM observability product (Opik). DeepEval Open-source Python evaluation framework with 30+ research-backed metrics, with an optional paid platform (Confident AI) for CI/CD and hosted reporting. Fiddler AI Hosted AI observability and guardrails platform for monitoring deployed ML models and LLM/agent applications. Galileo LLM observability and evaluation platform with real-time guardrails, aimed at monitoring agents and generative AI in production.

Related guides

How to choose an LLM observability tool Pick an LLM tool by whether you need tracing, scoring or a gateway first — most teams eventually need all three, rarely from one vendor. How to evaluate LLM applications Offline evals, LLM-as-judge, human review and production monitoring, aimed at the whole pipeline rather than the model alone.