Glossary

LLM-as-a-judge

Using a language model to score or compare the outputs of another model, as a scalable substitute for human review.

Also called: model-graded evaluation, AI judge

LLM-as-a-judge is an evaluation technique where a language model, usually a strong general-purpose one, is given an output, or a pair of outputs, along with a rubric or criteria, and asked to score, rank, or critique it, standing in for a human reviewer. It is used to grade qualities that are hard to check automatically, such as helpfulness, tone, or factual grounding.

This differs from traditional automated metrics, which compare output to a fixed reference answer using rules or string overlap; a judge model can assess free-form output against qualitative criteria the way a human rater would, at far lower cost and much higher speed. It is a core technique inside LLM evaluation pipelines and is increasingly used in LLM observability to flag likely problem responses, such as hallucination, from live traffic.

The main pitfall is that a judge model has its own biases and blind spots: it can favor longer or more confidently worded answers regardless of accuracy, and it can share the same failure modes as the model it is judging, particularly if it is the same model or model family. Judge results are therefore best treated as a scalable signal to prioritize human review, not a fully independent ground truth, and are periodically checked against actual human ratings, part of a broader guardrails strategy.

Last reviewed September 22, 2026

In the index now

Related terms

LLM evaluation Systematically testing a language model's outputs for quality, accuracy, safety, and consistency before and after deployment. Hallucination When a language model generates confident, fluent output that is factually wrong or unsupported by its source material. Guardrails Rules, filters, or checks placed around a language model to keep its inputs and outputs within acceptable, safe bounds. Large language model (LLM) A neural network trained on vast amounts of text to predict and generate language, the technology behind most modern AI assistants. LLM observability Monitoring a language model's live behavior in production, tracking its inputs, outputs, latency, cost, and failure patterns.

Related guides

How to choose an LLM observability tool Pick an LLM tool by whether you need tracing, scoring or a gateway first — most teams eventually need all three, rarely from one vendor. How to evaluate LLM applications Offline evals, LLM-as-judge, human review and production monitoring, aimed at the whole pipeline rather than the model alone.