How to evaluate LLM applications

"Evaluating an LLM" is a misleading phrase for what most teams actually need to do. You rarely control the underlying model. What you build, and what breaks in production, is a system around it: a prompt, maybe a retrieval step, maybe a set of tools the model can call, and the logic that turns its output into something a user sees. LLM evaluation means testing that whole pipeline, on your data, against failure modes you actually care about — not running a model on a public leaderboard.

You need this discipline as soon as an LLM feature reaches a user who did not choose to be a tester. A prototype you demo yourself does not need a dataset and a scoring rubric. A support-ticket classifier, a document Q&A tool, or anything that writes text a customer will read does.

Start with a dataset, not a metric

Every serious evaluation setup begins with a set of representative inputs and, where possible, a reference for what a good output looks like. Pull these from real usage as soon as you have any: actual questions users asked, actual documents they queried, the edge cases support tickets revealed. A dataset of 50 real, hard examples beats 500 synthetic easy ones. Add new cases whenever something breaks in production — an eval suite that never grows is testing yesterday's product.

Only once you have cases worth testing does the choice of metric matter.

Three layers, not one

• Offline evaluation. Before you ship a prompt or model change, run it against your dataset and score the outputs. This is where you catch regressions cheaply, the same way unit tests catch code regressions.
• Online (production) monitoring. Real traffic does not look like your dataset. Sample production outputs, score them continuously, and watch trends — latency, cost, and quality metrics — for drift as usage patterns or the underlying model change.
• Human review. Automated scores are a proxy. Route a sample of outputs, and always your worst-scoring ones, to a person who knows the domain. For anything involving judgment, nuance, or brand voice, human review is the ground truth the automated metrics are trying to approximate.

Skipping any one of these leaves a blind spot: offline-only misses real-world drift, production-only means every regression reaches users first, and no human review means you are trusting a judge you never audited.

Choosing how to score an output

Three approaches, usable together:

• Deterministic checks. Does the output parse as valid JSON? Does it contain a required field? Is it under a length limit? Cheap, fast, and unambiguous — use them for anything checkable in code before reaching for a model.
• LLM-as-a-judge. A second model scores the first model's output against a rubric — an overall-evaluation-criterion such as "answers the question using only the provided context, in under 200 words, with no unsupported claims." This scales to nuanced judgments that deterministic checks cannot express, but the judge itself needs periodic calibration against human ratings, or its errors become invisible.
• Reference-based metrics. When you have a known correct answer, compare against it directly — exact match, or semantic similarity for free text.

For systems built on retrieval, evaluate the retrieval and the generation separately. A wrong answer built on the right documents is a prompting problem; a wrong answer built on the wrong documents is a retrieval problem, and conflating the two sends engineers fixing the wrong half of the pipeline. Metrics such as faithfulness (does the answer only claim what the retrieved context supports), answer relevancy, and context precision and recall separate these failure modes; Ragas was purpose-built around exactly this set of metrics for RAG pipelines.

Guardrails are not the same as evaluation

guardrails run at inference time to block or reshape a bad output before a user sees it — a topic filter, a PII redaction step, a schema validator. Evaluation runs beforehand and afterward to measure whether the system is good, on average and in aggregate. You need both: guardrails catch the individual bad response, evaluation tells you whether last week's prompt change made bad responses more or less common.

Agents add a layer

If your system calls tools, plans multi-step actions, or hands off between sub-agents — agentic analytics territory — evaluate the trajectory, not just the final answer. Did it call the right tool? Did it stop when it should have? A correct final answer reached via three unnecessary tool calls and a near-miss is a system you got lucky with, not one you can trust to scale.

A shortlist by situation

• You want a free, open-source library you can run locally before committing to anything hosted: Arize Phoenix and Ragas both run entirely in a notebook or local server; Ragas is narrowly focused on RAG metrics, Phoenix covers broader tracing and evaluation.
• You want evaluation to be a CI/CD step, written like unit tests: DeepEval is built explicitly for this, with 30-plus research-backed metrics usable in a standard test runner.
• You want tracing first, with evaluation and self-hosting as a natural extension: Langfuse is open-source, self-hostable for free indefinitely, and scales onto a managed cloud plan without a rewrite.
• Evaluation and experiment comparison is the primary workflow, not an add-on to logging: Braintrust is organized around side-by-side experiments comparing prompt or model changes.
• Non-engineers — product managers, domain experts — need to review and edit prompts directly: Humanloop is built for that collaboration, not just engineers reading traces.

Questions to ask in a trial

1. Can we bring our own dataset and our own scoring rubric, or are we limited to built-in metrics?
2. How does the platform handle LLM-as-judge calibration — can we compare judge scores against our own human ratings?
3. What does it cost at our expected trace volume in six months, not this month?
4. Can a non-engineer review and annotate outputs without engineering help?
5. If we self-host, what is the operational burden — and if we don't, where does our data go?

Common mistakes

• Testing only the happy path. Your dataset should be disproportionately hard cases and known failures, not a random sample of easy questions.
• Trusting an uncalibrated judge. An LLM-as-judge that has never been checked against human raters can be confidently, consistently wrong in one direction.
• Treating a single score as the whole picture. Faithfulness, relevance, latency, and cost trade off against each other; optimizing one in isolation degrades the others.
• Never revisiting the dataset. Production traffic changes. An eval suite frozen at launch stops catching the failures that matter six months later.

For the wider category — tracing, guardrails, and production monitoring platforms beyond the evaluation-specific tools above — see every tool in this category and how to choose an LLM observability tool. If your application answers questions by writing SQL rather than free text, how to use AI for data analysis safely covers evaluation specific to that case.