How to choose an LLM observability tool

An LLM application fails in ways a normal application doesn't: it can produce a fluent, confident, wrong answer with no exception thrown. LLM observability and evaluation tools exist to catch that — by recording what a model was asked, what it produced, and scoring whether the output was actually good, both before shipping a change and while it runs in production. Almost nobody skips this category once an LLM feature reaches real users; the question is which of three jobs — tracing, evaluation, or gateway — to buy first, and from whom.

Decide which job comes first

Tracing and logging. The baseline need: capture every call, its latency, cost and full prompt/response, so a failure can be inspected after the fact. Helicone and OpenLLMetry are built around this — Helicone via a proxy or SDK with its own query language over logged requests, OpenLLMetry as a vendor-neutral OpenTelemetry instrumentation layer for LLM observability that can ship traces to more than twenty different backends rather than locking you into one.

Evaluation as a workflow, not an afterthought. Braintrust, Patronus AI and DeepEval treat evaluation as the primary product: datasets, scoring functions and experiments that tell you whether a prompt or model change improved or regressed quality, before it reaches users. DeepEval is a free, open-source Python library styled like a unit-testing framework, runnable in CI/CD with no usage limits; Patronus AI instead sells pre-built evaluator models for specific failure modes like hallucination and PII leakage, priced per evaluation call rather than per trace.

Gateway plus logging. Portkey is primarily a routing layer in front of multiple LLM providers — retries, fallbacks, load balancing, guardrails — with observability as a side effect of every call passing through it. If you need production reliability features anyway, getting logging bundled in avoids running a second tool purely to watch the first one.

Most mature setups end up combining at least two of these: a tracing layer for everyday debugging and an evaluation workflow for pre-release testing, sometimes from different vendors.

Decide how much you want tied to one ecosystem

LangSmith is built by the LangChain team and is the natural choice if your application is already built on LangChain or LangGraph — it captures traces from that framework with no extra glue. Langfuse and Arize Phoenix take the opposite approach: both are open source, framework-agnostic, and can be self-hosted for free indefinitely (Langfuse via Docker Compose or Kubernetes, Phoenix as a local-first library that runs before any data leaves your machine), only asking you to pay once you want the hosted, multi-user version. W&B Weave is a third variant of lock-in: it shares billing, seats and infrastructure with Weights & Biases' existing ML experiment tracking, which is a real advantage if you're already a W&B customer and a real cost if you're not.

Check whether it evaluates general outputs or RAG specifically

Ragas is not a general evaluation tool — it's purpose-built for RAG pipelines, where quality depends on both what was retrieved and what was generated from it, with reference-free metrics like faithfulness and context precision that don't require hand-labeled ground truth. It runs in-process as a library with no hosted API of its own, which is why teams typically pair it with a tracing tool like Langfuse to store and chart the resulting scores over time. If your application isn't RAG-based, a general evaluation tool such as DeepEval or Braintrust will fit better than a RAG-specific metrics library.

Decide whether you need production guardrails or after-the-fact review

Galileo is built around real-time guardrails that can intervene on live traffic — blocking or flagging a hallucination before it reaches the user — rather than only reporting on it in a dashboard afterward. That capability requires low-latency infrastructure and is priced and gated accordingly (it's an Enterprise-tier feature). Most of the rest of this category, including the open-source tracing tools, observes and reports rather than intervening; decide early whether after-the-fact visibility is enough or whether you need something in the request path.

Weigh collaboration against engineering control

Humanloop and PromptLayer both put weight on letting non-engineers — product managers, domain experts — version prompts and review outputs without touching application code, distinct from tools built primarily for engineers reading traces. Lunary adds a specific angle: GDPR-compliant, EU-hosted or fully self-hosted observability with PII masking built in, aimed squarely at teams with European data-residency requirements rather than being a generic feature checklist.

Expect pricing to scale with trace or event volume

Nearly every vendor here prices on some unit of usage — traces, spans, events, or evaluation calls — layered under a monthly platform fee once you're past a free tier. The free tiers are genuinely usable for evaluation (Langfuse and Phoenix are free to self-host with no limits at all), but budget for the jump: a Pro or Team tier commonly adds a flat monthly fee on top of the same per-unit charges, and enterprise features like SSO, audit logs and on-prem deployment are almost always reserved for custom-quoted plans. Compare the unit each vendor meters — a "trace," a "span" and an "observability unit" are not defined identically across vendors — before assuming two plans are priced the same way.

A shortlist by situation

• If you want free, low-friction proxy-based logging without instrumenting your code, look at Helicone.
• If evaluation and experimentation should be the primary workflow, with production monitoring layered on top, look at Braintrust.
• If you want an open-source library to unit-test LLM outputs in CI/CD, look at DeepEval.
• If you're already building on LangChain or LangGraph, look at LangSmith; if you want the same job open source and self-hostable, look at Langfuse.
• If you need real-time guardrails that can act on production traffic, not just report on it, look at Galileo.
• If you need vendor-neutral, OpenTelemetry-standard instrumentation you can route anywhere, look at OpenLLMetry.
• If you have GDPR or EU data-residency requirements, look at Lunary.
• If you need a production gateway (routing, fallbacks) with logging built in, look at Portkey.
• If non-engineers need to collaborate directly on prompts and review, look at Humanloop or PromptLayer.
• If you're evaluating a RAG pipeline specifically, look at Ragas; for pre-built evaluator models for safety and accuracy, look at Patronus AI.
• If you already run Weights & Biases for ML experiment tracking, look at W&B Weave.
• For a free, local-first library to trace and evaluate during development, look at Arize Phoenix.

Questions to ask vendors

1. What unit does pricing meter — traces, spans, events, or evaluation calls — and what's included before overage starts?
2. Can we self-host the core product indefinitely for free, or is the open-source tier feature-limited?
3. Does evaluation require hand-labeled ground truth, or does it support reference-free and LLM-as-judge scoring?
4. Can this intervene on production traffic in real time, or does it only report after the fact?
5. What data residency and compliance certifications (SOC 2, GDPR, HIPAA) does the hosted product actually hold today, not on a roadmap?

Common mistakes

Buying a tracing tool and expecting it to double as an evaluation workflow — logging what happened is not the same as scoring whether it was good. Choosing a RAG-specific metrics library for an application that isn't RAG-based. And underestimating trace volume: a free tier that looks generous in a demo can be consumed by a single day of production traffic once the feature ships.

For direct comparisons, see Langfuse vs LangSmith, DeepEval vs Ragas and Helicone vs Portkey. The full list of vendors in this category is at every tool in this category.