AI observability — logs, traces, cost and hallucinations

A traditional web service fails loudly. The database times out, the request 500s, the error rate spikes, the pager goes off. Your APM dashboard turns red and you know something is wrong before your users tell you.

An LLM system fails quietly. The model returns a fluent, well-formatted, confident answer in 800 milliseconds with a 200 OK. The request looks perfect in every dashboard you already have. The only problem is that the answer is wrong — it cited a policy that doesn't exist, it invented a refund the customer isn't owed, or it quietly started costing you three times more per call after a prompt change nobody flagged as risky.

This is the core reason AI observability is different. A successful HTTP response tells you almost nothing about whether the AI did its job. Latency, status codes and throughput — the entire vocabulary of classic observability — are necessary but nowhere near sufficient. You need a stack that treats the content of the interaction, its cost, and its correctness as first-class signals.

There are four pillars. Miss any one and you're flying blind on a different failure mode.

Pillar 1 — Logs: capture the full interaction

The first rule of AI observability: log the complete prompt and the complete response, every time, in a way you can query later.

This sounds obvious and it's the thing teams most often get wrong. They log the user's message and the final answer, but not the system prompt, not the retrieved context, not the tool outputs that were stitched into the prompt, not the model parameters. Then something goes wrong in production and they're staring at an output with no way to reconstruct the input that produced it. You cannot debug an LLM call you can't reproduce.

What a useful log entry contains:

• The full rendered prompt — system prompt, few-shot examples, retrieved context, and the user turn, exactly as the model saw it.
• The full response, including tool calls and intermediate reasoning if you use it.
• Model and parameters — model ID, temperature, max tokens, and any tool definitions in scope.
• Version tags — which prompt template version, which retrieval index version, which app release. This is what lets you say "quality dropped on the 14th" and immediately see that prompt v7 shipped on the 14th.
• Token counts and latency for the call.

Two constraints make this non-trivial. PII: prompts and responses routinely contain names, emails, order details, sometimes worse. You need field-level redaction or tokenisation before anything hits your log store, and a retention policy that a data-protection reviewer would sign off on. Volume: full prompt/response capture at scale is expensive to store. Sample aggressively for the boring 99% and capture 100% of anything that errored, got a thumbs-down, or tripped a quality flag (more on that below).

Pillar 2 — Traces: see across the whole chain

A modern AI feature is almost never a single model call. A RAG answer is: embed the query, search the vector store, rerank, assemble context, call the model, maybe call it again to check its own work. An agent is worse — a loop of model calls, tool invocations, and more model calls, branching on what each step returns.

When that chain is slow or wrong, "the LLM was slow" is not a diagnosis. You need distributed tracing adapted to AI: one trace per user request, with a span for every step, so you can see exactly where time and errors come from.

Instrument a span around each of:

• Retrieval — the vector search and rerank, with the query, the number of candidates, and which documents came back. Half of "the model hallucinated" turns out to be "retrieval returned nothing relevant."
• Tool / function calls — each external API the model invoked, its arguments and its result. Tool calls are where agents spend most of their wall-clock time and most of their failures.
• Model calls — each individual completion, with its tokens and latency, so a slow trace tells you which of the three model calls was the slow one.
• Guardrail / eval steps — moderation, groundedness checks, schema validation.

The payoff: when a request takes nine seconds, the trace shows you it was a reranker misconfiguration, not the model. When an agent gives a wrong answer, the span tree shows you the tool returned bad data three steps back. Adopt OpenTelemetry-style semantics if you can — the emerging GenAI conventions mean your AI spans sit in the same trace as your database and HTTP spans, and you debug the whole request in one view.

Pillar 3 — Cost: account for every token

LLM cost behaves unlike any other line item in your infrastructure. It scales with usage and with prompt size, both of which can move without a deploy. A change to a retrieval setting that pulls in twice as much context doesn't change your latency much — but it can double your token bill overnight, silently.

So you instrument cost the way a finance team instruments spend: per request, attributed along dimensions that let you answer "who and what is driving this."

Capture input tokens, output tokens and computed dollar cost on every call, tagged with:

• Feature — which product surface made the call. The chat assistant, the summariser, the background enrichment job.
• User / tenant / plan — so you can see that 4% of users generate 60% of cost, and whether your pricing covers it.
• Model — the mix of premium vs. cheap models, and whether an expensive model is being used where a cheap one would do.

Then close two loops on top of it. Budget alerts: a threshold per feature and per tenant that pages someone before the monthly invoice, not after. Attribution reviews: a weekly look at cost-per-feature that catches the prompt that grew a system message from 400 to 4,000 tokens. The most common cost blowups aren't traffic spikes — they're a context window that quietly got fatter, or a retry loop that runs three times more than anyone intended.

Pillar 4 — Quality and hallucination: eval in prod

This is the pillar that doesn't exist in classic observability at all, and it's the one that matters most. Everything above tells you the system is running. None of it tells you the system is right.

You cannot fully solve this offline. An eval suite in CI catches regressions on the cases you thought of; production is where the cases you didn't think of live. So quality has to be measured continuously, on live traffic.

The signals worth wiring up:

• Groundedness / faithfulness checks. For RAG, run an automated check — often a smaller, cheaper model acting as judge — that asks "is every claim in this answer supported by the retrieved context?" Answers that aren't grounded get flagged. This catches the confident invention that a status code never will.
• Eval-in-prod sampling. Run a rubric-based scorer over a sample of live responses — correctness, tone, format adherence, refusal appropriateness. You won't score every request, but a steady sample gives you a quality time-series you can alarm on.
• User feedback signals. Thumbs up/down, copy-to-clipboard, "regenerate", the follow-up where the user says "no, I meant…". These implicit and explicit signals are gold. Capture them and join them back to the exact trace and prompt version that produced the response.
• Drift detection. Track the distribution of your quality scores, response lengths, refusal rates and groundedness over time. A slow slide in the average is the model provider silently updating a model, your data distribution shifting, or a prompt change biting in a way your eval set missed.

The tooling landscape

You don't have to build all of this from scratch. A category of LLM-observability tools now covers most of it — Langfuse, Helicone, Arize Phoenix and others in that space give you trace capture, prompt/response logging, token-cost accounting and eval hooks out of the box, usually via a thin SDK wrapper or a proxy in front of your model provider. They differ in emphasis: some lean toward tracing and eval, some toward cost and caching at the proxy layer, some toward open-source self-hosting for the PII-sensitive.

Pick based on where your risk actually is. Don't treat any of them as a silver bullet — they give you the pipes. The four pillars, the sampling policy, and the loop back into your prompts are decisions you still own.

The loop that makes it worth it

Observability that only produces dashboards is a cost centre. The point is the feedback loop: a production signal changes what you ship.

Concretely: a thumbs-down in prod is captured with its full trace and prompt version → it becomes a row in your eval set → the eval fails on the next prompt candidate → the fix ships → the groundedness score in prod recovers. Prod feeds eval, eval gates the change, the change moves prod. When that loop is running, your eval suite gets sharper every week because it's fed by real failures instead of imagined ones.

Mistakes teams make

Logging only inputs and outputs, not the middle. Without the retrieved context and tool results, you can see that the answer was wrong but not why. Capture the whole chain or you'll be guessing.

Treating a 200 as success. The single most expensive assumption in AI systems. Your existing dashboards are green while quality quietly rots. Green means "it responded," not "it was right."

No prompt/version tags on logs. If you can't join a quality drop to the exact prompt, model and index version in play, you can't debug it. Version everything and tag every log with it.

Cost as a monthly surprise. By the time the invoice arrives, the damage is a month old. Per-feature, per-tenant budget alerts turn a five-figure surprise into a same-day nudge.

Evals that only run in CI. A CI eval suite tests the cases you imagined. Production is full of the ones you didn't. Sample and score live traffic, or you're only measuring your own imagination.

Sampling everything at 100% or 0%. Full capture bankrupts your log budget; no capture blinds you. Sample the ordinary path, capture 100% of errors, negative feedback and flagged responses.

TL;DR

A 200 OK from an LLM can still be a hallucinated, expensive, wrong answer, so classic APM isn't enough. Build four pillars: logs that capture the full versioned, PII-aware prompt and response; traces with a span per retrieval, tool and model call so you can see where latency and errors come from; cost accounting per request, user and feature with budget alerts; and quality measured in production via groundedness checks, eval sampling, user feedback and drift detection.

Sample the boring traffic, capture everything that failed or got a thumbs-down, and close the loop so prod signals become eval cases that gate your next prompt change. Reach for the Langfuse/Helicone/Phoenix-style tools for the plumbing, but own the pillars and the loop yourself.

If you're flying blind on your AI system, let's talk.