alexandre.

← Notes

Agent observability: one trace per task, one cost per step

An untraced agent is a black box that sends you a bill. The trace model I use in production on a legal agent: Langfuse for LLM calls, cost calculated alongside, and tracing that can’t bring production down.

  • agents
  • observability
  • Langfuse
  • costs

Take a typical message, the kind that eventually reaches every agent in production: ‘the agent took three minutes on this morning’s contract, and it missed the liability clause.’ You open the code. Between the uploaded file and the returned document, there is a chain of steps: extraction, triage, anchoring check, fix, wording review, comment, summary. Which one was slow? Which one failed silently? Which one weighs most on this month’s bill?

Without traces, no answers: you rerun the agent by hand, add print statements and hope to reproduce it, while it keeps running in production.

Four symptoms of an unobserved agent

  • The outage is reported, not measured. The only signal is an unhappy user, several hours later.
  • The failure is global. ‘The audit failed’ — but extraction, triage, fix and summary are distinct steps, with distinct causes.
  • The bill is a monthly total. You know what the project costs, never which step costs it, nor whether it was the prompt changed on Tuesday that doubled the bill.
  • Quality can’t be compared. Two prompt versions coexist, and nobody can say which one ran on which files.

What they have in common: the unit of observation is the service, whereas the unit the business cares about is the task: one contract audited, one request handled.

The trace model: task, step, call

Three levels are enough, and they map onto the vocabulary of Langfuse, the LLM call tracing tool I use:

  • Trace = one complete business task. On the contract review agent from previous notes, that’s one contract audit: audit:<file name>, with the model used and the document’s origin as metadata.
  • Span (a segment of a trace) = one step, including the ones that call no model, with its duration, its status and its prompt version.
  • Generation (the type Langfuse reserves for model calls) = one LLM call, with its prompt, its output, its model and its tokens.

The tree of an audit, simplified:

trace  audit:<file>                          model=… · source=…
├─ span        audit.stage extraction        extract-v2
├─ generation  audit_extract_chunk_k_of_n    extraction, one per contract chunk
├─ span        audit.stage triage            triage-v2.1
├─ generation  audit_triage                  clause-by-clause verdict
├─ span        audit.stage anchor            deterministic: checked, rejected
├─ generation  audit_reanchor                bounded re-anchoring micro-pass
├─ span        audit.stage fix               fix-v6
├─ generation  audit_fix_batch ×N            fixes, in parallel
├─ generation  audit_placement_decider       placement of missing clauses, if needed
├─ span        audit.stage wording           wording-v1
├─ generation  audit_wording_batch ×N        language review
├─ span        audit.stage client_comment    comment-v1
├─ generation  audit_comment_retry           client comment
├─ span        audit.stage aggregate         aggregate-v1
└─ generation  audit_aggregate               summary and email
output = { ok, error, stages[{stage, ok, n}], truncated }

Every generation carries, as metadata, the version of the prompt that produced it. Comparing two versions therefore happens in the trace, without going back through the evaluation harness.

The interesting node in this tree is the one that was missing for a long time. The anchoring gate is the guardrail that rejects a fix whose quotation can’t be found word for word in the contract. Consuming no tokens, it produces no generation, and in the first version of this instrumentation, no span either: it only surfaced in the root’s stages[] summary, and its duration existed only in a log line. It now has its own span, with the number of clauses checked and rejected. That’s what stops you wrongly blaming the model when an audit comes back almost empty, and what feeds the guardrail rejection rate further down. A deterministic step deserves its span just as much as an LLM call.

Instrumentation in practice

All the LLM instrumentation lives in a single place: the function that calls the model. A simplified version, faithful to the one in production:

def traced_generation(*, name, model, prompt_input, call, trace=None, metadata=None):
    """One Langfuse generation per model call. Must NEVER make the call fail."""
    client = _get_client() if langfuse_enabled() else None  # missing keys or SDK, broken SDK
    if client is None:
        return call()
    try:
        # Explicit parenting: implicit context doesn't cross thread pools.
        owner = trace.root_span if (trace and trace.root_span) else client
        span = owner.start_observation(as_type="generation", name=name, model=model,
                                       input=prompt_input, metadata=metadata or {})
    except Exception:
        return call()  # tracing broken: the model call goes out anyway
    try:
        response = call()
    except Exception as exc:
        try:
            span.update(level="ERROR", status_message=f"{type(exc).__name__}: {exc}")
            span.end()
        except Exception:
            pass
        raise
    try:
        span.update(output=getattr(response, "text", None),
                    usage_details=_usage(response))  # input / output / total tokens
        span.end()
    except Exception:
        pass  # a response already obtained is never lost to a tracing error
    return response

This code rests on three choices.

Tracing steps aside at the slightest problem. No keys, no SDK, or the SDK raises — whether opening the span or closing it — the model call goes out and the response comes back. The worst observability failure is the one that takes down the service it observes.

Parenting is explicit. The pipeline processes fix batches and extraction chunks across several threads. The ambient trace context doesn’t follow them on its own: without passing the trace as a parameter, and without copying the context into each thread, an audit ends up scattered into orphan traces, and cost per task becomes impossible to calculate.

On Cloud Run, you have to force traces out. The SDK exports in the background; by default, Cloud Run only allocates you CPU during the request. The export thread may never run, and the traces vanish. So you force the send (flush) at the end of the pipeline, including when a blocking step fails: the trace of a failed audit is precisely the one you want to read. And you deploy with --no-cpu-throttling when processing continues after the response, knowing that this setting means you’re billed for CPU outside requests.

Cost is calculated alongside, not in the tracing tool

Langfuse receives the tokens. The price stays with me: a model → rate table, resolved in this order: environment variable, hot-reloadable configuration document, default table. It’s what lets me say an audit costs around €0.60 on this agent, and above all know where that money goes, step by step.

Why keep them separate?

Rates change faster than code. Switching models on a Monday morning shouldn’t require a redeploy for the displayed cost to be right again.

Billing has subtleties the trace ignores. Reasoning tokens on recent models are billed at the output rate: I add them to output tokens in the cost calculation, while exposing them separately for analysis. On the trace side, they don’t surface as such: they’re buried in the total.

Cost is sensitive data. The per-step breakdown in euros sits behind its own allowlist, which can diverge from the administrators’ list without touching the code. On screen, missing data shows as a dash, never as a reassuring zero.

A trap worth knowing: a model with no known rate counts as zero in the total. If nothing flags it, a new model goes into production and the total cost keeps showing a credible figure — but a wrong one. Here, the audit’s cost is then marked incomplete, the dashboard total is displayed as a minimum (‘≥’) along with the models responsible, and a warning goes to the logs.

Langfuse doesn’t see everything: the infrastructure layer

Langfuse sees the model calls. It doesn’t see the HTTP request that triggered the audit, the time spent waiting for an instance, or the API error raised before the first prompt. The full architecture therefore has three layers, tied together by a single identifier:

  • Cloud Trace for the request’s path through the service, from HTTP entry to response;
  • Cloud Logging for structured logs, where every line carries the trace identifier;
  • Langfuse for the detail of LLM calls, attached to the same task.

The join hinges on a detail of OpenTelemetry, the tracing standard Langfuse is built on. At start-up, Langfuse installs its own trace provider if it doesn’t find one. So I install mine before it: Langfuse reuses it, its generations become spans of the HTTP request, and the Langfuse trace takes the same identifier as the Cloud Trace trace. The JSON logs read that same identifier from the current span. Two rules complete the set-up. Spans go to Cloud Trace stripped of their content: prompts and responses stay in Langfuse. And HTTP spans don’t go to Langfuse, otherwise every status refresh would become a trace there.

With this shared identifier, you go from an alert to the request, from the request to its logs, from the logs to the faulty model call, without searching. On the first audit traced end to end, the request, its twenty-four log lines and its nine LLM calls carried the same identifier in all three tools, including the batches processed in parallel. The agent’s chat follows the same path: every turn has its trace, and its tokens go into the cost calculation.

Four screens, three alerts

  1. Cost per task and per step, with the day-by-day curve and the month-to-date total. When an agent doubles in price, it’s usually a single step going off the rails.
  2. Latency per step, at the 95th percentile (p95), not the average: the average hides exactly the cases users talk about. On the first traced audit, a single wording review batch took 95 seconds of a five-minute audit: that’s the kind of case the screen has to bring out.
  3. Failure rate per step, compared from one prompt version to the next.
  4. Guardrail rejection rate. A gate that rejects everything signals an anchoring or format problem; a gate that no longer rejects anything signals that it’s broken.

The first lives in the application, the other three in Cloud Monitoring, computed from the structured logs. The definitions (metrics, dashboard, alerts) are versioned with the code and applied by a script, not clicked together in a console.

The three alerts: audit failure rate above a threshold over one hour, p95 latency of a step above a ceiling, daily cost above a budget. Each one has been triggered once for real, by injecting synthetic log lines into the service: metric, incident, then email received. An alert never triggered in testing isn’t an alert: it’s a hypothesis.

The gaps I know about

On the data side, the split is explicit: prompts and outputs go to the tracing tool, and to it alone; application logs carry only step metadata, never contract content.

That leaves what I don’t claim to have solved. The alert thresholds are first values, not yet calibrated on weeks of history: the test proves the alerting chain works, not that the threshold is right. Chat cost goes into the budget alert, but not yet into the application’s dashboard, which only counts audits. Langfuse can show a trace around ten minutes late, whereas Cloud Trace and the logs are near-instant: during an incident, they’re what you open first. And the rate of fixes actually applied by the lawyers — the measure that says whether the agent is genuinely useful — remains a backlog item, not a screen.

Before putting an agent into production

  • One trace = one business task, named after the business object, not the HTTP route.
  • One span per step, including deterministic steps: gates, validations, programmatic checks.
  • Tracing that steps aside without ever raising an error, opening and closing included, proven by a test, with explicit parenting as soon as there’s parallelism.
  • Forced trace sending (flush) at the end of processing on a platform that throttles CPU outside requests.
  • Tokens captured at source, cost calculated outside the tracing tool, rates editable without a redeploy, unpriced models listed separately rather than counted as zero.
  • A trace identifier shared by the request, the logs and the LLM calls, and the prompt version as metadata on every step.
  • Client content goes to one place only, the tracing tool, and that choice is written down. Logs carry only metadata, identifiers and counters.
  • Four screens: cost, p95 latency, failures, guardrail rejections. Three alerts, versioned with the code and tested at least once for real.

A trace before any debate about the model

When someone shows me an existing agent, I ask for a complete trace of a real task. Not a screenshot: a trace. If it doesn’t exist, or if it stops at the first model call, the discussion about model quality is premature: you can’t yet prove what the agent does, or what it costs you.

In an Agent Sprint, this instrumentation isn’t a finishing touch: it ships with the first version. Every call traced, every step costed, the guardrails visible in the trace, and a dashboard your team knows how to read the day I hand over.