Langfuse in Production: LLM Observability Beyond Console Logs

Your agent answered a question. Was it correct? How long did it take? What did it cost? Did it use the right prompt version? Langfuse answers all of those — and makes the answers actionable.

Updated 4 October 2026 with technical corrections.

A traditional web service fails loudly — a 500 status, a timeout, an exception in the logs. An LLM agent fails quietly. It answers with apparent confidence, burns through tokens, takes its time, and returns a response that's subtly wrong. Your application metrics show a successful request. Your users get a bad answer. Console logs tell you the call happened. They don't tell you whether it was good, what it cost, whether it used the latest prompt, or why the Critic stage took four times longer than usual.

That's the gap LLM observability fills. After running a four-stage multi-agent AI system in production for six months, here is what Langfuse actually surfaces — and what we would have missed without it.

What Langfuse Is

Langfuse describes itself as "an open-source AI engineering platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications." (langfuse.com). The numbers (as of August 2026): 33.6k GitHub stars, 90 billion observations processed per month, used by 21 of the Fortune 50. Version 3 moved trace storage to ClickHouse. Version 4 re-architected the data model around observations — Langfuse quotes up to 165× faster UI and API performance — and added full-text search, a filter search bar, the v2 Metrics and Observations APIs, and threshold-based alerts.

The platform has four pillars, and they are more integrated than they first appear:

1. Tracing — structured records of every LLM call, tool invocation, and retrieval step, with inputs, outputs, token counts, latency, and metadata
2. Prompt Management — version-controlled prompts stored centrally, decoupled from code deployment, cached client-side with zero latency impact
3. Evaluation — online scoring on live production traces, offline experiments comparing prompt or model variants, human annotation queues, and CI/CD integration
4. Metrics & Dashboards — cost, latency, volume, and quality trends with custom dashboards and threshold-based alerts

The integration matters because a quality regression shows up first in your evaluation scores, which link to the traces, which show you the exact prompt version that was active. Without that linkage you have three separate investigations. With it, you have one.

The Architecture: ADOT → Langfuse

Our production setup runs on EKS. Every pod — the orchestrator and the four pipeline stages, Data Scout, Analyst, Synthesizer and Critic — instruments with standard OpenTelemetry. The orchestrator pod runs an ADOT (AWS Distro for OpenTelemetry) collector sidecar that fans its spans out to X-Ray and Langfuse, and its metrics to CloudWatch. The four stage pods have no sidecar and export spans straight to Langfuse with the exporter below.

Orchestrator pod
  ├── Application (OpenTelemetry instrumented)
  │     └── OTLP/HTTP → localhost:4318
  └── ADOT Collector (sidecar)
        ├── X-Ray exporter       → X-Ray (distributed traces)
        ├── EMF exporter         → CloudWatch (pod metrics, error rates)
        └── OTLP HTTP exporter   → Langfuse  (LLM traces, token costs, evals)

Stage pods (Data Scout, Analyst, Synthesizer, Critic)
  └── Application → OTLP/HTTP straight to Langfuse

In the stage pods, the OTLP exporter points straight at the Langfuse ingest endpoint:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import base64

# Basic auth header from Langfuse public/secret key pair
langfuse_auth = base64.b64encode(
    f"{LANGFUSE_PUBLIC_KEY}:{LANGFUSE_SECRET_KEY}".encode()
).decode()

exporter = OTLPSpanExporter(
    endpoint="https://cloud.langfuse.com/api/public/otel/v1/traces",
    headers={
        "Authorization": f"Basic {langfuse_auth}",
        "x-langfuse-ingestion-version": "4",  # real-time ingestion on Langfuse v4
    },
)

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

The Langfuse SDKs send data asynchronously, batched in the background. There is no latency added to the request path — the application responds to the user while trace events are flushed in the background.

Tracing: What You Actually See in the UI

A single user research query that travels through all four pipeline stages produces one coherent trace tree in Langfuse:

• Top-level span: total wall-clock duration, total cost, user ID, session ID
• Data Scout span: tool calls to each data source (inputs, outputs, latency per tool)
• Analyst span: statistical analysis LLM call with the exact system prompt, the evidence payload, and the model's analytical output
• Synthesizer span: Opus 5 synthesis call with its thinking block (empty unless you request summarised thinking), token counts broken out (input, output, thinking)
• Critic span: adversarial verification call with the synthesis and the rejection/acceptance decision

The key to getting one coherent tree across four separate pods is W3C trace context propagation. The orchestrator injects the traceparent header into each HTTP call to a sub-agent; each sub-agent extracts it and starts its span as a child of the parent trace:

import requests
from opentelemetry.propagate import inject, extract
from opentelemetry import trace

# Orchestrator: inject context into outbound call
def call_sub_agent(url: str, payload: dict) -> dict:
    headers = {"Content-Type": "application/json"}
    inject(headers)  # adds traceparent, tracestate
    return requests.post(url, json=payload, headers=headers).json()

# Sub-agent: extract context from inbound request
def handle_request(raw_headers: dict, payload: dict):
    ctx = extract(raw_headers)
    with trace.get_tracer(__name__).start_as_current_span(
        "analyst-stage", context=ctx
    ) as span:
        result = run_analyst(payload)
        # Filterable metadata; token usage itself comes from the LLM call spans
        span.set_attribute("langfuse.observation.metadata.output_tokens", result["token_count"])
        return result

Cost Tracking: The Thinking Token Problem

Langfuse tracks per-token costs by model. You configure the model pricing in Langfuse's model settings — input cost per 1M tokens, output cost per 1M tokens. For most models this is straightforward.

For Claude Opus 5, which thinks by default, there is a catch: thinking tokens are billed as output tokens — the more expensive rate, not input. The thinking block (the model's internal reasoning before its final response) appears as a separate field in the API response. Additionally, when thinking blocks are replayed into subsequent turns as context, they are billed as input tokens at that point. If thinking tokens are missing from the usage Langfuse prices — or priced at the input rate — your cost estimates will be significantly lower than actual AWS Bedrock invoices.

With correct cost tracking, you can pull per-model cost breakdowns from the Langfuse Metrics API and expose them directly in a monitoring endpoint:

import json
from datetime import datetime, timedelta, timezone
from langfuse import Langfuse

langfuse = Langfuse()

def get_cost_summary(days: int = 30) -> dict:
    """Cost by model for the monitoring dashboard (Metrics API v2, Langfuse v4)."""
    now = datetime.now(timezone.utc)
    fmt = "%Y-%m-%dT%H:%M:%SZ"
    query = {
        "view": "observations",
        "metrics": [{"measure": "totalCost", "aggregation": "sum"}],
        "dimensions": [{"field": "providedModelName"}],
        "filters": [],
        "fromTimestamp": (now - timedelta(days=days)).strftime(fmt),
        "toTimestamp": now.strftime(fmt),
    }
    rows = langfuse.api.metrics.metrics(query=json.dumps(query)).data
    return {
        "period_days": days,
        "cost_by_model": {
            r.get("providedModelName") or "unknown": r.get("sum_totalCost") or 0
            for r in rows
        },
    }

Cost data from a typical month in our system:

Synthesizer and Critic together account for 60% of cost despite handling only the final two stages. That's the correct tradeoff for a research workload where synthesis quality is the primary deliverable — but you can't make that judgment without the data.

Prompt Management: Decoupling Iterations from Deployments

Our system prompt is approximately 8,900 tokens. It includes domain-specific reasoning heuristics, tool usage guidelines, output format instructions, and examples. We iterate on it regularly — roughly once a week, based on user feedback and corrections.

With the prompt hardcoded in the application, each iteration requires a code change, a code review, a build pipeline run, and a pod restart. For a weekly cadence, that's 52 deployments per year that exist solely to change text. With Langfuse Prompt Management, the prompt can live in Langfuse, and the application fetches it through the SDK:

from langfuse import Langfuse

langfuse = Langfuse()

def get_system_prompt():
    # SDK cache: served from memory, refreshed in the background (60 s TTL by default)
    prompt = langfuse.get_prompt("agent-system-prompt", label="production")
    return prompt, prompt.compile()

The SDK caches the prompt in memory and refreshes it in the background, so calls to get_system_prompt() return immediately. Flip the production label in the Langfuse UI and every pod picks up the new version within the cache TTL — 60 seconds by default. No deployment, no restart. Pass the prompt object to the generation (or wrap the call in propagate_attributes(prompt=prompt)) so each trace records the version.

One important note about prompt caching on AWS Bedrock: the system prompt is a prime candidate for Bedrock's prompt caching feature. Cache hits on the 8,900-token system prompt cut input cost for that prefix by about 90% (cache reads bill at a tenth of the input rate). Bedrock's prompt cache is server-side and keyed on the prefix, not the pod — a restart doesn't clear it, and every hit resets the 5-minute TTL. So when cost spikes right after a deployment, look first at what the release changed in the cached prefix: prompt text, tool definitions, or their order. A traffic gap longer than the TTL has the same effect; the 1-hour TTL ("ttl": "1h" on the cache point) covers that. Langfuse makes either case visible as a cost spike in the cost-per-query trend immediately after a deployment.

Evaluation: Turning Traces into Quality Signals

A trace tells you what happened. An evaluation tells you whether it was good. Langfuse provides three evaluation paths, and each answers a different question.

LLM-as-a-Judge (automated, online)

After each production query completes, a lightweight post-processing step can send the Critic's final output to a small, fast model such as Claude Haiku for automated evaluation on three dimensions: factual coherence (0–1), source citation quality (0–1), and query relevance (0–1). Each score is attached to the trace as a named score. Run it asynchronously after the user receives their response, and it adds no latency.

from langfuse import Langfuse

langfuse = Langfuse()

def score_response(trace_id: str, response: str, query: str):
    """Called async after the main response is returned."""
    # Use Haiku as the judge — cheap, fast, good enough for these dimensions
    scores = evaluate_with_llm(response, query)  # returns dict of dimension→float
    for dimension, value in scores.items():
        langfuse.create_score(
            trace_id=trace_id,
            name=dimension,
            value=value,
            comment="automated LLM-as-judge evaluation"
        )

Annotation Queues (human review, targeted)

Route traces where the Critic rejected the Synthesizer's output to a Langfuse annotation queue. A human reviewer reads the original query, the rejected synthesis, and the Critic's rejection reason, then marks the rejection as either valid (the synthesis had a real problem) or false positive (the Critic was overly strict). That ground truth calibrates the automated judge — specifically the factual coherence threshold that triggers a rejection.

Experiments (offline, pre-deployment)

Before deploying a new prompt version to production, run an offline experiment: take a dataset of 50 representative historical queries (stored in Langfuse Datasets), run both the current production prompt and the candidate prompt against them, and compare the LLM-as-judge scores. If the candidate's average factual coherence score drops more than 5% relative to the current prompt, it doesn't go to production.

This gates deployments on quality, not just test coverage. The experiment history in Langfuse shows every prompt version ever tested, the scores it achieved, and whether it was promoted to production.

What Langfuse Surfaced That We Wouldn't Have Found Otherwise

These are concrete things Langfuse revealed that were invisible before we had it:

The Critic rejection rate drift is the one that sticks with me. The rejection rate increased gradually over three weeks — well within the noise of day-to-day variance. No user complained explicitly; the Critic was doing its job. The synthesis was getting rejected, the user got a message saying the agent wasn't confident enough, and they moved on. Without the score trend in Langfuse, this would have looked like normal variance for months. The actual cause: a new data source we'd ingested had lower-quality summaries that were finding their way into synthesis inputs. The Critic was right to reject them.

Self-Hosting on EKS

Langfuse's core is MIT-licensed open source and can be deployed within your own cloud account; audit logs and data-retention policies are Enterprise add-ons when self-hosted. For regulated environments where sending production LLM inputs and outputs to a SaaS platform requires governance review, self-hosting keeps all trace data within your account boundary.

Langfuse v4 on EKS requires four dependencies:

The official AWS Terraform module provisions all four plus the Langfuse application itself. For lighter setups — particularly when you already have the dependencies running — the Helm chart can deploy just the application containers and point at the dependencies you already run.

The Three-Layer Observability Stack

Langfuse handles the LLM-specific layer. It doesn't replace infrastructure observability — it complements it. Our full observability stack:

Each layer catches failure modes the others miss. Langfuse catches a Critic rejection rate drift. Prometheus catches the pod that's consuming 7GB of memory and approaching its limit. CloudWatch catches the Bedrock API error spike when a model is temporarily unavailable in a region. Together they give a complete picture. None of the three alone does.

Disclaimer: The views and opinions expressed in this post are my own and do not represent those of my employer or any organisation I am associated with.