Production LLM observability stands on two legs: distributed tracing that records every model call, and evaluation that scores whether those calls were any good. The OpenTelemetry GenAI semantic conventions now give both legs a shared foundation, because they define a common span vocabulary for generative AI operations that any SDK or backend can emit and consume. If your team already runs an OpenTelemetry collector for services, extending it to model calls is the cheapest credible path to LLM observability, and the OpenTelemetry GenAI semantic conventions define spans, metrics, and events for GenAI clients including MCP and provider-specific conventions, which means traces from different frameworks land in one queryable format instead of five vendor silos.

Why tracing alone is not enough

A trace tells you what happened; it does not tell you whether the answer was correct, grounded, or safe. That gap is where incidents hide. A RAG pipeline can return fast, cheap, perfectly structured responses while semantic recall quietly degrades after an index rebuild, and nothing in latency or token counters will flag it. The closed loop that production teams converge on is trace, evaluate, collect datasets, improve prompts, redeploy. Tracing without evaluation gives you the first step only; evaluation without tracing gives you scores you cannot drill into when they drop.

Treat the two as one pipeline from day one: every trace should be scoreable, and every score should link back to the trace that produced it. Teams that bolt evaluation on later typically discover their early traces lack the attributes needed to reconstruct context, which forces a painful re-instrumentation pass.

The OpenTelemetry GenAI span model

The GenAI conventions describe telemetry as observed by the caller, and per the specification GenAI spans represent logical operations as observed by the caller, covering the duration from initiation until the response is fully received or the operation terminates through error or cancellation. Two practical consequences follow. First, if the client retries a transient failure internally, the span should still describe the whole logical operation including all retries, not one attempt per span, or your latency percentiles will lie to you. Second, span naming follows a predictable pattern built from the operation name and the requested model, so dashboards can group by model without custom parsing.

The attribute registry is the part your team will actually code against. gen_ai.operation.name distinguishes chat, embeddings, retrieval, execute_tool and agent operations. gen_ai.provider.name acts as a discriminator for the telemetry flavor, so spans for AWS Bedrock carry aws.bedrock and are not expected to mix in openai.* attributes. Token usage, request model and response model are first-class attributes, which is what makes cost attribution per trace possible without log scraping.

Content capture is the design decision that matters most. Recording full prompts and completions on gen_ai.input.messages and gen_ai.output.messages is opt-in, not default, and for EU deployments handling personal data it usually should stay off in the hot path. The conventions support hooks that route content to external storage with only references left on the span, which is the right shape for regulated workloads: metadata on the span, payloads in an object store you control, joined at investigation time.

Choosing a trace backend

With spans standardized, the backend becomes an operational choice rather than an instrumentation bet. Langfuse can act as an OpenTelemetry backend and ingest traces through its /api/public/otel OTLP endpoint, which accepts HTTP with JSON or protobuf payloads; gRPC ingestion is not supported yet, so plan your collector exporters accordingly. Langfuse is an open source LLM engineering platform that teams can self-host, built on the ClickHouse open source database, which makes it a natural fit where prompt content must not leave your infrastructure. Phoenix accepts traces over OpenTelemetry OTLP and ships auto-instrumentation for frameworks such as LlamaIndex and LangChain plus providers including OpenAI, Bedrock and Anthropic, with evaluation as the center of its workflow rather than an afterthought.

Because both self-hosted options speak OTLP, the decision is reversible. Emit standard spans through a collector and switching backends becomes a configuration change, not a rewrite. Instrument once at the OTel layer and keep backend-specific attributes in a thin wrapper, so a future migration does not touch business code.

Wiring evaluation into delivery

Evaluation earns its keep when it runs without a human pressing buttons. The pattern that works in CI: curate a golden dataset from real production traces, define a small set of evaluators, and gate deploys on their scores. Mix three evaluator types: deterministic checks for format and refusal behavior, retrieval-quality checks such as context precision and groundedness for RAG paths, and a cheap model acting as judge for open-ended quality. Pin the judge model and prompt version, or your eval scores drift for reasons unrelated to your changes.

Two failure modes dominate. Unversioned prompts make regressions unattributable, so every trace must record which prompt version produced it. And synchronous trace export in the request path can multiply tail latency; export spans asynchronously through a background processor and batch aggressively, since LLM call volumes make per-request flushes expensive.

A rollout checklist

1. Adopt the GenAI conventions via an instrumentation library; do not hand-roll span attributes.
2. Keep content capture off by default; enable hooks to external storage where debug access is required.
3. Propagate trace-level attributes such as user and session identifiers onto every span, using baggage, so aggregation works.
4. Stand up one backend behind your collector; start with cost and latency dashboards.
5. Sample real traces into a dataset; add evaluators one at a time.
6. Wire the eval suite as a CI gate before the next model or prompt change ships.

Trace-level cost attribution and eval-gated deploys are the two capabilities that separate observability theater from an operable LLM service. For cost context beyond per-call tokens, see our cloud AI cost optimization playbook, and for the surrounding platform decisions our search query analysis for engineers covers the tooling landscape. Budget for the observability stack the way you budget GPU capacity: as a line item that scales with traffic, decided before launch rather than after the first incident.

Sources

• OpenTelemetry GenAI semantic conventions repository
• Semantic conventions for generative client AI spans
• Langfuse OpenTelemetry integration documentation
• Langfuse open source repository
• Arize Phoenix documentation