You make LLM tracing and evaluation pay off in production by treating the trace — not the HTTP request, and not the prompt — as the unit of quality. Emit one instrumented span per logical model call, attach automated evaluations to a sampled slice of those traces, and alert on evaluation score drift with the same seriousness you apply to error rates. The OpenTelemetry GenAI semantic conventions supply the shared schema for the first half: spans, metrics, and events for GenAI clients, MCP, and provider-specific conventions including OpenAI. Evaluation platforms supply the second half by running judges against the spans you already export.
The rest of this guide covers what the conventions actually require a span to capture, which attributes deserve enforcement in code review, and a concrete workflow for wiring sampled online evaluations on top so a silent quality regression appears as a dashboard line rather than a support ticket.
What a Trace Must Capture
The conventions model GenAI spans as logical operations observed by the caller. In concrete terms, a span SHOULD cover the full duration of a logical operation, including all automatic retries, from initiation until the response is fully received or the operation is terminated by an error or cancellation. Two engineering consequences follow. First, retries your SDK performs transparently belong inside the same span, which keeps tail latency honest: a request that internally failed twice shows up as one slow span instead of three fast spans and one failure. Second, cancellation is an explicit end state, so client-side timeouts become visible in tracing instead of vanishing between requests.
Retry semantics also connect tracing to operational hygiene. If a burst of retries coincides with credential rotation, the trace should make that correlation obvious rather than something you reconstruct from logs; the same expectations shape a zero-downtime API key rotation runbook, where a single failed first attempt after a key swap is expected behavior, not an incident.
The Span Schema That Matters
The GenAI semantic conventions define signals for events, exceptions, metrics, inference token metrics, model spans, and agent spans, with technology-specific conventions for Anthropic, Azure AI Inference, AWS Bedrock, and OpenAI. Every one of these documents is still published at Development status, which is not a footnote: it means attribute names and requirement levels can change between releases, so pin your instrumentation library version and add a contract test that asserts the attributes your dashboards depend on.
The attributes below are the ones worth enforcing, with the requirement levels the conventions attach to them:
| Attribute | Requirement level | What to verify |
|---|---|---|
| gen_ai.operation.name | Required | One of chat, generate_content, text_completion per operation type |
| gen_ai.provider.name | Required | openai, gcp.gen_ai, gcp.vertex_ai and similar identifiers |
| gen_ai.request.model | Conditionally required if available | Exact model string; keep history for deprecation planning |
| error.type | Conditionally required on error | timeout, 500, server_certificate_invalid instead of generic errors |
| gen_ai.response.status | Well-known values | completed, failed, cancelled, in_progress |
Because gen_ai.request.model records the exact identifier a provider served, it is also your earliest signal that a silent model swap happened upstream — which is why it belongs in every alert about score drift, and why model deprecation windows should be planned with the attribute history in hand.
From Traces to Evaluations
Tracing tells you a call happened and how long it took; evaluation tells you whether the answer was worth the tokens. Online evaluation closes the gap by scoring production traces continuously instead of a held-out test set. Arize, for example, runs online evaluations automatically as new spans arrive, lets you sample a set of production traffic for monitoring, and executes evaluation scripts every five minutes — a deliberate cadence that trades immediacy for cost, because evaluators are themselves model calls.
The pattern is not tied to one vendor. A defensible setup layers three evaluator classes: deterministic checks (JSON schema validity, refusal detection, latency budget breaches) on every exported trace; LLM-as-judge scoring on a random sample; and human review on a stratified slice of that sample to calibrate the judges. Alert on the score distribution, never on a single bad trace.
Sampling and Cost Controls
Full-fidelity export of prompts and completions is where observability budgets and privacy obligations collide. The decision table below reflects a workable default I use when instrumenting a new LLM feature; treat it as a starting point and tighten it per jurisdiction:
| Signal | Store on | Rationale |
|---|---|---|
| Attributes, token counts, latency, status | Every span | Cheap, structured, powers SLOs and cost dashboards |
| Full prompt and completion text | Sampled slice | Storage cost and PII exposure; redact before export |
| LLM-as-judge scores | Sampled slice | Judge cost scales linearly with judged traces |
| Human labels | Weekly stratified sample | Calibrates judges against ground truth |
Redaction belongs at the collector, not scattered through application code: a single choke point that strips secrets, keys and personal data before spans leave your perimeter is easier to audit than per-service scrubbing.
A Trace-to-Eval Workflow
- Instrument model clients with an OpenTelemetry GenAI-compatible SDK and assert the required attributes in a unit test.
- Export spans through a collector you control and enforce redaction there before anything leaves the perimeter.
- Group dashboards by gen_ai.request.model and prompt template identifier so provider-side changes surface as step changes.
- Run deterministic evaluators on every exported trace: schema validity, refusal markers, latency and token budgets.
- Sample a fixed fraction of traces for LLM-as-judge scoring on a schedule, and compare score distributions week over week.
- Escalate drift: freeze prompt versions, bisect by model and template attributes, and replay the sampled traces against the previous version.
- Feed every human review back into the judge prompt so calibration improves instead of decaying.
Sources
- OpenTelemetry GenAI semantic conventions — repository (OpenTelemetry)
- Semantic conventions for generative AI systems — overview (OpenTelemetry)
- Semantic conventions for generative client AI spans (OpenTelemetry)
- Online LLM Evaluations in Arize (Arize AI)