Log the envelope, not the letter. For LLM observability, the telemetry that answers most operational questions — latency, token counts, model version, caller identity, error type — never needs the raw prompt, and on managed platforms the provider already writes the full request and response for you. Amazon Bedrock, for example, can collect the full request data, response data, and metadata associated with supported calls performed in your account in a Region, delivered to Amazon S3 or CloudWatch Logs. Your tracing tier should carry identifiers and counters; a separate, access-controlled store should carry payloads; and the overlap between the two should be deliberate and small. This guide covers what belongs in each tier, what must be redacted before anything is shipped, and a retention policy you can enforce with bucket policies rather than goodwill.
The two-tier logging model
Most teams over-log prompts because they conflate two needs: debugging a bad generation, and monitoring a healthy service. Split them. Tier 1 is the trace store — OpenTelemetry spans and metrics with model identifiers, token counts and latency. Tier 2 is the payload store — the raw prompt and completion JSON, written by provider-side invocation logging or your gateway, encrypted, tightly scoped and, ideally, not written by your application at all. When an incident needs the actual text, you join on request ID and pull exactly one record from Tier 2 with a break-glass access, instead of grepping a trace store that contains every customer conversation.
Both Bedrock destinations enforce this split naturally: log events carry metadata inline, while binary data or JSON bodies larger than 100 KB are stored as separate objects in S3 rather than in the log event itself. Your design should mirror that boundary.
What to log in traces
Tier 1 fields earn their place by being useful in aggregate and safe by construction. The OpenTelemetry GenAI semantic conventions exist precisely for this: a standard set of attribute keys such as gen_ai.request.model, gen_ai.response.id and gen_ai.operation.name for chat and generate_content operations, maintained across instrumentation vendors. Build your spans on those keys so dashboards survive SDK churn.
| Field | Always log? | Why |
|---|---|---|
| request id, trace id | Yes | Join key between tiers; never contains content |
| model and version | Yes | Attribute regressions to model changes |
| input and output token counts | Yes | Cost, quota and capacity analysis |
| latency, streaming flag, finish reason | Yes | SLO tracking; truncated vs stopped output |
| error type, status code | Yes | Failover and retry logic evidence |
| prompt hash | Yes | Dedupe prompt versions without content |
| caller identity or team tag | Yes | Chargeback and blast-radius analysis |
| raw prompt or completion text | No | Belongs in Tier 2 only |
| user identifiers, emails, free-text feedback | No | PII; redact at the edge before shipping |
A prompt hash is the cheapest high-value field: a SHA-256 of the rendered prompt template plus versioned variable names lets you count how many requests used each prompt variant, correlate quality regressions with prompt changes, and still ship the span to any vendor. The same reasoning applies to sampling decisions — you can sample Tier 1 aggressively because every field is a counter or an identifier.
What to redact at the edge
Redaction must happen before the first write, in-process, not in a downstream processor. Rules to enforce in the instrumentation layer: strip or mask direct identifiers (names, emails, phone numbers, national ID patterns) from any field that reaches Tier 1; never place retrieved RAG chunks into span attributes — log the chunk IDs and source document hashes instead; for tool-calling agents, log tool names, argument schemas and a hash of arguments, never raw argument JSON, which is where API keys and internal URLs leak. If your workload genuinely requires payload samples in telemetry, sample them per-intent with an allowlist, never a blocklist: allowlist what may appear, redact everything else, and treat any new field as redacted by default.
Also separate what the provider already does from what you must do. On the Microsoft side, the data privacy documentation for Models sold by Azure states that your prompts and completions are NOT used to train any generative AI foundation models without your permission or instruction, and are not available to other customers or to OpenAI. That governs the provider’s handling of your data — it does not govern your own telemetry pipeline, your vendor SDKs, or the S3 bucket your team can read with a normal role. Compliance at the provider never substitutes for redaction at the source.
Provider logging you did not write
Many incidents originate in telemetry the team forgot existed. Bedrock model invocation logging is account-wide once enabled: data is logged for all models that support the modalities you choose — text, image, embedding or video — whether as input or output. Enable text-only logging when image or embedding bodies would balloon storage and exposure. Two Bedrock details matter for compliance reviews: invocation logging is disabled by default, and after it is enabled, logs are stored until the logging configuration is deleted. Meanwhile, only the bedrock-runtime endpoint is captured — calls made through other endpoints, such as the same OpenAI-compatible APIs on bedrock-mantle, are not currently captured, so an inventory of your endpoints is part of your observability audit. Add these provider-side stores to your data inventory, give them the same retention and access review as your own, and remember that whoever can read the log destination can read every prompt.
A retention and access checklist
Enforce policy with infrastructure, not documentation. This ordered procedure has held up in regulated EU deployments:
- Fix retention in the destination — S3 lifecycle rules or CloudWatch log retention — not in the shipper, which can be bypassed.
- Grant read on Tier 2 payloads to a role, never to humans; break-glass access goes through a time-boxed elevation with audit.
- Encrypt Tier 2 with a customer-managed key so revocation is real; provider-managed keys cannot be selectively denied.
- Run a monthly re-identification check: sample 100 payloads and attempt to identify a user from content plus metadata. If you can, your redaction rules have gaps.
- Log every Tier 2 read as a security event, and alert on reads without a linked incident ticket.
- Review provider-side logging configuration quarterly — endpoints and modalities drift as platforms add features.
Tier sizing and cost follows the same discipline you would apply to capacity: if your log volume surprises you at invoice time, the analysis approach in AI gateway rate limiting and quota design applies to telemetry quotas too, and the tracing patterns that feed evaluation work were covered in LLM evaluation pipelines in CI.