Structured Outputs: Getting Reliable JSON From LLMs

Structured outputs solve the most common integration failure in production LLM features: the model returns prose, truncated JSON, or a payload that parses but does not match the contract your code expects. Schema-constrained decoding fixes this at the token level. You hand the API a JSON Schema instead of a prompt instruction, and the provider restricts generation so the response conforms. OpenAI documents that Structured Outputs ensures the model will always generate responses that adhere to your supplied JSON Schema, so an omitted required key or a hallucinated enum value stops being a parsing problem your service owns. This guide covers how the mechanism works on both major providers, the failure modes it does not remove, and the thin validation layer that still belongs in your pipeline.

How constrained decoding works

A language model generates one token at a time, sampling from a probability distribution over its vocabulary. Constrained decoding masks that distribution: at each step the provider computes which tokens could legally continue a string that satisfies your schema, sets every other token’s probability to zero, and samples from what remains. When the schema says the next field is an enum with three values, no other token can be produced there. Schema adherence becomes a property of the decoder, not of the prompt — which is why asking harder in the prompt (“return ONLY valid JSON, no markdown fences”) is a weaker mechanism than the same instruction enforced by a mask. The trade-off is real: everything the schema forbids is genuinely unreachable, including useful content the model wanted to add, so the schema you write is also a ceiling on what the model can express. Write schemas that describe the payload you need, not the payload you hope to police.

OpenAI strict mode rules

On the OpenAI API the mechanism is the json_schema format with strict: true. JSON mode and Structured Outputs both ensure valid JSON is produced, but only Structured Outputs ensure schema adherence — and OpenAI recommends Structured Outputs over JSON mode wherever the model supports it. Strict mode also narrows the accepted schema language: the supported examples show fully enumerated required lists and additionalProperties: false at each object level, with optional fields modeled as nullable unions rather than omitted keys. Two operational rules follow. First, derive the wire schema from your Pydantic or Zod types in CI instead of hand-writing a second copy — the documentation flags schema divergence between code types and JSON Schema as a recurring failure, and a CI rule that regenerates one from the other removes it. Second, version your schemas: a strict response is only reproducible against the schema version that constrained it, so store the schema hash next to every parsed output you persist.

Gemini response schema options

Google’s Gemini models implement the same idea through two parameters: response_mime_type set to application/json, plus a response_schema. The documentation is blunt that setting response_mime_type to application/json without specifying a response_schema acts only as a strong hint and carries a small risk of generating malformed JSON; the guaranteed path requires both parameters together. Schema support is a subset — only string enums are supported, and propertyOrdering controls the order in which properties are generated, which matters if you parse streaming output incrementally. Two limits are easy to miss: the response schema’s size counts toward your input token limit, and a complex schema can fail the request outright with an InvalidArgument error. Before debugging anything else, flatten nested arrays, shorten property names, and reduce the number of optional properties.

Failure modes to handle

Constrained decoding removes malformed JSON, not bad answers. Four failure classes survive the mask. First, refusals: when input trips a safety filter the model refuses, and because a refusal does not necessarily follow the schema you supplied, the OpenAI API returns a dedicated refusal field instead of parsed output — your consumer must branch on that field rather than crash on a missing key. Second, schema-valid nonsense: a value that satisfies the schema can still be factually wrong, so enum constraints validate shape, never correctness. Third, truncation: if generation hits the output token limit mid-object, the payload is both schema-constrained and unparseable, so check the finish reason before parsing. Fourth, data handling: a schema constrains the response, not what the provider stores around the request. If the unstructured text you are extracting contains customer PII, pair the pipeline with a zero-data-retention audit, and for EU workloads keep processing inside the regions your data residency plan already covers.

A validation workflow that holds

Keep a thin validation layer even with constrained decoding active: it is what makes refusals, truncation and schema drift observable instead of silent. The workflow below assumes a synchronous API, but the same steps apply to batch and streaming paths.

1. Define your types once in code (Pydantic or Zod) and derive the JSON Schema from them in CI; fail the build when the two diverge.
2. Enable schema-constrained modes — strict: true on OpenAI, response_schema plus application/json on Gemini — never prompt-only JSON in production paths.
3. Branch on refusal fields and finish reasons before attempting to parse the body.
4. Re-validate every parsed object against the recorded schema hash, and route mismatches to one bounded retry followed by a human review queue, not silent acceptance.
5. Alert on refusal rate and retry rate alongside HTTP errors; both are early signals that your input distribution has shifted.

The comparison below summarizes why the extra parameters are worth the plumbing:

Treat the table’s last row as the default for any new integration, and treat the validation workflow as the safety net that catches the cases the decoder cannot: refusals, truncation, and correct-looking answers that are simply wrong.

Sources

• OpenAI — Structured model outputs (accessed 2026-10-25)
• Google Cloud — Structured output for Gemini models (page last updated 2026-10-21)