Structured Output and JSON Mode: What Actually Works in 2026

Getting an LLM to return valid JSON used to mean regex-parsing broken output, retrying failed calls, and writing increasingly desperate system prompts ("respond ONLY with JSON, no markdown, no explanation"). That era is mostly over. By 2026, every major provider enforces schema conformance at the token-sampling level rather than relying on the model to politely follow instructions (FutureAGI). But "solved" doesn't mean "uniform" — the providers took different technical paths, and those differences break naive cross-provider code.

How constrained decoding actually works

The mechanism underneath structured outputs is constrained decoding: at every generation step, the inference engine masks the next-token probability distribution so tokens that would produce an invalid partial output get their logits set to negative infinity before sampling (Let's Data Science). The model literally cannot emit a token that breaks the schema, because that token was never in the sampling pool to begin with.

This is different from prompting the model to "output JSON" and hoping. Prompting relies on the model's learned behavior; constrained decoding relies on a parser (context-free grammar, pushdown automaton, or finite-state machine) validating every candidate token in real time (TMLS).

The engineering has gotten fast enough that it's no longer a meaningful latency tax:

That's the infrastructure story. The provider-facing story is messier.

The three approaches: OpenAI, Anthropic, Gemini

Each provider exposes structured output through a different API surface, and the surface shapes what you can and can't do downstream.

OpenAI ships native Structured Outputs with direct Pydantic/JSON Schema support and constrained decoding baked into the Responses/Chat Completions API. Schema adherence is reported at 99.9% (DevTk.AI; FutureAGI).

Anthropic doesn't have a separate "JSON mode" — it repurposes tool use. You define your schema as a tool specification and force Claude to call that tool, and the tool-call arguments are your structured payload. Reported adherence is close behind OpenAI at 99.8%, but the mechanism is instruction-following plus tool-schema validation rather than token-level grammar constraint for the full completion (Rost Glukhov).

Gemini exposes responseSchema combined with responseMimeType: application/json directly on the generation config, using constrained decoding similar to OpenAI's approach, at roughly 99.7% adherence (Medium — Rost Glukhov).

Where a shared schema quietly breaks

The practical trap in 2026 is assuming one Zod or Pydantic schema is portable across all three APIs. It isn't. All three providers ship grammar-constrained JSON generation now, so the open question shifted from "will this be valid JSON" to "which subset of JSON Schema does this specific vendor actually honor" (Rost Glukhov).

Concretely:

• anyOf/oneOf unions, recursive $ref, and certain string format constraints (email, date-time) are supported inconsistently across providers.
• Anthropic's tool-use schema is a JSON Schema subset scoped to what a tool definition can express — it isn't the full JSON Schema spec.
• Gemini's responseSchema has its own list of unsupported keywords that silently get ignored rather than erroring.

If you build a schema against OpenAI's spec and pass the identical object to Anthropic or Gemini, you can get a 200 response that "validates" against a looser interpretation of your schema but is missing constraints you assumed were enforced. The fix is boring but necessary: maintain a lowest-common-denominator schema subset if you need multi-provider portability, or maintain provider-specific schema variants generated from one source of truth.

Streaming: a real architectural constraint, not a nitpick

This is the difference most likely to bite you in production. OpenAI and Gemini support field-by-field progressive parsing during streaming — you can start rendering a UI as the name field completes, before the description field has even started generating. Anthropic's tool-use response comes back as a single block at the end of the stream: you get the whole structured payload atomically, not incrementally (FutureAGI).

If your product needs a "typing" effect on structured fields — a form auto-filling live, a dashboard populating card-by-card — Anthropic's current architecture won't support that pattern for the tool-call output itself, even though plain text responses stream normally. Pick your provider around this constraint before you design the UI, not after.

A minimal cross-provider example

// OpenAI — native structured outputs
const response = await openai.chat.completions.create({
  model: "gpt-5",
  messages: [{ role: "user", content: prompt }],
  response_format: {
    type: "json_schema",
    json_schema: { name: "lead", schema: leadSchema, strict: true }
  }
});

// Anthropic — forced tool use as the JSON mechanism
const response = await anthropic.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  tools: [{ name: "extract_lead", input_schema: leadSchema }],
  tool_choice: { type: "tool", name: "extract_lead" },
  messages: [{ role: "user", content: prompt }]
});

// Gemini — responseSchema + responseMimeType
const response = await model.generateContent({
  contents: [{ role: "user", parts: [{ text: prompt }] }],
  generationConfig: {
    responseMimeType: "application/json",
    responseSchema: leadSchema
  }
});

Same intent, three different request shapes, three different partial-support surfaces for the same nominal schema.

When constrained decoding still isn't enough

Schema-valid JSON is not the same as correct JSON. Constrained decoding guarantees the output parses and matches your types — it says nothing about whether the model hallucinated a value that fits the schema perfectly. A confidence_score: 0.94 field that's perfectly typed as a float between 0 and 1 can still be fabricated. Structured output solves the parsing-failure class of bugs, not the hallucination class. Treat schema conformance as necessary infrastructure, not a correctness guarantee, and keep validating extracted values against your actual data sources downstream.

Actionable takeaway

If you're integrating structured output today: don't assume schema portability across providers — test your exact schema against each target API's actual supported keyword subset before shipping, not after a customer's edge-case payload breaks silently. If your UI needs progressive field rendering, that constraint alone may decide your provider choice, independent of model quality. And treat "valid JSON" and "correct JSON" as two separate problems that need two separate solutions — one is now infrastructure, the other is still yours to solve.

Sources: FutureAGI — Evaluating LLM Structured Output Modes (2026), Rost Glukhov — Structured Output Comparison (Medium), Rost Glukhov — Structured Output Comparison (glukhov.org), DevTk.AI — AI Structured JSON Output Guide 2026, Let's Data Science — How Structured Outputs and Constrained Decoding Work, TianPan.co — Structured Outputs and Constrained Decoding, TMLS — Structured Outputs and Constrained Decoding