Metraipolis › archive › dc_f492b01d89fc

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs — arXiv:2610.33949

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.33949
License (as archived)
CC0-1.0
Archived
2026-10-27T08:12:38.907Z
SHA-256 of the archived text
0c91f2cfe5e39849888e7e8b5b5d40586b300f2f50e8a0ae354764bd31e1ce2e
Stable URL of this page
https://metraipolis.com/documento/dc_f492b01d89fc

How to cite

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs — arXiv:2610.33949 — arXiv (en), https://arxiv.org/abs/2610.33949 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f492b01d89fc (Metraipolis archive dc_f492b01d89fc, sha256 0c91f2cfe5e39849888e7e8b5b5d40586b300f2f50e8a0ae354764bd31e1ce2e)
BibTeX
@misc{dcf492b01d89fc,
  title = {Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs — arXiv:2610.33949},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.33949},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f492b01d89fc; sha256 0c91f2cfe5e39849888e7e8b5b5d40586b300f2f50e8a0ae354764bd31e1ce2e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33949. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pipelines that require structured outputs from hosted language models often specify a Pydantic schema in the prompt, validate the response, and retry when validation fails. Although native structured-output modes provide an alternative, application-side validation remains important for enforcing custom constraints. Yet pipeline reliability is often summarized by a single validity rate, which obscures why outputs fail and which interventions may help. We introduce a fine-grained taxonomy of schema-conformance failures and a deterministic classifier that assigns failure categories from model responses and Pydantic validation errors. We apply this framework to 3,997 generation attempts across nine OpenAI and Anthropic model/API configurations, using questions from HotpotQA and FinanceBench. Our results show that similar validity rates can mask different failure patterns, and that dominant failure categories vary with schema constraints and task. Markdown-fenced JSON is the most common failure: among 681 fenced failures, only 48.0% pass validation after fence removal, while the remainder conceal additional violations, predominantly field-length violations. In an exploratory prompt ablation, variants that suppress fences appear to shift failures toward length violations, but the shift is consistent with exposing violations that were already present, illustrating how aggregate validity rates can obscure changes in failure composition. We derive four engineering practices for validator, schema, retry and evaluation design, three supported by our data and one, on retry feedback, proposed but not tested, and release the classifier, completions, and annotations.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.33949 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.298Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).