Metraipolis › archive › dc_f2a54a91c1ac

Can Text-to-Image Models Draw from the Right Frame of Reference? — arXiv:2608.03357

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03357
License (as archived)
CC0-1.0
Archived
2026-10-31T12:58:23.885Z
SHA-256 of the archived text
d1ddd3ab53d1d05efe03d921d871c4edd619727e6d8ddf7338cbea4d92177839
Stable URL of this page
https://metraipolis.com/documento/dc_f2a54a91c1ac

How to cite

Can Text-to-Image Models Draw from the Right Frame of Reference? — arXiv:2608.03357 — arXiv (en), https://arxiv.org/abs/2608.03357 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f2a54a91c1ac (Metraipolis archive dc_f2a54a91c1ac, sha256 d1ddd3ab53d1d05efe03d921d871c4edd619727e6d8ddf7338cbea4d92177839)
BibTeX
@misc{dcf2a54a91c1ac,
  title = {Can Text-to-Image Models Draw from the Right Frame of Reference? — arXiv:2608.03357},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03357},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f2a54a91c1ac; sha256 d1ddd3ab53d1d05efe03d921d871c4edd619727e6d8ddf7338cbea4d92177839},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03357. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Spatial instruction following has become a crucial requirement for text-to-image (T2I) generation. A common challenge arises when directional expressions are interpreted under different frames of reference. For example, ``the left of'' may refer to the viewer's image coordinates or to the intrinsic orientation of an object, leading to different expected layouts. Existing T2I benchmarks reveal important layout failures, yet they rarely isolate whether models can follow a specified frame of reference when it differs from camera view. To mitigate this gap, we introduce FoR-T2I, a benchmark for evaluating this distinction with 1,200 prompt pairs built from controlled spatial layouts. In each pair, the camera-view (Cam) prompt states the target relation in camera view, while the frame-of-reference (FoR) prompt describes the same target placement through an oriented anchor object. Across 22 closed-source and open-source T2I models, mean final accuracy is 41.8\% lower on FoR prompts than on matched Cam prompts; even the best-performing model achieves only 44.3\% FoR accuracy. This suggests that current models struggle more when the same layout is described through an object's orientation rather than directly in image coordinates. We further analyze this gap by relation type and camera view, compare several training-free prompting and feedback-based mitigation strategies, and propose a VLM-gated rewriting approach that selects rewritten prompts using visual feedback, improving average FoR accuracy from 25.0\% to 29.2\% under the same generation budget.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.03357 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:45:33.244Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).