Metraipolis › archive › dc_3588fa68aae7
Intra-Prompt Parallel Decoding for Common-Context Question Answering — arXiv:2609.05707
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05707
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:34:41.827Z
- SHA-256 of the archived text
0df96c75e4cb1438a809d93633ddff9f64def08ce1db51a81bbf5d2a136d75ee- Stable URL of this page
https://metraipolis.com/documento/dc_3588fa68aae7
How to cite
Intra-Prompt Parallel Decoding for Common-Context Question Answering — arXiv:2609.05707 — arXiv (en), https://arxiv.org/abs/2609.05707 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3588fa68aae7 (Metraipolis archive dc_3588fa68aae7, sha256 0df96c75e4cb1438a809d93633ddff9f64def08ce1db51a81bbf5d2a136d75ee)
BibTeX
@misc{dc3588fa68aae7,
title = {Intra-Prompt Parallel Decoding for Common-Context Question Answering — arXiv:2609.05707},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05707},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3588fa68aae7; sha256 0df96c75e4cb1438a809d93633ddff9f64def08ce1db51a81bbf5d2a136d75ee},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05707. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In common-context question answering (CCQA) tasks, multiple input questions share a common context to base their answers from. However, Large Language Models typically generate each answer using an independent prompt. While existing batching and caching techniques help improve parallelism and reduce repeated computations, the separation of questions across prompts limits the achievable speedup, as modern GPUs are underutilized due to a memory bottleneck during attention. We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt. IPPD directly addresses the bottleneck by efficiently sharing both memory and computation during the attention process, as the next token for every question is decoded in a single inference step. IPPD uses virtual position IDs and attention mask manipulation to generate the same output as standard prompting without requiring fine-tuning or any changes to the LLM architecture. Since all parallelism occurs within a prompt, IPPD is fully compatible with batched inference, even when each prompt features a different context. Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.05707 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:24.974Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).