Metraipolis › archive › dc_bfd18b8048af
Audio Token Attention Is Predictable Before the Language Model Runs — arXiv:2609.38878
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38878
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:44:27.856Z
- SHA-256 of the archived text
113e98f567c8b8eda6aabab71146b9ca2ced82c8767d026da6274a0f9582c51e- Stable URL of this page
https://metraipolis.com/documento/dc_bfd18b8048af
How to cite
Audio Token Attention Is Predictable Before the Language Model Runs — arXiv:2609.38878 — arXiv (en), https://arxiv.org/abs/2609.38878 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bfd18b8048af (Metraipolis archive dc_bfd18b8048af, sha256 113e98f567c8b8eda6aabab71146b9ca2ced82c8767d026da6274a0f9582c51e)
BibTeX
@misc{dcbfd18b8048af,
title = {Audio Token Attention Is Predictable Before the Language Model Runs — arXiv:2609.38878},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38878},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bfd18b8048af; sha256 113e98f567c8b8eda6aabab71146b9ca2ced82c8767d026da6274a0f9582c51e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38878. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at $\rho \geq .69$ on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: this https URL Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Sound (cs.SD) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.38878 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:13.963Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).