Metraipolis › archive › dc_afca5f036aa0
BCMT: Blockwise Causal Memory Transformer — arXiv:2608.13578
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.13578
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T02:20:57.472Z
- SHA-256 of the archived text
84b31c6926e7049d28d0f3630cc58fe2f309f126932afff573b084ad67ecb050- Stable URL of this page
https://metraipolis.com/documento/dc_afca5f036aa0
How to cite
BCMT: Blockwise Causal Memory Transformer — arXiv:2608.13578 — arXiv (en), https://arxiv.org/abs/2608.13578 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_afca5f036aa0 (Metraipolis archive dc_afca5f036aa0, sha256 84b31c6926e7049d28d0f3630cc58fe2f309f126932afff573b084ad67ecb050)
BibTeX
@misc{dcafca5f036aa0,
title = {BCMT: Blockwise Causal Memory Transformer — arXiv:2608.13578},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.13578},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_afca5f036aa0; sha256 84b31c6926e7049d28d0f3630cc58fe2f309f126932afff573b084ad67ecb050},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13578. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length. We introduce BCMT (Blockwise Causal Memory Transformer), an architecture for long-context language modeling that decouples local token interactions from global context propagation. Dense causal self-attention is applied independently within local blocks, while each block produces an adaptive summary aggregated through an exponential causal memory. This memory is subsequently injected back into the token representations, enabling efficient propagation of long-range contextual information without relying on explicit global attention. Unlike standard Transformers and recurrent memory architectures, BCMT maintains neither dense interactions between distant tokens nor learned memory states. Its memory mechanism is fully parallelizable and remains compatible with standard implementations of dense self-attention. Experiments on language modeling with context lengths of up to 1024 tokens show that BCMT achieves validation performance comparable to that of Dense Transformers while significantly improving training throughput and reducing memory consumption. An ablation study further confirms that these improvements arise from the proposed memory mechanism. These results demonstrate that an exponential causal memory constructed from block summaries provides an effective alternative to dense global attention mechanisms for long-context language modeling. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.13578 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T02:02:02.842Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).