Metraipolis › archive › dc_1237ce904856

A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer — arXiv:2610.07301

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.07301
License (as archived): CC0-1.0
Archived: 2026-10-07T11:25:36.949Z
SHA-256 of the archived text: d094adc5e08297f65784aa89784b0d0fe3430bfaecca3d3eb32fdb6d32901bd1
Stable URL of this page: https://metraipolis.com/documento/dc_1237ce904856

How to cite

A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer — arXiv:2610.07301 — arXiv (en), https://arxiv.org/abs/2610.07301 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1237ce904856 (Metraipolis archive dc_1237ce904856, sha256 d094adc5e08297f65784aa89784b0d0fe3430bfaecca3d3eb32fdb6d32901bd1)

BibTeX

@misc{dc1237ce904856,
  title = {A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer — arXiv:2610.07301},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07301},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1237ce904856; sha256 d094adc5e08297f65784aa89784b0d0fe3430bfaecca3d3eb32fdb6d32901bd1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07301. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sparse attention mechanisms have become increasingly important for transformer models processing long input sequences due to their lower computational and memory complexity compared to full self-attention. Longformer achieves this through a sliding-window attention mechanism that produces a structured banded sparse attention matrix. However, existing sparse transformer accelerators primarily target attention generation or unstructured sparsity, leaving sparse matrix--dense matrix multiplication (SpMM) for structured sparse attention largely unexplored. This paper presents a pipelined FPGA architecture for accelerating banded SpMM in Longformer. The proposed design exploits the predictable sparsity pattern of Longformer's attention matrix through a custom row-wise storage scheme with implicit indexing, eliminating the overhead of conventional sparse matrix formats while enabling regular memory accesses. The architecture employs parallel processing elements, pipelined adder trees, and a dual-path computation strategy to maximize throughput and hardware utilization. Implemented in Verilog and evaluated on an RFSoC platform using Vivado 2024.2, the accelerator sustains one complete dot-product result per clock cycle after an initial latency of 11 cycles while maintaining power consumption below 2.9 W. Operating at 100 MHz, the design achieves over 100 million dot-product outputs per second, demonstrating the effectiveness of directly exploiting structured sparsity for sparse transformer acceleration.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.AR, nuovi lavori del giorno).
Categorie arXiv: Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2610.07301 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:16:09.455Z dalla pagina https://arxiv.org/list/cs.AR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).