Metraipolis › archive › dc_afe775a6db9f

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature — arXiv:2609.04266

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04266
License (as archived)
CC0-1.0
Archived
2026-10-16T22:42:16.051Z
SHA-256 of the archived text
6450853f609fff13ac6536c65a6fc137e9a7440667e9c578200e792b697af975
Stable URL of this page
https://metraipolis.com/documento/dc_afe775a6db9f

How to cite

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature — arXiv:2609.04266 — arXiv (en), https://arxiv.org/abs/2609.04266 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_afe775a6db9f (Metraipolis archive dc_afe775a6db9f, sha256 6450853f609fff13ac6536c65a6fc137e9a7440667e9c578200e792b697af975)
BibTeX
@misc{dcafe775a6db9f,
  title = {Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature — arXiv:2609.04266},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04266},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_afe775a6db9f; sha256 6450853f609fff13ac6536c65a6fc137e9a7440667e9c578200e792b697af975},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04266. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~$\mu\mathrm{m}^{2}$ including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~$\mu\mathrm{m}^{2}$. The circuit achieves a 242.97-ns evaluation latency at 13.44~mW total power, corresponding to 25.5~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5\% of the ideal-softmax baseline.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AR).
Categorie arXiv: Hardware Architecture (cs.AR) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.04266 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:26:48.906Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).