Metraipolis › archive › dc_ff9ddd712d4c

The Testability Gap: A Pre-Registered Census of Naming versus Implementation in 59 Cognitive-Architecture LLM Systems — arXiv:2611.00651

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00651
License (as archived)
CC0-1.0
Archived
2026-11-03T18:41:36.183Z
SHA-256 of the archived text
2abf31424656c6be2dc5b8d074d1de8a0fc177b7d6c40ad8f8581886929ee3f9
Stable URL of this page
https://metraipolis.com/documento/dc_ff9ddd712d4c

How to cite

The Testability Gap: A Pre-Registered Census of Naming versus Implementation in 59 Cognitive-Architecture LLM Systems — arXiv:2611.00651 — arXiv (en), https://arxiv.org/abs/2611.00651 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff9ddd712d4c (Metraipolis archive dc_ff9ddd712d4c, sha256 2abf31424656c6be2dc5b8d074d1de8a0fc177b7d6c40ad8f8581886929ee3f9)
BibTeX
@misc{dcff9ddd712d4c,
  title = {The Testability Gap: A Pre-Registered Census of Naming versus Implementation in 59 Cognitive-Architecture LLM Systems — arXiv:2611.00651},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00651},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff9ddd712d4c; sha256 2abf31424656c6be2dc5b8d074d1de8a0fc177b7d6c40ad8f8581886929ee3f9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00651. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A companion line of work audited individual cognitive-architecture LLM systems in depth and found that module names and module computations are empirically independent claims. This paper asks how far that gap extends across the system class. We pre-registered a census protocol (sample frame, metrics, and acceptance criteria frozen before any sample was downloaded) and surveyed the 59 most-starred Python repositories declaring cognitive architecture on GitHub, plus five classic architectures as a historical control. We measure a testability gap. First, an auditability gap: 22% of repositories document no code reference whatsoever. Second, an integrity gap: 42% of repositories have at least one dangling reference, and within that subgroup the median fraction is 8%, the top quartile 19.6%, and the maximum 79.5%; one 148-star system documents paths of which a majority do not exist. Three of four pre-registered hypotheses were falsified: the integrity gap is not universal (58% of repositories either offer no checkable references or offer references that all resolve); claim density does not predict dangling rate (rho = -0.10); and the classical-architecture control does not significantly outperform (p = 0.059 with five controls, all at zero dangling). Exactly one repository in 59 (1.7%) ships executable falsification criteria for its own claims. A second pre-registered round classified every dangling path by git lifetime: 85.5% of dangling references (159 of 186) are ghosts - paths that never existed at any commit - against 14.5% that were once real and later removed.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.00651 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:13:07.742Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).