Metraipolis › archive › dc_a4be0935887f

LEGION: Benchmarking Language Agents on Sparse-Signal Investigation Under Adversarial Noise — arXiv:2611.11896

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.11896
License (as archived)
CC0-1.0
Archived
2026-11-10T18:41:37.023Z
SHA-256 of the archived text
530c99c3b6ae2dac501dee28f8293f36e4a0d86ac169ed8195027d019b61a34c
Stable URL of this page
https://metraipolis.com/documento/dc_a4be0935887f

How to cite

LEGION: Benchmarking Language Agents on Sparse-Signal Investigation Under Adversarial Noise — arXiv:2611.11896 — arXiv (en), https://arxiv.org/abs/2611.11896 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a4be0935887f (Metraipolis archive dc_a4be0935887f, sha256 530c99c3b6ae2dac501dee28f8293f36e4a0d86ac169ed8195027d019b61a34c)
BibTeX
@misc{dca4be0935887f,
  title = {LEGION: Benchmarking Language Agents on Sparse-Signal Investigation Under Adversarial Noise — arXiv:2611.11896},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11896},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a4be0935887f; sha256 530c99c3b6ae2dac501dee28f8293f36e4a0d86ac169ed8195027d019b61a34c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11896. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Autonomous language-model agents are increasingly deployed on discovery-driven tasks where signal is sparse and most retrieved information is irrelevant. Current benchmarks fail to evaluate this regime due to dense signals, cooperative environments, and ground truth typically being only a few tool calls away. We introduce Legion, a benchmark framework coupling a stochastic enterprise-telemetry simulator (Legion-Sim) with a tiered library of 74 expertly-crafted adversarial scenarios (Legion-Bench) and an evaluation protocol that scores investigation efficiency alongside detection accuracy. Scenarios are injected into freshly simulated companies, allowing independent control over confounder density and event volume. We test five agent architectures on two simulated companies and find that while recall is high (0.82-0.88) precision lags behind (0.61--0.72). They fail to find all of the indicators of compromise of the attack or mistake genuine indicators with attack related ones, which is critical in the context of cybersecurity investigation. Failures are consistent across reasoning tiers and persist across confounder density and event volume. We release Legion, the scenario library, performance on common agent architectures, and pipeline for reproducibility and extension.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.11896 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.526Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).