Metraipolis › archive › dc_bc989680ff21

Overview of the TREC 2025 Million Large Language Models track — arXiv:2609.31921

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31921
License (as archived)
CC0-1.0
Archived
2026-10-07T20:54:01.546Z
SHA-256 of the archived text
e11f07ec2a8e37cc8c69376e248dc52e99564843b25d282b40809db005fb6325
Stable URL of this page
https://metraipolis.com/documento/dc_bc989680ff21

How to cite

Overview of the TREC 2025 Million Large Language Models track — arXiv:2609.31921 — arXiv (en), https://arxiv.org/abs/2609.31921 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc989680ff21 (Metraipolis archive dc_bc989680ff21, sha256 e11f07ec2a8e37cc8c69376e248dc52e99564843b25d282b40809db005fb6325)
BibTeX
@misc{dcbc989680ff21,
  title = {Overview of the TREC 2025 Million Large Language Models track — arXiv:2609.31921},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31921},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc989680ff21; sha256 e11f07ec2a8e37cc8c69376e248dc52e99564843b25d282b40809db005fb6325},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31921. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic AI envisions ecosystems of intelligent agents collaboratively solving complex tasks with minimal human intervention. In such ecosystems, each agent possesses specialized expertise, making effective expert selection central to overall system performance. While most current approaches assume a small number of well-documented models, real-world expertise is far more diverse and cannot be adequately captured through static metadata or hand-written descriptions. We anticipate a future with millions of specialized language models (LLMs), each excelling in different domains or problem types. Rather than relying on predefined capability statements, we propose a retrieval-based paradigm in which an assistant agent infers expertise dynamically by examining models' observable behavior. Upon receiving a user query, the assistant ranks candidate LLMs based on demonstrated competence, enabling efficient and adaptive expert selection. The TREC Million LLM Track operationalizes this paradigm by shifting the retrieval target from documents to expert LLMs. Participants are given a discovery set consisting of queries, answers, and log-probabilities from more than one thousand LLMs and are challenged to infer meaningful expertise representations for each model. Given an unseen test query, systems must then rank the LLMs according to their expected performance, providing the first large-scale benchmark for expertise retrieval in agentic AI.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.MA).
Categorie arXiv: Information Retrieval (cs.IR) ; Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.31921 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:24:41.650Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).