Metraipolis › archive › dc_f6629388edac
From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings — arXiv:2610.41375
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41375
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T05:23:27.852Z
- SHA-256 of the archived text
3704a80f9f0af15b2bbbff354db8ea3ae8e1f1c9310e707e38f3684b1a8658b9- Stable URL of this page
https://metraipolis.com/documento/dc_f6629388edac
How to cite
From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings — arXiv:2610.41375 — arXiv (en), https://arxiv.org/abs/2610.41375 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f6629388edac (Metraipolis archive dc_f6629388edac, sha256 3704a80f9f0af15b2bbbff354db8ea3ae8e1f1c9310e707e38f3684b1a8658b9)
BibTeX
@misc{dcf6629388edac,
title = {From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings — arXiv:2610.41375},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41375},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f6629388edac; sha256 3704a80f9f0af15b2bbbff354db8ea3ae8e1f1c9310e707e38f3684b1a8658b9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41375. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Choosing among the hundreds of thousands of available large language models (LLMs) requires a thorough understanding of each candidate model's properties. In practice, each property is measured by its own benchmark suite, so profiling a model on many properties means paying for many separate evaluations. To address this, model embeddings compress each model into a fixed-length vector from which many properties can be predicted. Existing methods, such as EmbedLLM, build these vectors from correctness patterns and question features, and often need large probe batteries to do so. However, many behaviors that are important when choosing a model, such as its verbosity, safety and style, show up more in how it answers than in whether it is correct. We introduce BULLPEN (Behavioral Understanding of Large Language models via Profile ENcodings), a framework that extends EmbedLLM in three ways. First, BULLPEN organizes the downstream tasks that prior work uses and adds 20 task benchmarks we built on behavior, style and provenance. Second, it is the first framework to read a model's own answer text jointly with its correctness and the question text. Third, it encodes models from a few hundred questions instead of the full 24,118-question pool. Across 260 open-weight models, our answer-text encoder scores 0.48 +/- 0.10 on the new tasks against 0.35 +/- 0.10 for the best prior encoder. Our encoders that read answer text keep at least 91% of their full-pool skill from only 400 questions (1.3% of the inference compute of the full question pool), while a correctness-only baseline keeps 83% with the same 400 questions. Because these embeddings need only model outputs, they support behavioral auditing and low-cost profiling, and could extend to API-only models. Code and data: this https URL Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.41375 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T05:18:52.117Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).