Metraipolis › archive › dc_2ac78de4fe3c

Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering — arXiv:2608.07038

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.07038
License (as archived)
CC0-1.0
Archived
2026-10-19T09:53:03.232Z
SHA-256 of the archived text
fabd024b76d8c10ab6d32d62240230f05e25bbaa43331e070454d0a09114a6c4
Stable URL of this page
https://metraipolis.com/documento/dc_2ac78de4fe3c

How to cite

Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering — arXiv:2608.07038 — arXiv (en), https://arxiv.org/abs/2608.07038 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ac78de4fe3c (Metraipolis archive dc_2ac78de4fe3c, sha256 fabd024b76d8c10ab6d32d62240230f05e25bbaa43331e070454d0a09114a6c4)
BibTeX
@misc{dc2ac78de4fe3c,
  title = {Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering — arXiv:2608.07038},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.07038},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ac78de4fe3c; sha256 fabd024b76d8c10ab6d32d62240230f05e25bbaa43331e070454d0a09114a6c4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07038. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20\% with human judgment, outperforming traditional automated metrics at 35.04\%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5\%-24.7\% and reduces API cost to 0.06$\times$-0.84$\times$ of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.

Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CR).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2608.07038 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:51:05.659Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).