Metraipolis › archive › dc_7b1146973775

Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi — arXiv:2610.05682

Source
arXiv
Original
https://arxiv.org/abs/2610.05682
License (as archived)
CC0-1.0
Archived
2026-10-06T05:27:06.067Z
SHA-256 of the archived text
49f93bac975b85f502bbd8648ac3ba6c01a172321a49ce6eecf41b5efe9b034b
Stable URL of this page
https://metraipolis.com/documento/dc_7b1146973775

How to cite

Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi — arXiv:2610.05682 — arXiv, https://arxiv.org/abs/2610.05682 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7b1146973775 (Metraipolis archive dc_7b1146973775, sha256 49f93bac975b85f502bbd8648ac3ba6c01a172321a49ce6eecf41b5efe9b034b)
BibTeX
@misc{dc7b1146973775,
  title = {Knowing the Rules, Applying the Rules: Evaluating Language Models on Traditional Chinese Bazi — arXiv:2610.05682},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05682},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7b1146973775; sha256 49f93bac975b85f502bbd8648ac3ba6c01a172321a49ce6eecf41b5efe9b034b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05682. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowing domain rules does not guarantee applying them to a case. We study this distinction in traditional Chinese Bazi through 3,000 Chinese multiple-choice questions spanning 14 Theory and 11 Case categories. Six endpoint systems are evaluated, with primary results reported on a 2,492-item model-informed refinement. Theory accuracy exceeds Case accuracy for every system, and gaps of 16.60-29.56 percentage points remain when invalid responses are excluded. The contrast is more specific than a general case-reasoning deficit. Across six systems, Twelve Stages and Nayin reach mean accuracies of 89.10% and 88.62%, while Shensha Basics reaches 75.96%. Within Case, Luck Pillars averages 84.62%, but Career and Family Relations average only 36.98% and 38.19%. Overall rankings also conceal different category strengths. On the original 3,000 items, paired DeepSeek native/disabled comparisons associate native configurations with Theory gains of 6.53 and 12.20 points for Flash and Pro, respectively; Case changes are -3.67 and +1.27 points. These are provider-configuration associations, not isolated causal effects of reasoning. The results motivate task-specific evaluation of cultural-domain applications rather than reliance on aggregate knowledge scores. The benchmark measures agreement with a model-generated, model-verified answer key, not real-world predictive validity. Final-set results are post-selection descriptions, and incomplete provenance and expert validation constrain their interpretation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.05682 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.059Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).