Metraipolis › archive › dc_75bd36aa32fe

Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following — arXiv:2609.27421

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27421
License (as archived)
CC0-1.0
Archived
2026-10-08T10:57:11.120Z
SHA-256 of the archived text
69797e9c66b396f9ec1e8255b1b4088dbf3e6099920dbf330f2d3355f557ac06
Stable URL of this page
https://metraipolis.com/documento/dc_75bd36aa32fe

How to cite

Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following — arXiv:2609.27421 — arXiv (en), https://arxiv.org/abs/2609.27421 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_75bd36aa32fe (Metraipolis archive dc_75bd36aa32fe, sha256 69797e9c66b396f9ec1e8255b1b4088dbf3e6099920dbf330f2d3355f557ac06)
BibTeX
@misc{dc75bd36aa32fe,
  title = {Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following — arXiv:2609.27421},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27421},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_75bd36aa32fe; sha256 69797e9c66b396f9ec1e8255b1b4088dbf3e6099920dbf330f2d3355f557ac06},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27421. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-constraint instruction following requires a model to respond to a query under many simultaneously active constraints. Even strong instruction-tuned models still routinely violate some of them. Existing approaches either augment supervision with sequence- or token-level RL rewards from external verifiers or learned graders, or use on-policy distillation (OPD) against a single full-context teacher whose probability mass becomes diluted as more constraints become simultaneously active. We propose CC-OPD (Counterfactual Constraint-Conditioned On-Policy Distillation), which inverts the standard supervision-generation direction in distillation. Rather than enriching the teacher with information beyond what the student sees, CC-OPD ablates each constraint from the teacher's conditioning in turn, and constructs the per-constraint signal from the resulting per-token probability differentials. The resulting per-token leave-one-out log-likelihood shifts are summed, clipped, and added to the vanilla OPD reward as a token-level shaping term. All shaping terms are obtained from the frozen teacher, without an external verifier during distillation, and the reward equals vanilla OPD wherever the aggregate shift is zero. Across two Qwen model pairs and seven benchmarks, CC-OPD achieves the highest average among all evaluated student-training methods. A 1.5B student trained with CC-OPD surpasses its own 7B RL-trained teacher on the MulDimIF benchmark.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.27421 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:06:16.731Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).