Metraipolis › archive › dc_bde19f06b237

SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization — arXiv:2608.04084

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04084
License (as archived)
CC0-1.0
Archived
2026-10-31T11:25:28.203Z
SHA-256 of the archived text
1cf3992ffc9a3884a988c85a7ea949e34ffc99cac45f141d80d02ef7d6376894
Stable URL of this page
https://metraipolis.com/documento/dc_bde19f06b237

How to cite

SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization — arXiv:2608.04084 — arXiv (en), https://arxiv.org/abs/2608.04084 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bde19f06b237 (Metraipolis archive dc_bde19f06b237, sha256 1cf3992ffc9a3884a988c85a7ea949e34ffc99cac45f141d80d02ef7d6376894)
BibTeX
@misc{dcbde19f06b237,
  title = {SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization — arXiv:2608.04084},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04084},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bde19f06b237; sha256 1cf3992ffc9a3884a988c85a7ea949e34ffc99cac45f141d80d02ef7d6376894},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04084. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing algorithm, or the alignment between training-signal granularity and the target categories? We probe the question with SpecDrop, a fixed parameter-free routing scheme: each of $K$ branches receives weight $p_a$ for its assigned category and a small leakage $p_i > 0$ otherwise, merged through a category-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference. On vision tasks where each image has one superclass label (CIFAR-100 on ResNet-110; ImageNet-1K on ViT-S/16), SpecDrop reaches 79.23% on CIFAR-100 and 79.89% on ImageNet-1K, exceeding parameter-matched baselines that do not use the label (+4.75 over dense on CIFAR-100; +6.53 over the No-Routing+SE control on ImageNet-1K). These gains quantify what category supervision buys when deployed through routing -- not an advantage over label-aware deployments of the baselines: given the same label, masking a dense model's outputs is stronger for accuracy alone (85.2 / 83.7). SpecDrop's contribution is converting the label into trained-in modular structure: 68%/100% branch-category alignment, and masking gains of 0.00 (CIFAR) / +1.06 (ImageNet) -- the output-space restriction is largely internalized during training. On fuzzy partitions, where training units span multiple categories (SlimPajama-6B language modeling with a 30M Transformer; SuperNI instruction tuning over Llama-3.2-1B with LoRA), the routing mechanism reduces to the matched No-Routing controls within seed noise, the null our thesis predicts. Granularity alignment, not algorithm choice, localizes when routing helps. Code: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.04084 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:13:23.204Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).