Order-Only On-Policy Distillation

Order-Only On-Policy Distillation trains a student to match the teacher’s ordering of top-K tokens at student-generated prefixes, avoiding teacher probability values and temperature tuning.

On-policy distillation trains a student on its own generations by matching the teacher’s token probabilities. These probabilities entangle which tokens the teacher prefers with how confident it is, and the confidence reflects the teacher’s own capacity and calibration that a smaller student may be unable to reproduce. Because teacher temperature changes these values without changing the ordering, OPD is sensitive to the temperature choice. The paper asks whether the teacher’s ordering of candidate tokens is enough.

At each student-generated prefix, O3PD trains the student to reproduce the order of the teacher’s top-K tokens, without fitting their probability values. The objective is therefore invariant to the teacher temperature and needs no temperature tuning. The approach is evaluated across six teacher–student pairs on HMMT25, AIME24, and AIME25.

• O3PD outperforms OPD on the average over HMMT25, AIME24, and AIME25 in every teacher–student pair, by up to 2.9 points. §Abstract
• Gains reach up to 5.6 points on AIME25. §Abstract
• Other ranking losses yield similar gains, indicating that the benefit comes from ordinal supervision itself. §Abstract
• O3PD is invariant to the teacher temperature and needs no temperature tuning. §Abstract
• O3PD departs further from the teacher’s probabilities than OPD yet reproduces its ranking more faithfully and yields a stronger student. §Abstract

• Not stated in the abstract; the evaluation covers six teacher–student pairs, leaving generalization beyond those pairs open.
• Not stated in the abstract; the reported gains are summarized as up to 2.9 points on the benchmark average and up to 5.6 points on AIME25, without per-pair detail in the abstract.
• Not stated in the abstract; the analysis reports further departure from teacher probabilities but does not quantify student calibration in the abstract.

• On-policy distillation trains a student on its own generations by matching the teacher’s token probabilities. §Abstract
• Teacher probabilities entangle which tokens the teacher prefers with how confident it is. §Abstract
• Teacher confidence reflects the teacher’s own capacity and calibration, which a smaller student may be unable to reproduce. §Abstract
• OPD is sensitive to the teacher temperature choice even though temperature does not change the ordering. §Abstract
• O3PD trains the student at each student-generated prefix to reproduce the order of the teacher’s top-K tokens. §Abstract
• O3PD does not fit the teacher’s probability values and is invariant to the teacher temperature. §Abstract
• Across six teacher–student pairs, O3PD beats OPD on the average over HMMT25, AIME24, and AIME25 by up to 2.9 points. §Abstract
• The ranking, not the probability values, carries the knowledge that on-policy distillation needs to transfer. §Abstract