On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the β=1 member of a broader policy-optimization family, where β weights the KL penalty anchoring the student to a reference policy. This equivalence turns β from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce β-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of β selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that β-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back—without sacrificing the efficiency that makes OPSD practical.
We show that vanilla on-policy self-distillation is exactly the β=1 instance of KL-regularized policy optimization.
β explicitly trades off proximity to the reference policy against privileged teacher guidance.
The optimal policy is a geometric interpolation between the reference policy and privileged teacher.
Scheduled logit interpolation and return-to-go credit approximate the RL-derived solution while retaining OPSD's efficiency.
Derive the target with policy optimization, then train it with self-distillation.
The implementation changes the target and the credit assignment while preserving the structure of on-policy distillation.
The practical schedule controls wk = 1/βk in [0, 1], equivalently βk ≥ 1. The default 200-step schedule increases the teacher weight from 0.5 to 0.8, moving smoothly from a conservative target toward stronger teacher guidance.
At each sampled prefix, β-OPSD mixes logits from a stop-gradient copy of the current student and a fixed privileged teacher. This is a tractable local approximation to the trajectory-level geometric interpolant.
Responses are sampled from the current student. The student sees only the problem, while the teacher can additionally condition on the ground-truth solution as privileged training context.
Discounted return-to-go propagates future student-target mismatch to earlier tokens. The experiments use γ = 0.99; interpolant logits and return weights are detached, leaving the student log-probability as the gradient path.
Avg@12 performance at the 100-step checkpoint of the default 200-step schedule
| Method | AIME 2024 | AIME 2025 | HMMT 2025 | Average |
|---|---|---|---|---|
| Qwen3-8B | ||||
| Base model | 73.06 | 67.78 | 29.72 | 56.85 |
| Vanilla OPSD | 75.83 | 66.67 | 33.06 | 58.52 |
| SFT | 72.78 | 64.72 | 30.83 | 56.11 |
| GRPO | 75.83 | 68.61 | 31.11 | 58.52 |
| β-OPSD (ours) | 78.33 (+2.50) | 70.28 (+3.61) | 31.94 (-1.12) | 60.18 (+1.66) |
| Qwen3-4B | ||||
| Base model | 77.78 | 64.17 | 29.44 | 57.13 |
| Vanilla OPSD | 75.00 | 65.00 | 28.33 | 56.11 |
| SFT | 23.61 | 19.72 | 10.56 | 17.96 |
| GRPO | 76.39 | 64.17 | 30.28 | 56.95 |
| β-OPSD (ours) | 73.89 (-1.11) | 69.17 (+4.17) | 30.56 (+2.23) | 57.87 (+1.76) |
| Qwen3-1.7B | ||||
| Base model | 50.00 | 37.22 | 14.44 | 33.89 |
| Vanilla OPSD | 44.17 | 35.56 | 13.33 | 31.02 |
| SFT | 37.50 | 29.17 | 12.22 | 26.30 |
| GRPO | 48.33 | 38.61 | 15.00 | 33.98 |
| β-OPSD (ours) | 53.33 (+9.16) | 40.83 (+5.27) | 16.11 (+2.78) | 36.76 (+5.74) |
Gains in parentheses are absolute differences relative to Vanilla OPSD at the same model scale. β-OPSD achieves the strongest average result at all three scales, although individual benchmark results can decrease at larger scales.
Average performance improves over Vanilla OPSD by 1.66 points at 8B, 1.76 points at 4B, and 5.74 points at 1.7B.
On Qwen3-1.7B, β-OPSD improves AIME 2024 by 9.16 points and improves all three benchmarks.
At larger scales, β-OPSD has the strongest overall average but does not win every individual benchmark.
Controlled ablations isolate the contribution of the interpolant target and return-to-go credit assignment.
| Method | A24 | A25 | H25 |
|---|---|---|---|
| Teacher + RTG | 47.30 | 35.53 | 14.44 |
| β-OPSD target | 53.33 | 40.83 | 16.11 |
With identical return-to-go credit, the interpolant improves AIME 2024, AIME 2025, and HMMT 2025 by 6.03, 5.30, and 1.67 points.
| Method | A24 | A25 | H25 |
|---|---|---|---|
| Local token gradient | 49.44 | 32.78 | 13.06 |
| Return-to-go | 50.56 | 38.33 | 16.67 |
With the same interpolant target, return-to-go improves the benchmarks by 1.12, 5.55, and 3.61 points.
A dynamic stop-gradient student endpoint paired with a fixed privileged teacher performs best across all three benchmarks.
The 0.5 → 0.8 teacher-weight schedule gives the strongest overall result among the schedules tested, although other schedules can lead on individual benchmarks.
@misc{xu2026betaopsd,
title = {{$\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation}},
author = {Xu, Jiawei and Liu, Minghui and Zhang, Juzheng and Goldstein, Tom and Huang, Furong},
year = {2026},
eprint = {2607.28582},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2607.28582}
}