Stay on-policy
Generate fresh trajectories after every update. Train on the residual-conditioned states the current model actually visits.
ORLOn-Policy Residual Learningfor Diffusion Language Models
1Together AI 2University of Illinois Urbana-Champaign 3University of California, Berkeley 4University of California, Santa Cruz 5Stanford University 6The University of Texas at Austin
arXiv preprint
REAL-TIME REPLAY · BATCH 1
Coding problems back to back, in real time
Recorded wall-clock decoding of HumanEval problems with thinking on: each lane answers the same problems one after another.
0.0 s1× real time
Loading the recorded runs…
6.2×
Higher tokens per forward
Up to · QwenDLM 4B~6.9×
End-to-end throughput
QwenDLM + dlmrt · single H1004.2M
Self-generated training tokens
LLaDA-2.1-mini post-training5.2×
Faster training steps
Up to · system co-designAbstract
Residual context carries predictive distributions across denoising steps. But an effective learning recipe must account for the fact that these predictions—and the trajectories they induce—change as the model learns.
ORL closes this loop. It trains on fresh, verified trajectories from the current model, replays the exact residual inputs encountered during generation, and uses adjacent entropy contraction to encourage more decisive refinement. A co-designed training system makes this practical at scale.
Across QwenDLM, LLaDA-2.1-mini, and DiffusionGemma, ORL improves parallel decoding efficiency. With QwenDLM, it delivers up to 6.2× higher TPF and approximately 5× throughput at comparable or better quality (about 6.9× together with our dlmrt engine).
Read the paper’s full abstract
Residual context is emerging as a key ingredient for unlocking the speed–quality Pareto frontier of diffusion language models (DLMs) by carrying forward predictive distributions across denoising steps. Yet turning this seemingly simple mechanism of computation recirculation into an effective and efficient learning paradigm is far from simple: there is no clear recipe for how residual context should be learned. We introduce ORL, a framework for on-policy residual learning in DLMs. Algorithmically, we identify that residual learning is inherently on-policy. ORL learns directly from these model-induced residual trajectories on the fly and introduces an entropy-contraction objective that encourages sharpening predictions as the model revisits its own intermediate predictions. System-wise, exact residual supervision requires capturing, retaining, and repeatedly processing many trajectory-dependent intermediate states. ORL co-designs the learning algorithm with asynchronous residual capture and transport, specialized block-diffusion attention, and packed compute-balanced training, making on-policy residual learning practical at scale. On a 4B DLM converted from Qwen3-4B, ORL improves tokens per forward (TPF) by up to 6.2× and, with our proposed dlmrt engine, end-to-end throughput by approximately 6.9× at comparable or better quality. In single-request serving on one H100, this model achieves a stronger speed–quality Pareto frontier than autoregressive models accelerated with EAGLE3 and DFlash speculative decoding under both strict and relaxed verification, running 1.74× faster than DFlash at matched accuracy. ORL also transfers to existing DLMs: on LLaDA-2.1-mini, 36 H100-hours and 4.2M self-generated tokens, about 10× less data than prior LLaDA recipes, yield up to 1.5× higher TPF and up to 3× higher throughput at comparable quality, and on DiffusionGemma, ORL raises TPF by 1.1–1.2×. With dlmrt and NVFP4 quantization, ORL models decode a single request at up to 5,046 tokens/s on one B200, 7,708 tokens/s on eight B200s, and 2,788 tokens/s on a consumer RTX 5090, with comparable quality.
Why learn residuals on-policy?
Residuals change the states a model visits. Learning from them requires fresh trajectories, an explicit objective for refinement speed, and a system that can replay the exact inputs.
On-Policy Residual Learning
Reward refinement
Supervise intermediate predictions with verified outputs. Apply an entropy-contraction loss only when progress falls short.
Keep the loop stable
Control residual-pathway updates, gate supervision by uncertainty, and anchor confident predictions with capped reverse KL.
The training loop
ILLUSTRATIVE EXAMPLE · “WHAT IS 6 × 7?”
1. Generate & capture
The current model generates a response to the prompt. At each denoising step, the engine captures the token canvas and the exact pre-adapter residual embedding z consumed by that step, together with the rollout statistics needed for training.
CAPTURED STATE · SCHEMATIC TOKENIZATION
Theanswer[MASK][MASK]
state = (canvas, stop-gradient(z))Original Figure 3, p. 5, of the arXiv preprint. Numbered markers and example added for explanation; tokenization and response are illustrative.
Learning how fast to refine
AEC asks the next state to reduce normalized entropy by a fraction ρ. Its hinge loss becomes zero once that target is met. Each state’s entropy is averaged over its own unresolved positions.
ℒAEC = [max(0, Hnext − (1 − ρ) Hsource)]²
Equation 2 · The source entropy is detached from gradients.INTERACTIVE · ILLUSTRATIVE VALUESρ = 0.25
Source stateH 0.40
Next stateH 0.36
Unresolved, darker = more uncertain Resolved, left out of H
Contraction shortfallℒ = (0.36 − 0.30)² = 0.0036
Drag the next state along the entropy axis.
Where the Speed Comes From
Wall-clock speed is tokens per forward pass times forward passes per second. Fix the number of forward passes and compare how far each decoder gets on one problem: ORL refines a whole block per pass.
INTERACTIVE · RECORDED GENERATIONS
Same question, same forward-pass budget
One GSM8K problem, decoded greedily three ways with 4B models. Drag to set how many forward passes each decoder has run.
PROMPTLoading the recorded generations…
0forward passes
Experimental Results
Three key results: generation quality and parallelism, end-to-end throughput, and data-efficient post-training.
Generation quality and parallelism
| Model / training | GSM8K | MATH-500 | HumanEval | MBPP | ||||
|---|---|---|---|---|---|---|---|---|
| Acc. (%) | TPF | Acc. (%) | TPF | pass@1 (%) | TPF | pass@1 (%) | TPF | |
| Qwen-based models and other DLM baselines | ||||||||
| d3LLM–Dream | 81.4 | 4.9 | 38.2 | 3.9 | 57.1 | 3.2 | 55.6 | 3.0 |
| Fast-dLLM-v2-7B | 77.5 | 2.2 | 48.7 | 2.6 | 61.7 | 2.6 | 50.1 | 2.0 |
| RCD–SDAR-4B-b64 | 87.0 | — | 68.4 | — | — | — | — | — |
| TraDo-4B (TraceRL, dynamic) | 90.3 | — | 71.8 | — | — | — | — | — |
| SDAR-8B-b32 | 88.9 | 2.9 | 63.6 | 4.8 | 73.5 | 2.4 | 58.0 | 2.4 |
| QwenDLM (4B) | 90.4 | 2.4 | 88.0 | 2.1 | 82.9 | 1.8 | 79.2 | 2.1 |
| QwenDLM + ORL | 92.4 | 5.9 | 92.2 | 5.8 | 89.6 | 11.1 | 79.4 | 8.0 |
| LLaDA-based models | ||||||||
| Fast-dLLM–LLaDA | 74.7 | 2.8 | 30.8 | 2.0 | 37.8 | 2.6 | 38.6 | 2.1 |
| dParallel–LLaDA | 72.6 | 5.1 | 30.2 | 3.2 | 39.0 | 4.9 | 40.0 | 2.4 |
| D2F–LLaDA | 73.2 | 2.9 | 28.7 | 2.4 | 36.6 | 2.7 | 38.0 | 1.9 |
| d3LLM–LLaDA | 73.1 | 9.1 | 30.4 | 5.7 | 39.6 | 6.0 | 40.6 | 4.2 |
| dOPSD–LLaDA-8B-Instruct | 72.87 | — | 36.00 | — | 39.63 | — | 39.54 | — |
| dUltra-coding | 81.5 | 8.4 | 35.6 | 6.7 | 35.9 | 7.2 | 37.0 | 7.3 |
| DMax-Math | 92.1 | 5.5 | 75.4 | 5.9 | — | — | — | — |
| LLaDA-2.1-mini | 91.9 | 5.6 | 84.6 | 8.2 | 86.6 | 11.3 | 77.5 | 7.7 |
| LLaDA-2.1-mini + ORL | 92.0 | 8.5 | 87.0 | 10.2 | 89.0 | 13.1 | 77.3 | 10.3 |
| DiffusionGemma-based models | ||||||||
| DiffusionGemma | 95.8 | 26.0 | 92.3 | 23.7 | 90.9 | 24.5 | 87.7 | 23.1 |
| DiffusionGemma + ORL | 96.0 | 30.0 | 92.3 | 27.4 | 91.9 | 27.9 | 87.4 | 28.7 |
Blue rows highlight ORL; underlined model names identify its starting points. Base and ORL use their selected family-specific decoders, so comparisons include both model and decoder. Baseline results are quoted from published papers; MATH entries from the d3LLM evaluation use the full MATH test set with 4-shot prompting, as noted in the manuscript. — indicates unreported values.
Quality–throughput trade-off
1,298 tokens / sec
QwenDLM + ORL achieves 91.0% average GSM8K / HumanEval accuracy for a single request on one H100.
DFlash · relaxed verification746 tok/s
DFlash · strict verification573 tok/s
The relaxed comparison allows accuracy within 0.5 percentage points of ORL. Strict verification preserves the AR target’s accuracy, which is 2.4 points higher than ORL. QwenDLM + ORL runs in dlmrt, our inference engine; AR and speculative decoding run in vLLM. Native decoders and clean-request throughput; §4.1 and Appendix G.
Completion throughput tokens / second →
AR
240
EAGLE3 strict
353
EAGLE3 relaxed
405
DFlash strict
573
DFlash relaxed
746
QwenDLM + ORL
1,298
Single H100 · single request · §4.1
Relaxed verification trades accuracy for speed.
Data-efficient post-training
On LLaDA-2.1-mini, ORL uses just 4.2M self-generated tokens and 36 H100 GPU-hours of post-training—about 10× less additional data than the roughly 40M tokens reported for d3LLM-LLaDA and dParallel.
Original Figure 7 from the arXiv preprint. The data comparison concerns additional training, not total pretraining budgets.
Efficient Exact Residual Training
5.2×training-step speedup
at 16 rollouts per prompt
at 16 rollouts per prompt
Asynchronous state capture
Snapshot states during generation and transfer them off the critical path. Colocated training reuses GPU-resident residual pages via CUDA IPC.
Packed, balanced training
Reuse clean context, pack variable-length states, and distribute work by estimated compute cost while preserving response-normalized losses.
Staircase attention
A specialized Triton kernel enforces block-diffusion visibility and skips invisible tiles without allocating a dense attention mask.
BibTeX
Citation for the arXiv preprint.
BibTeX · arXiv preprint
@misc{yu2026orl,
title = {On-Policy Residual Learning for
Diffusion Language Models},
author = {Yu, Yifan and Jian, Yuqing and Li, Jisen and
Hu, Yuezhou and Zhang, Tianyi and Wu, Xiaoxia and
Zhu, Rui-Jie and Wang, Junxiong and Zhou, Zhongzhu and
Lai, Fan and Zou, James and Xu, Chenfeng},
year = {2026},
eprint = {2610.20687},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.20687}
}