Flow-OPD: On-Policy Distillation for Flow Matching Models
Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, Feng Zhao
arXiv:2605.08063 · 102▲ · cs.CV, cs.AI
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
Flow-OPD is a post-training method for text-to-image flow matching models that replaces multi-reward reinforcement learning with on-policy distillation from multiple single-task teacher models. It trains domain-expert teachers via single-reward GRPO, then distills them into one student by routing prompts to the relevant teacher and matching velocity fields, showing that the reverse-KL divergence between student and teacher policies reduces analytically to a weighted MSE loss on the vector fields.
Why it matters
Practitioners trying to align a single image model across conflicting objectives (text rendering, compositional accuracy, aesthetics) hit a seesaw effect where optimizing one reward degrades others. This approach gives dense per-step supervision instead of sparse scalar rewards, and because the KL reduces to a closed-form MSE the training avoids the high-variance policy gradient machinery (log-prob tracking, PPO clipping), which is simpler to implement than Flow-GRPO.
Practical takeaway
If you are fine-tuning a flow matching model like SD 3.5 and mixing reward functions causes catastrophic forgetting on non-target metrics, you can instead train separate specialist teachers and distill them via velocity-field matching with hard task routing. Note the constraint: teacher and student must share the same architecture, and student quality is capped by the teachers.
Key result
On SD-3.5-Medium, Flow-OPD (Merge variant) raised GenEval from 0.63 to 0.93 and OCR accuracy from 0.59 to 0.94, versus GRPO-Mix which reached only 0.73 GenEval and 0.83 OCR, roughly a 10-point average gain over vanilla multi-reward GRPO across four benchmarks. Caveat: baselines are the authors' own Flow-GRPO reruns, and the reported teacher-surpassing effect is supported mainly by qualitative edge cases rather than aggregate metrics.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.