On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Mind Lab, :, Vin Bo, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Wenhao Li, Zhihui Li, Allen Lin, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Shiyang Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Adrian Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang
arXiv:2606.02437 · 237▲ · cs.LG, cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
A position paper from Mind Lab arguing that LoRA adapters should be treated as persistent per-user state on top of shared foundation models, organized around three axes: scaling up the base model (trillion-parameter MoE LoRA RL), scaling down the adapter (rank studies, a minor-subspace initialization called OLoRA-tail), and scaling out to many adapters (memory capacity laws, user simulation, majority voting). It bundles a range of experiments plus a serving/lifecycle system (MinT) that manages adapter revisions through storage, CPU cache, and GPU batch tiers.
Why it matters
For practitioners running LLM personalization, the concrete claims are about serving economics: a rank-16 LoRA adapter for Qwen3-30B is 1.7GB versus a 61GB merged full checkpoint, so you can serve many user-specific adapters over one resident base rather than duplicating models. The RL findings (LoRA on a stronger base beating full fine-tuning of a smaller model under equal budget, and routing mismatches in MoE training) are mostly relevant if you are building RL fine-tuning infrastructure.
Practical takeaway
If you fine-tune with LoRA under RL, watch for training-inference mismatch in MoE models where routing diverges between rollout and training, and consider minor-singular-vector initialization (OLoRA-tail) instead of principal-subspace methods like PiSSA, which the paper shows can collapse under RL. For very low ranks, expect high seed-to-seed variance rather than an outright capacity limit.
Key result
On AIME24, majority voting across distinct LoRA variants of Qwen3-30B raised accuracy from 0.3644 at k=1 to 0.4867 at k=198, versus repeated sampling from one model which saturated at 0.4378 (k=24). Caveat: this is one controlled setting on one task, the variants differ only by data permutation/masking, and the authors explicitly call it an empirical law in one regime, not a general result.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.