MinT: Managed Infrastructure for Training and Serving Millions of LLMs
Mind Lab, :, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Zhihui Li, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Changhai Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang
arXiv:2605.13779 · 223▲ · cs.LG, cs.AI, cs.DC
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
MinT is a managed infrastructure system for LoRA post-training and serving that keeps expensive base models resident on GPUs and moves only exported LoRA adapter revisions (not merged full checkpoints) through training, export, serving, and rollback. It provides a Tinker-compatible service API on top of vLLM samplers and PEFT/Megatron trainers, with a policy-record abstraction that tracks each adapter's base version, rank, checkpoint, and cache tier.
Why it matters
If you run many fine-tuned model variants (tenant-specific, product branches, rollback points) over a few base models, this avoids materializing and shipping a full checkpoint per variant. The training-to-serving handoff moves a few hundred MB instead of tens of GB, and one resident base can time-slice across many policies, so you fit more concurrent training and serving into the same GPU allocation.
Practical takeaway
If you serve multiple LoRA adapters over a shared base, treat cold adapter loading as an explicit scheduled stage: use readiness gates so new adapters are exposed only after prewarm, and pack fragmented MoE adapter tensors to cut load-object fanout. Watch for the caveat that a huge addressable catalog does not mean simultaneous residency, one engine caps around 64 distinct adapters per decoding batch and a few hundred cached in CPU.
Key result
Adapter-only handoff cut the measured training-to-serving handoff step by 18.3x on a 4B dense model (252 MiB adapter and 0.036s load vs 8.06 GB checkpoint and 71.8s materialize) and 2.85x on a 30B MoE model. The million-adapter claim is a built-and-audited catalog artifact (1,000,000 adapters, 256 sampled read-OK), not a claim of simultaneous GPU residency, and warm serving held 100% of a 5s TTFT SLO only through 2 requests/sec (dropping to 72% at 4 rps) on a single 64-adapter TP=4 engine.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.