HRM-Text: Efficient Pretraining Beyond Scaling
Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori
arXiv:2605.20613 · 321▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
HRM-Text replaces the standard decoder-only Transformer with a Hierarchical Recurrent Model that splits computation into a slow strategic (H) module and a fast execution (L) module, stabilized for language via two tricks the authors call MagicNorm (a hybrid Pre/Post normalization scheme) and a warmup schedule for truncated backpropagation through time. Instead of pretraining on raw web text, it trains a 1B-parameter model from scratch only on instruction-response pairs, computing loss just on the response (a task-completion objective) with a PrefixLM attention mask that lets instruction tokens attend bidirectionally.
Why it matters
The headline claim is that you can pretrain a competitive model from scratch for about $1,500 on 16 GPUs in under two days, versus the trillion-token, compute-cluster runs that lock most people out of foundational pretraining. If it holds up, small labs and individuals could train architecture experiments from scratch rather than only fine-tuning existing checkpoints.
Practical takeaway
Watch for the released code and weights, and note the two ideas that are cheap to try on your own recurrent or looped models: response-only loss plus a PrefixLM mask (bidirectional attention over the prompt) gave large gains even on a plain Transformer in their ablation. Be skeptical until the approach is reproduced at larger scale, since results stop at 1B and come from a single checkpoint.
Key result
The 1B model scored 60.7% MMLU, 84.5% GSM8K, and 82.2% DROP while using roughly 96-432x less compute and 100-900x fewer tokens (40B unique) than 2-7B open models like Llama3.2 3B and Qwen. Caveats: it is an existence proof from one checkpoint capped at 1B scale, it skips raw-text base pretraining so the comparison to base models is not apples-to-apples, factual coverage (MMLU) lags larger models, and a contamination test flagged DROP as marginally inflated at n=13.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.