Morphing into Hybrid Attention Models
Disen Lan, Jianbin Zheng, Yuxi Ren, Xin Xia, Xuanda Wang, Xuefeng Xiao, Xipeng Qiu, Yu Cheng
arXiv:2606.30562 · 42▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
FlashMorph is a method for choosing which layers of a pretrained Transformer to keep as full attention when converting the rest to linear attention (a hybrid model). Instead of fixed placement rules or scoring layers one at a time, it attaches a frozen linear-attention branch to each layer, adds a learnable scalar gate per layer, and jointly optimizes all gates on synthetic passkey-retrieval data with a regularizer that pushes layers toward linear attention, then keeps the top-K layers by gate value.
Why it matters
For teams converting an existing Transformer to a cheaper long-context architecture, the layer-selection step itself becomes drastically cheaper: 20M tokens and about 2.1 GPU hours versus 15 to 2500+ GPU hours for prior methods. The resulting 3:1 hybrid model runs long contexts faster and with less memory (up to 2.81x prefill speedup at 256K tokens on a 1.7B model) while mostly retaining retrieval and reasoning quality.
Practical takeaway
If you are distilling a Transformer into a hybrid linear-attention model, you can pick the retained full-attention layers by training only a handful of gate scalars rather than running expensive per-layer ablations or architecture search. Watch for this joint-gate approach appearing in conversion pipelines like HALO/HypeNet, though results are shown only on Qwen3 models up to 30B and quality on the hardest long-context tasks still degrades at 128K+.
Key result
On Qwen3-1.7B NIAH retrieval, FlashMorph matches or beats the strongest baseline (HALO) on the hardest NIAH-Single-3 setting (94.4 at 128K, 73.2 at 256K vs HALO's 67.4 and 52.8) while using 20M layer-selection tokens vs HALO's 234M and PostNAS's 50B. Caveat: at the largest 30B-A3B MoE scale its recall-intensive numbers trail or roughly match uniform interleaving, so the advantage is clearest at smaller scales.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.