Efficiency

Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang

The paper introduces AgenticSTS, an LLM agent for the deck-building game Slay the Spire 2 that replaces the usual growing chat transcript with a bounded memory contract: every decision prompt is rebuilt fresh from five typed slots (protocol, state schema, game rules, episodic summaries, and triggered strategic skills) via typed retrieval, so no raw cross-decision history is appended. It also ships a testbed of 298 tagged trajectories, frozen memory snapshots, and analysis scripts so individual memory layers can be ablated in isolation.

Sergey Ioffe, Christian Szegedy

This paper introduces Batch Normalization, a layer inserted before each nonlinearity that normalizes activations to zero mean and unit variance using per-mini-batch statistics, then applies a learned scale and shift. The normalization is part of the network architecture so gradients backpropagate through it, and at inference time it uses population statistics computed from moving averages.

Hao Zhang, Yiming Hu, Yong Wang, Mingqiao Mo, Xin Xiao, Xiangxiang Chu

BlockPilot is a lightweight predictor that picks a per-sample block size for diffusion-based speculative decoding, replacing the fixed block size inherited from training. It uses the target model's last-token predictive distribution after prefilling as input to a two-layer MLP classifier that selects the block size from a small local range around the training value.

Min Zhao, Hongzhou Zhu, Kaiwen Zheng, Zihan Zhou, Bokai Yan, Xinyuan Li, Xiao Yang, Chongxuan Li, Jun Zhu

The paper distills bidirectional video diffusion models into few-step autoregressive students for real-time interactive video generation, replacing the expensive causal ODE initialization step of prior work (Causal Forcing) with causal consistency distillation (causal CD). Causal CD gets its training signal from a single online teacher ODE step between adjacent timesteps on real videos, avoiding the need to precompute and store full PF-ODE trajectories.

Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie

Code2LoRA is a hypernetwork that reads a code repository (compressed into embeddings by a frozen Qwen3-Embedding model) and generates a repository-specific LoRA adapter for a frozen code LLM in a single forward pass, so repo knowledge lives in parameters instead of the prompt. It has two variants: Static, which maps one repository snapshot to an adapter, and Evo, which uses a GRU to update a hidden state per commit diff so the adapter tracks an evolving codebase.

DreamX Team, Yancheng Bai, Rui Chen, Xiangxiang Chu, Rujing Dang, Hao Dou, Bingjie Gao, Qiwen Gu, Siyu Hong, Jiachen Lei, Geng Li, Jifan Li, Ruimin Lin, Qingfeng Shi, Bingze Song, Lei Sun, Jing Tang, Ruitian Tian, Jun Wang, Jiahong Wu, Pengfei Zhang, Shen Zhang, Jiashu Zhu

DreamX-World 1.0 is an interactive text/image-to-video world model built by fine-tuning Wan2.2 to support camera navigation, revisiting earlier scenes, and prompted multi-object events across photorealistic, game, and stylized domains. It combines a camera-conditioning method called E-PRoPE (projective positional encoding applied to spatially downsampled tokens), geometry-based memory retrieval, DMD distillation into a few-step autoregressive generator, and RL alignment, with serving optimizations to hit real-time streaming.

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

This is the original word2vec paper. It introduces two log-linear model architectures, Continuous Bag-of-Words (CBOW) and Skip-gram, that learn dense word vectors by removing the expensive non-linear hidden layer used in prior neural language models and training on billions of words.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

FlashAttention is an exact attention algorithm that restructures the computation to avoid writing the large N x N attention matrix to GPU high bandwidth memory (HBM). It uses tiling to compute softmax block by block in fast on-chip SRAM and recomputes the attention matrix during the backward pass instead of storing it, all fused into a single CUDA kernel.

Fangfu Liu, Kai He, Tianchang Shen, Tianshi Cao, Sanja Fidler, Yueqi Duan, Jun Gao, Igor Gilitschenski, Zian Wang, Xuanchi Ren

Gamma-World is a video world model that generates synchronized, action-controllable video streams for multiple agents (game players or robot arms) sharing one environment. It introduces two components: Simplex Rotary Agent Encoding, a parameter-free extension of 3D RoPE that assigns each agent a rotary phase at the vertex of a regular simplex so agents stay distinct but permutation-symmetric, and Sparse Hub Attention, where learnable hub tokens mediate cross-agent communication instead of dense all-to-all attention.

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori

HRM-Text replaces the standard decoder-only Transformer with a Hierarchical Recurrent Model that splits computation into a slow strategic (H) module and a fast execution (L) module, stabilized for language via two tricks the authors call MagicNorm (a hybrid Pre/Post normalization scheme) and a warmup schedule for truncated backpropagation through time. Instead of pretraining on raw web text, it trains a 1B-parameter model from scratch only on instruction-response pairs, computing loss just on the response (a task-completion objective) with a PrefixLM attention mask that lets instruction tokens attend bidirectionally.

Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

KVarN is a calibration-free KV cache quantizer that combines a Hadamard rotation in the channel dimension with an iterative Sinkhorn-style variance normalization across both the token and channel axes of the K and V matrices before round-to-nearest quantization. It targets the specific error accumulation that happens when the cache is quantized on-the-fly during long autoregressive decoding, rather than the parallel prefill setting most prior methods were tuned for.

Kwai Keye Team, Bin Wen, Changyi Liu, Chengru Song, Chongling Rao, Guowang Zhang, Han Li, Haonan Fan, Hengrui Ju, Jiankang Chen, Jiapeng Chen, Jiawei Yuan, Kaixuan Yang, Kaiyu Jiang, Kun Gai, Lingzhi Zhou, Na Nie, Sen Na, Tianke Zhang, Tingting Gao, Xuanyu Zheng, Yulong Chen, Fan Yang, Haixuan Gao, Lele Yang, Mingqiao Liu, Muxi Diao, Qi Zhang, Qile Su, Wei Chen, Wentao Hong, Xingyu Lu, Yancheng Long, Yankai Yang, Yingxin Li, Yiyang Fan, Yu Xia, Yuzhe Chen, Ziliang Lai, Chuan Yi, Haonan Jia, Tianming Liang, Weixin Xu, Xiaoxiao Ma, Yang Tian, Yufei Han, Feng Han, Hang Li, Jing Wang, Jinghui Jia, Junmin Chen, Junyu Shi, Ruilin Zhang

A technical report on Kwai Keye-VL-2.0-30B-A3B, an open-source Mixture-of-Experts multimodal model (30B total, 3B active) that adapts DeepSeek Sparse Attention (DSA) to a GQA-based backbone so it can process 256K-token contexts, aimed at hour-long video understanding. It also describes a post-training recipe combining Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) with Context-RL and Video-RL to add agent, code, and search skills without wiping out base reasoning.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample

LLaMA is a family of transformer language models (7B to 65B parameters) trained only on publicly available datasets, using architectural changes like pre-normalization with RMSNorm, SwiGLU activations, and rotary positional embeddings. The models are trained on far more tokens than Chinchilla scaling laws recommend (up to 1.4T tokens), trading extra training compute for cheaper inference.

Jian Yang, Shawn Guo, Wei Zhang, Tianyu Zheng, Yaxin Du, Haau-Sing Li, Jiajun Wu, Yue Song, Yan Xing, Qingsong Cai, Zelong Huang, Chuan Hao, Ran Tao, Xianglong Liu, Wayne Xin Zhao, Mingjie Tang, Weifeng Lv, Ming Zhou, Bryan Dai

The paper studies how many times to loop a Parallel Loop Transformer (PLT), a looped-Transformer variant that reuses shared blocks with cross-loop position offsets (CLP) and shared-KV gated sliding-window attention to keep latency and KV-cache memory flat regardless of loop count. It trains LoopCoder-v2, a 7B coder, from scratch on 18T tokens at loop counts of 1, 2, 3, and 4, and uses per-loop diagnostics (hidden-state dynamics, attention routing, output-distribution shift) to explain why performance peaks at two loops.

Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang, Jinrui Zeng, Bowen Cao, Lingwei Meng, Mocheng Li, Zezhong Wang, Haonan Yin, Naifu Xue, Minyu Chen, Cenyuan Zhang, Zefan Zhang, Hao Wei, Jiawei Zhou, Haoran Xu, Hao Yang, Ronglai Zuo, Tongda Xu, Yonghao Li, Jian Chen, Hebin Wang, Zeyu Gao, Yang Li, Wei Zhao, Qimin Zhong, Siqi Liu, Yumeng Zhang, Leyan Cui, Zhangyu Wang, Wai Lam

The paper applies looped transformers (a shared transformer block iterated repeatedly across depth) to world models, calling the result LoopWM. It adds a spectrally constrained state-retention matrix to keep latent updates bounded, Poisson-sampled variable loop depth during training, an early-exit gate for adaptive inference, and a deferred-decoding mode that skips intermediate observation reconstruction during multi-step rollouts.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

LoRA (Low-Rank Adaptation) freezes a pre-trained model's weights and injects trainable rank-decomposition matrices (a pair of small matrices B and A whose product approximates the weight update) into each Transformer layer, so only those small matrices are trained. In their Transformer experiments the authors apply it mostly to the query and value attention projection matrices.

Mind Lab, :, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Zhihui Li, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Changhai Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang

MinT is a managed infrastructure system for LoRA post-training and serving that keeps expensive base models resident on GPUs and moves only exported LoRA adapter revisions (not merged full checkpoints) through training, export, serving, and rollback. It provides a Tinker-compatible service API on top of vLLM samplers and PEFT/Megatron trainers, with a policy-record abstraction that tracks each adapter's base version, rank, checkpoint, and cache tier.

Kangsheng Duan, Ziyang Xu, Wenyu Liu, Xiaohu Ruan, Xiaoxin Chen, Xinggang Wang

Moebius is a 0.22B-parameter image inpainting model built on a latent diffusion U-Net whose transformer blocks are replaced with a Local-Lambda Mix Interaction (LLambdaMI) block that summarizes spatial context and semantic priors into fixed-size linear matrices instead of quadratic attention. It is trained with an adaptive multi-granularity knowledge distillation scheme that aligns the small student to the larger PixelHacker teacher entirely in latent space using gradient-norm-balanced losses.

Disen Lan, Jianbin Zheng, Yuxi Ren, Xin Xia, Xuanda Wang, Xuefeng Xiao, Xipeng Qiu, Yu Cheng

FlashMorph is a method for choosing which layers of a pretrained Transformer to keep as full attention when converting the rest to linear attention (a hybrid model). Instead of fixed placement rules or scoring layers one at a time, it attaches a frozen linear-attention branch to each layer, adds a learnable scalar gate per layer, and jointly optimizes all gates on synthetic passkey-retrieval data with a regularizer that pushes layers toward linear attention, then keeps the top-K layers by gate value.

Mind Lab, :, Vin Bo, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Wenhao Li, Zhihui Li, Allen Lin, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Shiyang Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Adrian Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang

A position paper from Mind Lab arguing that LoRA adapters should be treated as persistent per-user state on top of shared foundation models, organized around three axes: scaling up the base model (trillion-parameter MoE LoRA RL), scaling down the adapter (rank studies, a minor-subspace initialization called OLoRA-tail), and scaling out to many adapters (memory capacity laws, user simulation, majority voting). It bundles a range of experiments plus a serving/lifecycle system (MinT) that manages adapter revisions through storage, CPU cache, and GPU batch tiers.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean

The paper introduces the Sparsely-Gated Mixture-of-Experts (MoE) layer, a neural network component containing up to thousands of feed-forward expert sub-networks where a trainable gating network selects only a few experts to run per input token. It uses noisy top-k gating plus auxiliary load-balancing losses, and applies the layer between stacked LSTM layers for language modeling and machine translation.

Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, Yunhai Tong

PerceptionDLM is a multimodal diffusion language model that captions multiple image regions at once inside a single denoising pass, instead of the region-by-region autoregressive decoding used by prior models. It combines a SigLIP-2 vision encoder and an 8B LLaDA diffusion backbone with region prompting (learnable per-region embeddings), RoI-aligned feature replay, and a structured attention mask that isolates each region's tokens while sharing global context.

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng

The paper introduces Program-as-Weights (PAW), a system that compiles a natural-language description of a fuzzy task (things like log alerting, JSON repair, or intent classification that resist clean rule-based code) into a small neural artifact. A 4B compiler reads the spec plus an auto-generated pseudo-program of paraphrase-and-examples, then emits a per-function LoRA adapter that gets hot-attached to a frozen 0.6B interpreter model, which runs the task locally.

Bin Wu, Mengqi Huang, Shaojin Wu, Weinan Jia, Yuxin Wang, Zhendong Mao, Yongdong Zhang

Stream-R1 modifies distribution matching distillation (DMD) for streaming autoregressive video generation by reweighting the distillation loss with a pretrained video reward model at two levels: a per-rollout scalar weight (exponential of the reward score) and a per-pixel spatiotemporal weight derived from backpropagating the reward model's gradient. It replaces the uniform weighting that treats every rollout, frame, and pixel as equally reliable supervision with reward-guided weighting that concentrates optimization on regions where refinement helps most.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre

The paper studies how to split a fixed training compute budget between model size and number of training tokens, fitting three empirical estimators (minimum-over-training-curves, IsoFLOP profiles, and a parametric loss function) across over 400 models trained from 70M to 16B parameters. It finds model size and training tokens should scale in roughly equal proportion (doubling one means doubling the other), then validates this by training Chinchilla, a 70B model on 1.4T tokens, at the same compute as the 280B Gopher.

Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, Mengyang Feng, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuzheng Wang, Zoubin Bi

Wan-Streamer is a single Transformer that handles text, audio, and video as both input and output, using block-causal attention plus conditional flow matching to generate synchronized speech and video responses in a streaming, full-duplex manner. It replaces the usual cascade of separate VAD, ASR, LLM, TTS, and avatar-rendering modules with one end-to-end model built around causal VAEs, causal encoders/decoders, and a thinker-performer split at inference time.

Songhao Wu, Zhongxin Chen, Yuxuan Liu, Heng Cui, Cong Li, Rui Yan

The paper introduces EmbedFilter, a linear post-processing transformation that improves text embeddings extracted zero-shot from LLMs. It works by taking the SVD of the LLM's unembedding matrix and projecting embeddings onto only the mid-range ('bulk') singular vectors, discarding the 'edge spectrum' (largest and smallest singular values) that the authors show encodes high-frequency, semantically empty tokens.