Issue #5
Sent 2026-06-08
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
On AIME24, majority voting across distinct LoRA variants of Qwen3-30B raised accuracy from 0.3644 at k=1 to 0.4867 at k=198, versus repeated sampling from one model which saturated at 0.4378 (k=24). Caveat: this is one controlled setting on one task, the variants differ only by data permutation/masking, and the authors explicitly call it an empirical law in one regime, not a general result.
arXiv:2606.02437 · 237▲ · cs.LG, cs.CL
- LLMs
- Efficiency
- RL
A position paper from Mind Lab arguing that LoRA adapters should be treated as persistent per-user state on top of shared foundation models, organized around three axes: scaling up the base model (trillion-parameter MoE LoRA RL), scaling down the adapter (rank studies, a minor-subspace initialization called OLoRA-tail), and scaling out to many adapters (memory capacity laws, user simulation, majority voting). It bundles a range of experiments plus a serving/lifecycle system (MinT) that manages adapter revisions through storage, CPU cache, and GPU batch tiers.
Full breakdown →Cosmos 3: Omnimodal World Models for Physical AI
Cosmos3-Nano-Policy-DROID reached a 39.7% average success rate across 120 RoboLab tasks (specific-instruction setting, 10 rollouts each) versus 28.1% for the closest baseline (pi-0.5), and ranked first on the RoboArena real-world leaderboard at submission time. Note the robot policy was trained and evaluated only on the DROID platform, so cross-embodiment generality is not directly demonstrated by this number.
arXiv:2606.02800 · 139▲ · cs.CV, cs.AI, cs.LG, cs.MM, cs.RO
- Multimodal
- Robotics
- Vision
Cosmos 3 is a family of omnimodal models (4B, 16B, 64B params) that jointly process and generate language, image, video, audio, and action within a single Mixture-of-Transformers architecture. It uses a dual-tower design where one parameter set handles autoregressive reasoning (next-token prediction) and a separate set handles diffusion-based generation of pixels/audio/actions, with the two towers sharing a joint attention operation and both initialized from a pretrained Qwen3-VL.
Full breakdown →Audio Interaction Model
On MMAU under audio instructions the model scores 58.15 average, slightly above its Qwen2.5-Omni-3B base (57.81) and competitive with 7B systems, while ASR regresses modestly (LibriSpeech clean WER 3.17 vs base 2.87). The proactive-response claim rests on their own new Proactive-Sound-Bench (644 hand-designed events, 61.2 Single / 62.8 Multi tier), where they beat larger baselines but on a benchmark they authored themselves.
arXiv:2606.05121 · 121▲ · cs.SD, cs.AI, cs.CL, cs.MM, eess.AS
- Multimodal
- Inference
- Training
The paper builds Audio-Interaction, a streaming audio language model (fine-tuned from Qwen2.5-Omni-3B) that consumes audio in 400ms chunks and predicts a per-chunk control token (<silent> or <response>) to decide whether to speak or keep listening. It is trained via a framework called SoundFlow using a 2.6M-item synthetic streaming corpus (StreamAudio-2M) and served with a FIFO-scheduled asynchronous inference scheme that decouples encoding from decoding.
Full breakdown →Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
Up to 14.1% improvement in overall MTEB average score (ECHO baseline on Qwen2.5-0.5B, from 46.03 to 52.55) at tau=2 (half dimensions). Gains are consistent but more modest on larger models (roughly 3-8% on Llama-3.1-8B and Mistral-7B), and the evaluation used a subset of retrieval tasks (8 datasets) due to compute limits.
arXiv:2606.07502 · 99▲ · cs.CL, cs.IR
- LLMs
- Efficiency
- Inference
The paper introduces EmbedFilter, a linear post-processing transformation that improves text embeddings extracted zero-shot from LLMs. It works by taking the SVD of the LLM's unembedding matrix and projecting embeddings onto only the mid-range ('bulk') singular vectors, discarding the 'edge spectrum' (largest and smallest singular values) that the authors show encodes high-frequency, semantically empty tokens.
Full breakdown →Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution
On the evolution track cross-repo split, Code2LoRA-Evo hits 60.3% exact match, +5.2 pp over a single shared LoRA (55.1%) and matching/exceeding the per-repo LoRA upper bound on the in-repo split without per-repo training. Caveat: the out-of-distribution holdout gain shrinks to ~1.8 pp EM, and the authors flag that OOD scores are inflated because those repos have shorter assertion targets (median 7 vs 12-13 chars).
arXiv:2606.06492 · 95▲ · cs.SE, cs.AI, cs.CL
- LLMs
- Efficiency
- Inference
Code2LoRA is a hypernetwork that reads a code repository (compressed into embeddings by a frozen Qwen3-Embedding model) and generates a repository-specific LoRA adapter for a frozen code LLM in a single forward pass, so repo knowledge lives in parameters instead of the prompt. It has two variants: Static, which maps one repository snapshot to an adapter, and Evo, which uses a GRU to update a hidden state per commit diff so the adapter tracks an evolving codebase.
Full breakdown →KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
On AIME24 with Qwen3-4B, KVarN scored 60.0% accuracy at 2.3 bits/element versus 55.5% for KIVI and 56.7% for QuaRot at the same bit budget, close to the 61.1% FP16 baseline. Gains elsewhere are smaller (for example MATH500 79.2% vs 78.9% QuaRot), and results cover three model families (Qwen3-4B, Llama-3.1-8B, Phi-4-14B) averaged over three runs.
arXiv:2606.03458 · 67▲ · cs.LG
- LLMs
- Efficiency
- Inference
KVarN is a calibration-free KV cache quantizer that combines a Hadamard rotation in the channel dimension with an iterative Sinkhorn-style variance normalization across both the token and channel axes of the K and V matrices before round-to-nearest quantization. It targets the specific error accumulation that happens when the cache is quantized on-the-fly during long autoregressive decoding, rather than the parallel prefill setting most prior methods were tuned for.
Full breakdown →Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
0.730 average curated recall across eight retrieval benchmarks (web, finance, patents, multi-hop QA), which is +11.4 points over the next best open agent (Tongyi DeepResearch 30B); it beats GPT-5.4, Sonnet-4.6, and Kimi-K2.5 under the same protocol, though Opus-4.6 still leads on average. Caveat: results are point estimates with no confidence intervals, and frontier models were run under a different (Context-1) harness rather than Harness-1's own.
arXiv:2606.02373 · 59▲ · cs.AI, cs.CL, cs.IR
- Agents
- RL
- LLMs
Harness-1 is a 20B search agent (built on gpt-oss-20b) trained with reinforcement learning inside a stateful harness that moves search bookkeeping out of the model and into the environment. The environment maintains a candidate pool, an importance-tagged curated set, an evidence graph, verification records, and deduplicated/compressed observations, while the policy only decides what to search, which documents to keep, what to verify, and when to stop.
Full breakdown →K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
On the verified subset the best model, GPT-5.5, reached 45.67 percent versus its reported 84.4 percent on English BrowseComp, while Korean open-weight models scored 0.00 to 10.33 percent; on the adversarially filtered synthetic split the top score was 26.00 percent. Caveat: results come from a single browsing harness, one search backend, and single-run pass@1, and the synthetic split is separately reported because failure-based filtering (against gpt-5.4-mini and gemini-3-flash-preview) is baked into its construction.
arXiv:2606.02404 · 59▲ · cs.CL
- Agents
- LLMs
- Data
K-BrowseComp is a web-browsing agent benchmark of 400 problems grounded in Korean web contexts: a 300-question human-verified subset built by native Korean annotators, plus a 100-question synthetic split generated by a browsing agent (Claude Code) that constructs questions backwards from seed pages and filters them against a taxonomy of failure modes. Each question has a single stable answer requiring multi-hop or parallel-constraint retrieval across multiple Korean websites, and models are evaluated pass@1 under a fixed 10-call search budget.
Full breakdown →Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.