Orca: The World is in Your Mind
Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, Runze Xiao, Huaihai Lyu, Senwei Xie, Euan Liu, Klara Tian, Tianfeng Long, Yichi Zhang, Zhengliang Cai, Ruike Chen, Jifan Zhao, Ruochuan Shi, Zihan Tang, Jing Lyu, Wenxing Tan, Ningbo Zhang, Yangtao Hu, Yuming Gao, Xiansheng Chen, Junkai Zhao, Congsheng Xu, Boan Zhu, Ziqi Wang, Yupu Feng, Qiongqiong Zhang, Yingli Zhao, Yulong Ao, Shaoxuan Xie, You Liu, Guocai Yao, Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, Yance Jiao, Xinyan Yang, Jiaxing Wei, Xu Liu, Tengfei Pan, Shaokai Nie, Chunlei Men, Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, Xiaolong Zheng, Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan Wang, Pengwei Wang
arXiv:2606.30534 · 235▲ · cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
Orca is an encoder-decoder model that learns a shared 'world latent space' from video and language by predicting the latent representation of adjacent frames (unconscious learning) and event-conditioned future states described by text (conscious learning), plus a standard VQA loss. After pretraining, the backbone (built on a Qwen VLM) is frozen and small task-specific decoders are trained for text, image prediction, and robot action generation to test whether the shared latent transfers.
Why it matters
The interesting practical signal is that action generation improved even though no action-labeled data was used in pretraining, suggesting video-based state-transition pretraining could partly compensate for the chronic scarcity of robot demonstration data. For most practitioners this is still a research artifact, not a deployable tool, since it runs at 0.8B and 4B scale on one-tenth of the collected data and readout modules are trained per task.
Practical takeaway
Watch for the pattern of freezing a pretrained backbone and training only lightweight readout heads (LoRA on Stable Diffusion 3.5 for images, a DiT flow-matching action expert for control) as a cheaper alternative to full multimodal finetuning. The FlagScale training optimizations (FSDP2, chunked cross-entropy, communication prefetching) are reusable regardless of whether the world-model framing pans out.
Key result
On real-robot out-of-distribution tasks, Orca-4B scored 32.4 overall rule-based vs 29.4 for pi-0.5 (a VLA pretrained on large-scale robot data) and 10.5 for a Qwen3.5 baseline, with action experts trained from scratch on only 200 trajectories per task across 5 tasks. Caveat: the eval covers just 5 short tasks and success rates are very low (6% binary success for Orca), so the comparison shows partial-progress advantages more than reliable task completion.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.