In-Context World Modeling for Robotic Control
Siyin Wang, Junhao Shi, Senyu Fei, Zhaoyang Fu, Li Ji, Jingjing Gong, Xipeng Qiu
arXiv:2606.26025 · 62▲ · cs.RO, cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
In-Context World Modeling (ICWM) is a training and inference method for Vision-Language-Action (VLA) robot policies that prepends a short sequence of self-generated random exploratory movements (recorded as start-image, action, end-image triplets) as context before the task query. The Transformer backbone infers the system configuration (camera viewpoint, kinematics) implicitly from this context in a single forward pass, without any parameter updates or task demonstrations at test time.
Why it matters
Standard VLA policies conditioned only on current observation and instruction degrade sharply when deployed under new camera angles or robot morphologies, currently requiring per-setup fine-tuning. ICWM lets a policy self-calibrate to a novel configuration at deployment using task-agnostic probing, trading a one-time 5 to 6 second probing phase (about 20 movements) for improved robustness, with the context hidden states cacheable via KV cache to keep per-step latency near baseline.
Practical takeaway
If you deploy imitation-learned manipulation policies across varying camera mounts or grippers, you can try having the robot perform random workspace movements first and feed those clips as context rather than fine-tuning per site. Note the capability must be trained in explicitly: a plain behavior-cloning policy given the same context collapses to under 1% success.
Key result
On real UR5e experiments across 6 held-out camera viewpoints, standard multi-view behavior cloning drops from 68% to 17% success under viewpoint shift, while ICWM raises the novel-viewpoint average by 71% relative over that baseline. On LIBERO simulation, ICWM improved OOD viewpoint success by 13.0% over multi-view BC, though absolute OOD numbers remain low (e.g. LIBERO-Long averaging 25.0% vs 19.8%) and one viewpoint (135 degrees) fails for all methods due to occlusion.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.