SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
Seokju Cho, Ryo Hachiuma, Abhishek Badki, Hang Su, Byung-Kwan Lee, Chan Hee Song, Sifei Liu, Subhashree Radhakrishnan, Seungryong Kim, Yu-Chiang Frank Wang, Min-Hung Chen
arXiv:2606.13673 · 110▲ · cs.CV, cs.AI
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
SpatialClaw is a training-free agent framework that gives a vision-language model a persistent Python kernel (preloaded with input frames, perception tools like SAM3 and Depth Anything 3, and libraries like NumPy/SciPy) and lets it write and run one code cell per step, inspecting intermediate results before writing the next cell. This 'code as the action interface' approach replaces two prior designs: single-pass code (commit to a full program before seeing any output) and structured JSON/XML tool-calls (fixed API menu).
Why it matters
For anyone building spatial or 4D reasoning on top of VLMs, this shows the action interface matters more than adding more tools. The same configuration works across six different open-source backbones (27B to 397B) from two model families with no per-model or per-benchmark tuning, so you can bolt stronger spatial reasoning onto an already-deployed VLM without fine-tuning or extra training data.
Practical takeaway
If you are wiring perception tools into a VLM agent, expose them as ordinary Python callables in a stateful kernel that persists variables across turns and lets the model see intermediate visualizations, rather than a fixed JSON tool schema. This lets the model compose operations (for example, chaining a depth map with a segmentation mask, or using scipy KDTree for nearest-object queries) that no predefined API anticipated.
Key result
59.9% average accuracy across 20 spatial reasoning benchmarks with a Gemma4-31B backbone, beating the structured tool-call agent SpaceTools by 11.2 points. In a controlled comparison holding the toolset fixed, SpatialClaw scored 59.9% vs 56.7% for structured tool-calls and 55.2% for single-pass code. Caveat: benchmarks over 1,000 samples were subsampled to 1,000, and the authors note remaining errors are dominated by perception quality rather than the interface, so further interface gains are limited.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.