OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data
Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, Pengfei Wan
arXiv:2606.13432 · 113▲ · cs.CV, cs.AI
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
OmniDirector is a video generation framework that clones camera motion from a reference video by rendering the extracted camera poses as a 'camera grid' (a video of grid lines moving through an empty 3D room), then feeding that grid into a Multi-Modal Diffusion Transformer via token concatenation. It also uses a hierarchical prompt expansion agent built on Qwen3-VL to describe camera motion (split into inter-shot and intra-shot descriptions) and fuse it with the reference image and user prompt at inference time.
Why it matters
Prior camera-cloning methods either failed on multi-shot videos with cut transitions or required scarce cross-paired training data (pairs of videos with identical camera moves but different content). Because a camera grid can be auto-generated from any ordinary video, training data scales to internet size (1.8M videos here), and the empty-scene rendering decouples camera motion from scene content, which reduces content leakage from the reference video into the output.
Practical takeaway
If you build video tooling, watch for reference-video camera control that handles multi-shot sequences and shot transitions rather than single continuous shots. The paper also reports that at inference you can swap the rendered camera grid for a raw RGB or Canny edge video and still drive camera motion without retraining, suggesting the representation generalizes across visual proxies.
Key result
On a self-constructed 1,094-sample evaluation set, OmniDirector reached 72.74% translation precision (T-Pre) versus 52.21% for CamCloneMaster, a 39.3% relative gain, plus lower rotation/translation errors and much higher transition accuracy (96.52% temporal vs 38.94% for the only baseline that attempts transitions). Note the eval set and several metrics (leakage, semantic transition) are judged by the authors' own pipeline using Gemini 3.1 Pro, not an external benchmark.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.