VDAWorld
Input “A tower of wooden blocks is built on a table. The top block is released...”
What if a vision-language model wrote its own world model — in code?
</> The edit that produced this world
Abstract
World Modelling via VLM-Directed Abstraction and Simulation
Generative video models, a leading approach to world modeling, face fundamental limitations. They often violate physical and logical rules, lack interactivity, and operate as opaque black boxes ill-suited for building structured, queryable worlds. To overcome these challenges, we propose a new paradigm focused on distilling an image caption pair into a tractable, abstract representation optimized for simulation.
We introduce VDAWorld, a framework where a Vision-Language Model (VLM) acts as an intelligent agent to orchestrate this process. The VLM autonomously constructs a grounded (2D or 3D) scene representation optimized for simulation, by selecting from a suite of vision tools, and accordingly chooses a compatible physics simulator (e.g. rigid body, fluid) to act upon it. VDAWorld can then infer latent dynamics from the static scene to predict plausible future states.
Our experiments show that this combination of intelligent abstraction and adaptive simulation results in a versatile world model capable of producing high quality simulations across a wide range of scenarios. We demonstrate several applications of VDAWorld across interactive control and counterfactual generation, and physical and logical reasoning, achieving state-of-the-art results on several benchmarks.
Method
The VLM is the agent, not the simulator.
Given an image and a caption, the model writes three programs: how to build the scene, what acts on it, and how it steps forward. It then runs its own code, looks at the frames, and fixes what it got wrong.
Results
Strong performance on physical estimation.
We evaluate on Physics-IQ, a benchmark of 174 real-world test scenes spanning solid mechanics, fluid dynamics, thermodynamics and magnetism. VDAWorld scores higher than every baseline in both the one-trial and best-of-three settings.
| Model | Solid mech. | Fluid dyn. | Thermo. | Magnetism | Combined |
|---|---|---|---|---|---|
| Best of three trials | |||||
| VDAWorld | 54.9 | 44.7 | 32.6 | 25.5 | 49.7 |
| PhysGen | 29.7 | 24.6 | 29.6 | 23.0 | 28.3 |
| PhysGen3D | 12.2 | 10.2 | 0.4 | 17.2 | 11.7 |
| Wan 2.2 | 47.3 | 49.7 | 46.5 | 20.0 | 46.2 |
| One trial | |||||
| VDAWorld | 42.6 | 34.9 | 17.4 | 22.0 | 37.3 |
| PhysGen | 26.9 | 22.0 | 28.7 | 21.5 | 25.7 |
| PhysGen3D | 6.6 | 7.7 | 0.4 | 17.2 | 7.1 |
| Wan 2.2 | 39.2 | 39.5 | 23.6 | 11.3 | 36.8 |
| VideoPoet † | 35.1 | 24.6 | 29.1 | 44.0 | 32.8 |
| Runway | 27.5 | 27.2 | 20.7 | 17.9 | 27.1 |
| Lumiere † | 27.3 | 23.5 | 41.5 | 19.7 | 26.9 |
| Lumiere | 22.0 | 25.4 | 33.8 | 19.5 | 23.5 |
Results
Further results showing physical correctness.
Where video models produce plausible-looking frames that quietly break object permanence or gravity, a simulator cannot: the dynamics are enforced rather than learned.
A tower of blocks
A double pendulum
Results
State-of-the-art physical and logical VQA performance.
Alongside Physics-IQ we introduce two new benchmarks, each designed to test a kind of correctness that pixel-space models are not equipped to provide.
The first, HSPBench, requires a model to understand how a physics setup will evolve in order to answer a question about it. It consists of 40 high-school level problems — mass-spring systems, damped pendulums, projectile motion, elastic collisions — each specified precisely enough that the future is deterministic. For every scene we ask two questions with a single correct answer, such as where a ball will be on the x-axis after five seconds, giving 80 question-answer pairs. Because the answer follows from the initial conditions, the benchmark measures adherence to physical law rather than visual plausibility.
The second, ConwayBench, leaves physics behind entirely. We task the model with predicting the evolution of Conway's Game of Life from a single input frame, across six scenes with different starting configurations and textures. The rule is discrete, deterministic and utterly unforgiving. Because VDAWorld generates an executable simulation rather than predicting pixels, it acts as a guaranteed correct rule engine and achieves a perfect F1 score; video models degrade within a few frames and accumulate artefacts thereafter.
Answering questions from a VDAWorld simulation also beats asking a frontier VLM the same question directly — a VQA error of 75 against 133 for Gemini Pro — which suggests the gain comes from having a world to interrogate, not merely from the backbone model.
| Metric | VDAWorld | Wan 2.2 + Gemini | Veo 3 + Gemini | Gemini Pro | Gemini Flash |
|---|---|---|---|---|---|
| VQA error ↓ | 75 | 227 | 291 | 133 | 175 |
| Spatial ↑ | 0.66 | 0.19 | 0.22 | — | — |
| Spatiotemporal ↑ | 0.25 | 0.05 | 0.04 | — | — |
| Weighted spatial ↑ | 0.73 | 0.28 | 0.32 | — | — |
| Combined ↑ | 0.55 | 0.17 | 0.19 | — | — |
A ball is dropped from rest into a frictionless half-pipe with circular cross-section radius R=1.5 m. The ball is of mass m=0.5 kg. Standard gravity g=9.81 m/s^2 acts downwards. No friction acts anywhere. A 2-metre ruler with 1-metre alternating white and black segments is shown in the lower left.
Two circular pucks move on a frictionless surface in a rectangular bounding box. The red puck (mass 2.0 kg) starts with velocity (5.0, 1.5) m/s. The blue puck (mass 1.0 kg) starts with velocity (-1.0, -0.5) m/s. All collisions are perfectly elastic and frictionless. A 2-metre ruler with 1-metre alternating white and black segments is shown in the lower left. The coordinate system is defined with positive x to the right and positive y upwards.
Control
Full counterfactual controllability.
There is no latent space to search and no prompt to coax. Four kinds of edit are available: the caption, the initial state, the action, and the transition function itself.
Intervention. Edit the initial state to add two more dominos.
Intervention. Edit the action to change the tool's trajectory.
Intervention. Edit the caption to change what happens.
Intervention. Edit the transition function to change Conway's rule.
Appearance
A path to photorealism.
VDAWorld predicts structure and dynamics, not texture. Where photorealistic output is needed, the simulation can condition an off-the-shelf transfer model.
Cite
BibTeX
@article{omahony2025vdaworld,
title = {VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation},
author = {O'Mahony, Felix and Cipolla, Roberto and Tewari, Ayush},
journal = {arXiv preprint arXiv:2512.11061},
year = {2025}
}Acknowledgements. We thank Dr Matthew Johnson for discussions and advice. We acknowledge the use of resources provided by the Isambard-AI National AI Research Resource, operated by the University of Bristol and funded by the UK Government's Department for Science, Innovation and Technology via UKRI and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].