Programmable World Model

What it claims

Programmable World Model separates the engine that tracks world state from the video model that renders it, on the argument that pixel-trained video world models can produce plausible frames but cannot maintain an authoritative, editable, rule-governed state across occlusion and long horizons. The differentiator over the existing summary is the reasoning behind the representation choice and the explicit contrast with prior explicit-state systems. Code is at github.com/AlayaLab/pwm; project page at alaya-lab.github.io/pwm.

How it works

Input is a single reference image plus a natural-language description. A VLM-based coding agent writes the initial entity layout, assigns states/attributes, and generates executable programs defining interactions, event triggers, and win/lose objectives (the figure shows rules like on_hit: hp -= damage, on_hp_zero: status = down, win: all_enemies_defeated). A lightweight state executor applies player actions and events per those rules; a deterministic state compiler projects the evolving state-augmented 3D OBBs along the target camera trajectory and rasterizes them into pixel-aligned spatiotemporal controls; the pretrained video model renders from those. Users revise behavior by issuing further instructions to the coding agent. A separate data curation pipeline recovers the spatial/semantic supervision needed to train the renderer.

The core argument is a representation trade-off (Figure 2): text gives no world-space geometry, 2D boxes/masks are view-dependent, full 3D scenes/G-buffers demand costly supervision and force the system to specify low-level motion. OBBs sit in the middle — view-independent, reprojectable, compact. The paper also names a training–inference asymmetry: training extracts representations from already-realized dynamics, while inference must construct time-varying structure from a high-level state transition before any visuals exist, so a representation cheap to obtain in training is not necessarily cheap to produce at inference. Finer representations push the system toward solving the full animation/simulation problem; OBBs leave articulation and secondary motion to the generative prior.

Evidence

On the authors' own CombatStateBench: 94% Count Accuracy and 98% State Accuracy, stated to substantially outperform existing interactive video world models. The benchmark measures whether generated video preserves visible alive-character counts and visually realizes death states under diverse camera and entity motion. The page gives no competitor numbers, no per-baseline breakdown, and no compute or model-size figures.

Limits

The paper's positioning against the two closest systems is worth checking directly: StatePlay predicts state jointly with observations, so state errors accumulate; MASS advances shared state via a learned Logic Engine, so it remains susceptible to transition-prediction error. PWM's claimed advantage is a rule-executing lightweight engine whose state is programmable, editable, and verifiable — but the page substantiates this only on a self-authored combat benchmark, with no baseline numbers shown and no evidence outside combat scenarios. Renderer quality depends on the data curation pipeline recovering good supervision, which the fetched text describes only in outline. Dated September 2026 (arXiv v1).