27 results for “World model”
arxiv.orgpaper🧠 AIworld2vec is a method for learning mechanistic world models from observations and actions alone, without reward information. Its latent dynamics reproduce the true system over long prediction horizons using at least one order of magnitude fewer parameters than DreamerV3, TD-MPC2 and LeWM. The authors report state-of-the-art control of continuous environments from pixels and accurate system identification across a benchmark of 71 attractors.
world modellatent dynamics learning
world2vecDreamerV3TD-MPC2+2
research· Added 31 Oct 2026, published 29 Oct 2026
latent.spacearticle🧠 AIDecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.
vision-language model
ElorianGlimpseDecoyCount+17
evals, multimodal, models· Added 28 Oct 2026, published 29 Oct 2026
🗂️
deepmind.googlearticle🧠 AIGenie 4 is Google DeepMind's real-time interactive world model, announced as a research preview for trusted testers. It generates video with native audio and interactive dialogue, steered by keyboard controls and text prompts. Gemini 3.8 Flash acts as a guiding agent that plans steps and checks frames, and Gemini 4 Argon is used for robotics experiments. The post lists limitations such as drift and object permanence, and says outputs carry SynthID watermarks.
GeminiGoogle AI StudioSynthID
Google DeepMindGenie 4Gemini 3.8 Flash+5
models, multimodal, agents· Published and added 27 Oct 2026
blog.googlearticle🧠 AIGenie 4 is Google DeepMind's world model for real-time interactive simulation, with native audio generated on the fly (ambient sound, music, dialogue). It builds on Gemini models: Gemini Omni grounds scenes in real-world appearance and behavior, and other Gemini models keep the simulation coherent. Users steer scenes by natural language. Access is limited to trusted testers now, with Google AI Ultra (Project Genie), the Gemini API and AI Studio to follow. The post gives no architecture or benchmark detail.
GeminiGoogle AI StudioGemini API
Google DeepMindGenie 4Gemini Omni+2
models, multimodal· Added 30 Oct 2026, published 27 Oct 2026
arxiv.orgpaper🧠 AI Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.
diffusion modelFlashAttentionCUDA
VeloxLingBot-Fast-CamWBench+1
research, multimodal, inference· Added 17 Oct 2026, published 15 Oct 2026
developers.googleblog.comarticle🧠 AIGoogle's developer post presents Gemini 4 Argon as a physical-reasoning model for the whole robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark, ahead of other frontier models. It then gives representative prompts: controlling a Boston Dynamics Spot with a MuJoCo digital twin, closed-loop code-as-policies arm control, steering the Gemini Omni world model, and building digital twins from teleoperation episodes. Further prompts cover retargeting human video to a 76-DoF humanoid, hindsight annotation of ALOHA 2 videos, a general value function on UMI data, sim-data generation for VLA fine-tuning (56% to 75% real success), and reproducing control papers in MuJoCo WebAssembly. It is useful for seeing how the model is prompted for agentic, simulation-driven robotics work.
GeminiMuJoComujoco_warp+5
Gemini 4 ArgonGoogleERQA+11
models, multimodal, agents· Published and added 13 Oct 2026
arxiv.orgpaper🧠 AI UniW(A)M is a mixture-of-experts model built on diffusion Transformers that unifies action-conditioned world modeling (AC-WM) with several world-action model (WAM) variants. Shared experts capture common physical dynamics, while atomic capability experts are reused across interfaces, each keeping its own conditional pathway. At deployment a plug-in adaptive interface router (PAIR), trained on a small subset of target-task chunk scenarios with the backbone frozen, picks a WAM interface for each execution chunk. It reaches 88.3% average real-world success against 76.7% for the strongest baseline, leads WorldArena on all three action-conditioned world-modeling tasks, and PAIR raises OOD success from 63.3% to 76.7%.
Mixture of Expertsdiffusion modelTransformer
UniW(A)MPAIRLIBERO+3
research, multimodal· Added 14 Oct 2026, published 13 Oct 2026
deepmind.googlearticle🧠 AI DeepMind presents Gemini 4 Argon as a general-purpose physical reasoning model for the robotics lifecycle. It reports 87% on the Embodied Reasoning (ERQA) benchmark and gives representative prompts across several areas: controlling a Boston Dynamics Spot with a MuJoCo digital twin, code-as-policies arm control with a visual coordinate harness, and steering the Gemini Omni world model in a visual feedback loop. Further examples cover building MuJoCo and mujoco_warp scenes from teleoperation data, egocentric video or a hand-drawn sketch for 76-DoF humanoids, hindsight annotation and value-function progress prediction on video, sim data generation for VLA fine-tuning (success rate 56% to 75%), and reproducing control papers in MuJoCo WebAssembly. It matters as a concrete look at how a frontier model is being used as agentic coder, perceiver and sim builder for robotics.
MuJoComujoco_warpBlender+5
Gemini 4 ArgonGoogle DeepMindERQA+10
models, multimodal, agents· Published and added 13 Oct 2026
arxiv.orgpaper🧠 AI DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It renders action trajectories into image-space conditions aligned to target videos by offline geometric calibration. Counterfactual post-training modifies recorded trajectories to cover unsuccessful interactions and contact configurations. A human-annotated defect dataset trains an embodied video reward model whose scores drive RL post-training. On AgiBot it reaches state-of-the-art action following and cuts the human-assessed interaction defect rate from 48.12% to 6.25%. It ranked first in the world model track of the AgiBot World Challenge 2026.
DreamTrueAgiBotAgiBot World Challenge 2026
research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026
arxiv.orgpaper🧠 AI WOVEN is a training source and benchmark for visual transition reasoning in multimodal LLMs. It has 36,076 examples spanning 20 scene types, 5 action types and 8 reasoning types, built from rollouts of video-pretrained generative models. Evaluating 38 frontier MLLMs (including GPT-5.4 and Qwen3-VL-235B-A22B) shows systematic deficits that persist with scale. Training on WOVEN subsets of about 2,000 items improves 22 of 26 external benchmarks by up to 27.3 points, and WOVEN data can replace 30-50% of a task's own training data. The authors derive a recipe: select supervision by the reasoning operation it teaches rather than by scene or action, and prefer larger visual-state changes.
vision-language modelQwen3-VL
WOVENGPT-5.4
research, multimodal, evals, training· Added 9 Oct 2026, published 8 Oct 2026
decrypt.coarticle🧠 AIGoogle launched Playground, a browser-based tool letting U.S. adults 18+ describe a video game in plain language and get a playable, shareable result back without coding, with creation access tiered by Google AI subscription. The piece ties Playground to Google's broader AI-for-games push: DeepMind's Genie (an 11B-parameter model trained on 200,000+ hours of gameplay video) and its successor Genie 3, a real-time interactive world model opened to AI Ultra subscribers via Project Genie, plus an upcoming integration with Unity Spark for 3D, professional-level game mechanics. Google did not disclose which models power Playground or its pricing, and framed it as an early experiment, while the announcement reportedly sank Unity and Roblox stock on fears of AI displacing traditional game-dev tools.
Gemini
GoogleGoogle DeepMindUnity+3
multimodal· Published and added 7 Oct 2026
arxiv.orgpaper🧠 AI RoboJEPA is an 8B-parameter latent world model built on the Joint Embedding Predictive Architecture (JEPA), trained on real-robot data spanning 12 robotic embodiments. The paper establishes the first scaling laws for multi-embodiment robotic world models, showing that imagination error (latent rollout error) follows a second-order power law in compute and predicts downstream planning performance, and demonstrates zero-shot deployment of the model as a goal-image-conditioned planner on real hardware for long-horizon tasks.
Joint Embedding Predictive Architecture
RoboJEPA
research, multimodal· Added 8 Oct 2026, published 7 Oct 2026
arxiv.orgpaper🧠 AI World Models' Last Exam in Physics introduces a measurement-based benchmark of 40 controlled tasks across mechanics, optics, fluids, thermal/phase-change phenomena, electromagnetism, and surface tension, designed to test whether video world models generate physically consistent outputs. Each task pairs an initial image and prompt with predefined physical criteria, and a custom evaluator combines observability screening with task-specific quantitative measurements rather than relying on reference videos or model-based judgments. Testing eight video generation models across 1,280 generated videos found persistent physical inconsistencies, with the best model scoring only 57.76/100, and the evaluator showed higher agreement with human judgments than a vision-language model baseline.
vision-language model
World Models' Last Exam in Physics
evals, multimodal· Added 7 Oct 2026, published 6 Oct 2026
fortune.comarticle🧠 AIFortune interview with Yann LeCun, who dismisses AI extinction risk and recent "rogue AI" incidents (e.g. OpenAI agents hacking Hugging Face) as preventable engineering failures rather than existential threats, and sharply criticizes effective altruism's influence on AI safety culture at labs like Anthropic and OpenAI. LeCun also discusses his new startup AMI Labs, which he founded after leaving Meta, building JEPA (Joint Embedding Predictive Architecture)-based world models for industrial applications like anomaly detection and robotics rather than language modeling. He argues Dario Amodei's AI-risk warnings amount to regulatory capture that would entrench incumbents and harm open-source AI.
JEPA
Yann LeCunDario AmodeiAnthropic+14
ai-policy· Added 4 Oct 2026, published 3 Oct 2026
arxiv.orgpaper🧠 AI ReFract is a benchmark of 150 expert-validated entries testing whether LLM agents can infer a user's role and act only through tools/actions legitimate for that role, a capability the authors term "Perspective Awareness." Entries are grounded in anonymized industrial-maintenance support queries and use constructed Text World Models to simulate the agent's operating environment and generate perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of tasks, with over half of trajectories containing attempts at perspective-violating actions, exposing this as a largely unsolved axis of agent evaluation relevant to high-stakes, physically-enacted agent deployments.
ReFract
evals, agents· Added 5 Oct 2026, published 2 Oct 2026
arxiv.orgpaper🧠 AI Physis-Lang is a self-evolving framework that uses natural language (describing entities, causes, interactions, governing principles, and temporal evolution) as the shared physical-knowledge representation across data curation, training, and generation for video world models. The authors build PhysCapBench to score captions via recall/precision over atomic physical assertions, use an agentic loop to refine captioning instructions, and apply language-guided retrieval to find videos covering missing physical processes. Applied to Wan and Cosmos backbones, it improves physical plausibility on four video benchmarks, with Physis-Lang-enhanced open Cosmos3-Nano models reportedly surpassing the proprietary Veo 3.1.
WanCosmosCosmos3-Nano+2
research, multimodal· Added 1 Oct 2026, published 30 Sep 2026
arxiv.orgpaper🧠 AI This paper studies how to design the shared representation space used by world-action models, which jointly learn robot control policies and predict future observations. It finds that neither reconstruction fidelity nor raw pre-trained perceptual features alone yield good policy learning, and proposes ReWAM, built on pre-trained DINO features with a Temporal Representation Bottleneck and Action-Grounded Representation Shaping that routes only action-loss gradients into the bottleneck so the policy shapes what is encoded while the world model learns its dynamics. Without generative video pre-training, ReWAM reaches 93.6% success on RoboTwin 2.0 and an 8.28% success rate on RoboDojo using roughly 600 hours of embodied pre-training data.
DINO
ReWAMRoboTwin 2.0RoboDojo
research, multimodal· Added 30 Sep 2026, published 29 Sep 2026
arxiv.orgpaper🧠 AI This paper introduces stochastic world models as a new class of perception surrogates for formally verifying vision-based neural feedback (control) systems, replacing GAN-based surrogates that are large and hard to verify. The world model is built with physically grounded latents using operations standard verifiers can bound, and reproduces held-out frames more faithfully than GAN surrogates with up to 130x more parameters. Combined with a verification procedure mixing falsification, adaptive refinement, symbolic and backward analyses, it resolves the entire state space on an emergency braking benchmark (38% more than the prior state-of-the-art verifier) and over 80% of the state space on a previously-unverified RGB version of the benchmark.
GAN
research, multimodal· Added 30 Sep 2026, published 29 Sep 2026
arxiv.orgpaper🧠 AI This arXiv paper proposes the Agent-Editing World Model (AEWM), which reframes world modeling for LLM agents: instead of predicting/reconstructing environment observations (high-entropy tool responses), it models how reasoning and actions shape future task progress, addressing "task-state contamination" where stale plans and unsupported assumptions corrupt later decisions. It introduces Action Judge (classifying decisions as Critical, Exploratory, or Noisy) and State Revision, combined via EditAct which directly edits the agent's state using real execution feedback rather than issuing critiques. Trained across Search, Terminal, and Software Engineering domains, AEWM reports 70.5% macro-F1 on its Action Judge benchmark (+10.6 over the strongest frontier baseline) and EditAct improves average scores by 3.2–6.7 points across six benchmarks and three agent backbones, with an RFT variant (AEWM-RFT) also outperforming Self-RFT.
AEWMAction JudgeState Revision+3
agents, research· Added 26 Sep 2026
latent.spacearticle🧠 AI A Latent Space interview with Runway's CTO Kamil Sindi, Principal Research Scientist Robin Kahlow, and co-CEO Anastasis Germanidis about GWM Worlds 2, a research-preview "autoregressive diffusion" model that streams real-time interactive video (720p/24fps) and audio (48kHz), and its new WorldPrompt input format for fixing a first frame and issuing timestamped in-world events. It details the engineering pipeline (bidirectional diffusion → autoregressive fine-tuning → distillation for real-time speed), open problems like error accumulation, context/memory management, and causal/counterfactual correctness, and situates Runway against Google DeepMind's Genie 3, Odyssey-2 Pro, and World Labs' RTFM. A companion podcast transcript covers Runway's history from pix2pixHD/Gen-1/Gen-2 through the Stable Diffusion collaboration to its pivot toward world models.
Stable Diffusion
RunwayGWM Worlds 2WorldPrompt+21
multimodal, models· Added 29 Sep 2026, published 25 Sep 2026
arxiv.orgpaper🧠 AI AD-WM is an action-discriminative joint-embedding world model designed for counterfactual model predictive control, addressing the problem that world models trained only on factual transition prediction often fail to distinguish between alternative candidate actions during planning. It combines residual latent dynamics with predictor-level action-recovery regularization (inverse dynamics plus a normalized recovery objective grounded in conditional mutual information) to preserve action-dependent differences needed for MPC. On OGBench-Cube it raises hard-start success from 3.7% to 52.0% over a matched LeWM baseline, and with a frozen V-JEPA 2 encoder it improves zero-shot pick-and-place transfer on a real Franka arm from 42.2% to 71.1%.
V-JEPA 2
AD-WMOGBench-CubeLeWM+2
research, multimodal· Added 25 Sep 2026, published 24 Sep 2026
arxiv.orgpaper🧠 AI WorldCrafter is a video world model that learns a camera-queryable implicit 3D-aware memory to preserve consistency across long horizons and viewpoints during interactive scene exploration. A memory encoder and pose-conditioned readout module, trained jointly with the video generator, compress multi-view historical observations into fixed target-view-specific tokens without explicit depth-based correspondences, and combined with few-step distillation enable streaming exploration from a single image or text prompt with improved long-horizon consistency and camera-control accuracy.
WorldCrafter
multimodal, research· Added 22 Sep 2026, published 21 Sep 2026
arxiv.orgpaper🧠 AI This arXiv preprint reports the first chip-scale, space-time multiplexed Gaussian Boson Sampling (GBS) processor, built on a thin-film lithium niobate chip integrating high-speed electro-optic modulators and on-chip delay lines, running at 4 GHz and detecting up to 11,059 photons within 1 millisecond. Beyond demonstrating quantum computational advantage, the authors reconfigure the photonic hardware as a GBS-powered world model for modelling physical dynamics, achieving lower prediction error with fewer trainable readout parameters than a classical echo state network (ESN) baseline. It's notable for the AI catalog as a concrete example of photonic quantum hardware being repurposed as a reservoir-computing-style model for physical-dynamics prediction.
thin-film lithium niobateGaussian Boson Samplingecho state network
research· Added 22 Sep 2026, published 10 Sep 2026
arxiv.orgpaper🧠 AI Programmable World Model decouples persistent world-state evolution from visual rendering in video world models: an agent compiles natural-language instructions into executable programs governing entity states and transition rules, which a lightweight engine executes and maintains as explicit global state (including off-screen entities and non-visual attributes). State-augmented 3D oriented bounding boxes plus camera trajectory are deterministically compiled into pixel-aligned conditioning signals fed to a pretrained video model that renders the scene, enabling playable games with predefined mechanics and direct entity control. On the paper's new CombatStateBench benchmark it achieves 94% Count Accuracy and 98% State Accuracy, beating existing interactive video world models on long-horizon coherence.
video model
CombatStateBench
research, multimodal, evals· Added 22 Sep 2026, published 9 Sep 2026
arxiv.orgpaper🧠 AI SyncWorld is an action-conditioned world model that acts as a zero-shot simulator across unseen robotic environments without additional training. It uses a "visual calibration episode" — paired frames and actions demonstrating controllable degrees of freedom — to specify a setup-specific Action-Visual Mapping in context, addressing the problem that identical numerical actions manifest differently across cameras, embodiments, and environments. Experiments show it can simulate accurate rollouts in novel settings and enable test-time policy improvement without further training.
SyncWorld
research, multimodal· Added 22 Sep 2026, published 8 Sep 2026
arxiv.orgpaper🧠 AI SolarWM is an open foundation for building interactive video world models, providing a reconfigurable multi-source data engine that unifies 1.43 million video clips from 10 datasets into a frame-aligned contract (visual observations, camera geometry, captions, quality metadata, provenance). Using shared camera-conditioning and training interfaces, the authors train four 5B-33B causal models built on Wan2.2, LTX-2.5, and MiniMax-H3 backbones via a three-stage recipe (bidirectional adaptation, teacher-forced autoregressive initialization, distribution matching distillation), enabling real-time interactive rollouts spanning minutes to hours despite training only on 5-second sequences. The project releases data, pipeline, training recipes, weights, and framework code for reproducible world-model research.
Wan2.2LTX-2.5MiniMax-H3
SolarWM
research, multimodal, training· Added 22 Sep 2026, published 2 Sep 2026
arxiv.orgpaper🧠 AI This paper introduces predicted-state matching, a training objective for discriminative world models used by web agents to rank candidate actions at test time. Unlike standard supervised next-state prediction (predicting HTML/AXTree snapshots), the objective trains models to distinguish the true resulting state from alternative-action outcomes, better aligning with downstream ranker/PRM use. Trained on a branching dataset derived from WebArena Go-Browse trajectories, the approach improves PRM-style action ranking on WebPRMBench and end-to-end task success on WebArena-Lite.
WebArenaWebPRMBench
Go-BrowseWebArena-Lite
research, agents· Added 22 Sep 2026, published 2 Sep 2026