How to engineer an AI harness for game development

What it claims

The author, a game programmer of about 15 years, argues that LLMs fail on games because much of a game is assets, scenes and runtime state rather than text, so the model cannot see it. The fix is to make the game's own architecture expose state, inputs and evaluation as text. The claimed payoff is that an agent can run the full change, verify and assess loop "in seconds" per change, cheaply enough to run on every change. No measurements back this. It is an experience-based argument.

How it works

The architectural choices the summary lists matter because of how they connect:

• Determinism first. All non-deterministic calls go through one seeded RNG, so the Nth call returns the same value every run. The author warns that retrofitting this late in production "would likely require fundamental changes."
• One object type. With a single Thing fat struct, serialization has one path in and one path out. This avoids polymorphic deserialization and object-linking problems. Even the game manager is a Thing. The author says it is not memory-expensive.
• Commands as the interface. Every state change is a serializable command run by a processor. A bug can be reproduced by replaying a start state plus the logged commands. An agent can also write command sequences in a text file and replay them on a running build without compiling.
• Simulation as a separate DLL/library. The engine stays "dumb": it sends commands and applies the states or diffs it gets back. As a result, simulation tests run as plain C# projects with no engine. The same seam simplifies networking. Precedents cited: Brian Cronin's Roguelike Celebration talk on Monster Train, and Solium Infernum.
• Visual evaluation. Raw before/after screenshots pollute context. Instead:
• add toggles to hide UI or gameplay layers,
• crop to the area of interest,
• use odiff's similarity percentage (e.g. "85% similar") as a numeric target the agent drives toward across turns.
• Profiling as text. GUI profilers (Unity, Unreal, dotMemory) are not legible to agents. samply dumps captures to the console instead. The page notes Unity is working on agent-readable profiling.

flowchart TD
  A[Seeded state / save file] --> B[Agent edits code, data, or command script]
  B --> C[Replay commands on standalone sim]
  C --> D[Sim tests + samply profile]
  D --> E{Visual change?}
  E -- no --> B
  E -- yes --> F[Screenshot, layer toggles, crop]
  F --> G[odiff similarity %]
  G --> B

How to use it

• Keep static data (types, damage, speeds) outside the code and load it at runtime. Then you, a teammate or an agent can tune values against a shipped build without editor access.
• The author's team on Cantata gave about 7 non-programmers builds plus Depot, their in-house data editor, which the author is now productizing. Google Sheets or custom tools are named as alternatives.
• The tools named are odiff (free CLI image diff) and samply (free profiler). The page gives no commands for either.

Limits

• There is no quantitative evidence. "Seconds per loop" is asserted, not measured.
• In a postscript, the author admits they are shipping a current project that uses none of this. They say it targets larger projects and is overkill for small ones.
• The examples are C#/engine-flavoured. Balance and gameplay assessment beyond tests and profiling is left as game-specific.