OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

What it claims

OnTrack is a streaming version of Otap, an earlier batch method that scores a finished agent trajectory by aligning its execution DAG against reference DAGs with optimal transport. OnTrack's contribution is making that alignment run live, at about a millisecond per step. At the end of a run, one final solve makes its output identical to the batch evaluator's, provided the dependencies discovered online match what batch extraction would find (Prop. 1). It emits per-step verdicts rather than a score: OK, EXPLORING, WARN, LOOP, STALLED, CAUSAL INVERSION, OFF TRACK, BLOCK.

How it works

Each agent step is a node in a growing DAG. An edge is added when a later step consumes an earlier step's output. The authors name four reasons why re-running the batch method after each step fails, and give a fix for each:

• Prefix vs. whole. Mid-run, a batch score penalises every reference step the agent hasn't reached yet. OnTrack uses frontier masking: it only expects the reference steps the agent could have reached so far.
• Provisional structure. A step looks disconnected until something uses its output. A new node's structural weight therefore starts at zero and phases in, and edges discovered late are inserted retroactively.
• Compute budget. OnTrack does not re-solve the alignment each step. It runs one warm-started Sinkhorn iteration per event and only runs the solver to convergence before acting on an intervention.
• Decisions, not scores. The aggregate score means different things at different trajectory lengths, so no fixed threshold works. Verdicts come from per-step signals with a short grace window, never from the aggregate.

The solver uses unbalanced, entropic, fused Gromov–Wasserstein OT. Losses across references are combined with a soft-min (temperature 0.05). Each graph carries a dependency-hop distance matrix, normalised to [0,1] and undirected by default.

Dependency extraction is deterministic string matching, not an LLM call. An artifact ledger maps each identifier to the step that first surfaced it:

flowchart LR
  A[Harness declares parents] --> E[Edge]
  B[Tool logs touched objects] --> L[Artifact ledger lookup]
  C[No metadata: schemas/rules pick args] --> L
  L --> E

The no-metadata case is described as the common one in practice, and it is the case in the evaluation corpus.

Capabilities fall into three layers that switch off as their inputs disappear, with no reconfiguration:

• L1 vital signs (loops, stalls, unproductive repeated calls) are always on.
• L3 pre-execution gate blocks an irreversible action whose declared prerequisites are missing. It needs tool schemas.
• L2 plan-deviation alignment needs references.

Reference provenance also scales how much authority a verdict carries:

Evidence

The only numbers on the fetched page are the abstract figures already in the summary, from SWE-bench trajectories. Ablations cited from Otap motivated the design: the structure term is load-bearing, restricting to a single reference costs 16.9 AUROC points, and heuristic dependency extraction is the largest error source. Those are prior-work results, not OnTrack's.

Limits

• Dependencies can be invisible. Example: sed -i ... $(find . -name '*.py' | head -1) resolves its target inside the shell, so no harness records the link to the earlier step. The graph is therefore always treated as provisional, and Prop. 1's batch equivalence depends on extraction being correct.
• Without references there is no plan-deviation detection. Only loops, stalls and the prerequisite gate remain.
• Truncated page. The fetched text stops in the problem formulation. Results tables, the calibration finding, the verdict-engine details (Sec. 4.5) and Sec. 6.4's cost-overrun analysis were not visible.
• No adoption details on the visible page. There is no code, licence or install information, and evaluation is on SWE-bench only.