Scope Before You Persist: Preventing Cross-Family Interference in Agent Memory

What it claims

The paper's actionable finding is that on this benchmark, **every method that deploys a single global skill edit performs below the do-nothing baseline**. Static (never edits) scores 0.775 mean trajectory utility; ORC-gated global memory reaches only 0.713, Replay 0.707, Latest-only 0.703, Self-judge 0.736. The cause is cross-family interference: an edit certified as valid on the family that produced it degrades unrelated families when applied globally. The fix is scoping retrieval to the certifying family, which lifts utility above Static and raises accepted-update volume ~5× with no harmful acceptances.

How it works

Frozen actor and editor (no weight training). Each round the actor solves 4 discovery tasks with the current 360-character skill policy, the editor proposes a new 360-char policy, and a gate accepts or retains the incumbent. The gate is ORC (Orthogonal Regression Control): it scores candidate-minus-incumbent pass rates on three channels per task — 4 public examples, 4 private exact-output tests (hidden from the prompt), and ≥6 answer-free metamorphic checks — partitioned into the current family plus one group per historical family. Each group/channel gets a one-sided 95% paired-bootstrap lower bound; the candidate deploys only if the current family's private bound exceeds δ_new=0, and all public/metamorphic/historical bounds stay above −δ_reg (0.02), with no unsafe program. The AND conjunction stops a gain in one channel averaging out a regression in another. Scoped-ORC changes only one thing: an accepted candidate is stored in the originating family's slot rather than replacing the global policy; retrieval serves the family slot if it exists, else the base policy. Prompt budget stays fixed at 360 chars.

Evidence

Main table, eight paired 12-round streams (metric: mean trajectory utility, hidden balanced checkpoint bank):

Frozen-compute matches Static exactly despite ~5× the endpoint cost, isolating deployment (not extra calls) as the driver. The fixed-output scope intervention (proposals, decisions, scores held constant, only retrieval changed) raises 0.713→0.816 and cuts harmful deployments 6/8→0. Prespecified paired contrasts were weak: ORC−Replay mean trajectory difference was 0.006 [−0.045, 0.048], p=0.84 — the gate alone did not beat Replay. The scope effect, not the gate, carries the paper. The 27-seed full extension gives Scoped−Global +0.063 [0.037, 0.094], 63 vs 12 accepts, ≥2 accepts in 19/27 streams, 0/63 harmful.

How to use it

Actor/editor is Qwen3-Coder-Next on a pinned Parasail BF16 endpoint via OpenRouter (fallback disabled), temperature 0, 700 actor / 500 editor tokens. Benchmark is PROCSTREAM-RSI v3.1, a 12-round code-repair stream with a fixed family schedule (boundary, rotation, ordering, normalization, …), 123 tasks per stream, exact-output public/private/hidden inputs pairwise disjoint. Transfer diagnostics: 32 HumanEval+ tasks (no feedback) and a cross-actor test with GPT-OSS-120B (AkashML BF16, 2,000-token cap). All thresholds fixed after a two-seed pilot before the eight-seed main study.

Limits

This is an author-prepared manuscript, "not a record of conference acceptance." The benchmark deliberately measures adaptation on familiar, fixed contracts — oracle semantics and family order are constant across streams; seeds vary only tokens, names, and cases, not the underlying convention. Static's high 0.775 reflects "substantial prior competence," so the setting is one where a good agent already does well and edits mostly risk harm. The gate's own accuracy showed no significant edge over the simpler Replay baseline; the benefit is attributed to scope, which by construction cannot protect a global edit against families not yet seen. Single model family (Qwen3-Coder-Next) drives the main results. The fetched text cuts off mid-Table 2, so the ORC−Latest-only and ORC−Batch-ORC contrasts are not visible here.