AI Safety Considerations for Agents With Limited Time to Act

What it claims

Researchers at The Alan Turing Institute formalise a latency-L controller: an agent that must choose an action after at most L observations of a partially observable environment. The central quantity is R, the worst-case number of observations needed to tell apart any two environment states whose safe-action sets do not overlap. R depends on the observation model, not on the agent. Comparing R to L sorts any deployment into one of three regimes:

• Safe: R < L < ∞. ε-safety can be assured.
• Application-incompatible: L_available < R < ∞. Guaranteed safety is impossible, but the gap can be measured. The state becomes identifiable only after a delay.
• Unbound: R = ∞. No finite time horizon gives safety. The state is non-identifiable.

flowchart TD
  A[Compute R from observation model] --> B{R finite?}
  B -- no --> U[Unbound: no safety at any L]
  B -- yes --> C{R < available L?}
  C -- yes --> S[Safe]
  C -- no --> I[Application-incompatible: measurable gap]

How it works

Basic condition. If two states z and z′ produce identical observations within L steps, and their safe-action sets S(z) and S(z′) are disjoint, then no latency-L controller can be safe on both. Guaranteed safety therefore requires a non-empty intersection of S(z) across every set of states that the L observations cannot tell apart.

Theorem 1 (infinite state space). Suppose some subset of states is order-isomorphic to the natural numbers under the ambiguity relation. Then every pair of states is distinguishable after finitely many observations, yet the supremum over all pairs is infinite.

The practical consequence for testing is that sampling a finite set of states can produce misleading upper bounds on R. A safety proof must show one of the following:

1. The set of states that have a conflicting-safe-action partner is finite, so R can be computed explicitly. This holds trivially if the union of all safe actions is finite.
2. No such isomorphic map exists.

The page's example of condition 2 failing is signals such as sound or light whose arrival time grows with distance, where distance can be arbitrarily large.

Theorems 2–3 (signal mixture, the "cocktail party" case). Two states that are instantly distinguishable when signal sources are observed separately (ambiguity 0) can become permanently indistinguishable (ambiguity ∞) under a non-injective mixture operator. The construction:

• One state emits one-hot binary vectors at indices i and j from two separate sources.
• The other state emits a single vector with both i and j set.
• Under plain arithmetic addition, the two states produce the same observation.

So R goes from 0 when signals are clean to ∞ when they are mixed, even with only two states. The authors note that linear superposition is enough to cause this.

Evidence

The paper is purely theoretical. It contains no experiments or benchmarks, and each result is an existence proof by construction.

Limits

• Scope of the impossibility. The theorems show that bad environments exist. They do not show your environment is one of them. The authors explicitly disclaim completeness.
• Deterministic environments only. Results are given for deterministic environments. ε is defined for stochastic ones but mostly dropped.
• Mixture of observations only. Only observational signal mixture is treated. Superposition of the latent states themselves is excluded.
• No method for computing R. The paper says choosing L requires "a measurable quantity" but gives no procedure for estimating R in practice.
• Critique of Guaranteed Safe AI. The paper argues that the Guaranteed Safe AI framework (Dalrymple et al. 2024) never specifies the observation map, the environment, or an action deadline. Anyone building on that framework should add these explicitly.
• Incomplete text. The fetched text cuts off partway through the Discussion section.