Training activation classifiers for safety and alignment

What it claims

OpenAI runs activation probes on all assistant completions starting with GPT-5.6. The probes read one middle layer of the residual stream and stop a completion partway through when it crosses a threshold. The post says these probes are more robust to jailbreaks than much larger text classifiers in its red-teaming, cheaper to run, and more accurate on more capable models. The figures come from internal experiments and are "not necessarily representative of deployed classifiers."

How it works

The architecture follows attention probes (Kantamneni et al., 2025): an MLP, then QV-attention, then an exponential moving average. The EMA replaces a rolling mean and cuts streaming state from O(window_size × num_heads) to O(num_heads). The authors report essentially no loss in predictive power. A learnable pre-MLP bias, initialised from the negative mean of the training activations, stabilises optimisation. Many domains use multi-output probes that share most weights. According to the authors, this may help output categories that have little data.

Each message carries a single label. At inference the probe scores every message prefix. In training, those prefix scores are combined with a softmax into one message-level score, where Kramar et al. (2026) use a hard max. Softmax aggregation modestly improves the recall versus flag-rate Pareto front.

The training setup has these parts:

• Data: up to 1M messages, a mix of synthetic and de-identified conversations. Gains continue up to that scale.
• Labels: produced by OpenAI's most capable reasoning models acting as LLM judges.
• Optimiser: AdamW for fewer than 5 epochs.
• Seeds: at least 32 random initialisations per hyperparameter configuration. The best seed is chosen on a validation set and then checked on a held-out set.
• Sweeps: the hyperparameters worth sweeping are learning rate, source layer, and probe capacity.
• Throughput: training is bandwidth-bound, so tens of probes are vectorised and fused on one accelerator in about the time one probe takes.

In deployment, probes run on the inference engine with a minimal pooling buffer and emit one logit per sampled token. This makes them easier to run fail-closed than an external text classifier.

flowchart LR
  A[Sampled token] --> B[Probe logit]
  B -->|below threshold| C[Keep streaming]
  B -->|above threshold| D[Stop stream]
  D --> E[Reasoning model judges]
  F[Tool call] --> G{Marked safe?}
  G -->|no| H[Blocked]
  G -->|yes| I[Executes]

When a logit exceeds the threshold, streaming to the user stops and a reasoning model makes the final call. Tool calls are held until they are marked safe.

Evidence

• Prefix robustness: benign tokens were placed before a violative suffix. MultiMax (Kramar et al., 2026) was significantly more affected by these prefixes than Max of Rolling Means Attention. Max of Rolling Means Attention behaved much like the authors' EMA attention.
• Architectures that did not help: linear probes (including multi-layer concatenation), MultiMax, and covariance probes all failed to improve performance. Full-attention probes matched the chosen design but cannot stream with a small buffer.
• Seed variance: about 85% of the variance in probe performance comes from initialisation randomness, not data order. This is the justification for training many seeds jointly.
• Transfer: probes transfer well across post-training stages and less well across pre-training checkpoints.
• Scaling: probe performance keeps rising on frontier models.

The page gives no absolute recall or escalation numbers.

Limits

• Probes detect only known, labelled kinds of behaviour.
• A probe must be retrained for each new pre-trained base model. Within one model family, it degrades as weights drift from the training checkpoint, so periodic retraining is needed.
• Jailbreak robustness is highest for attacks not optimised against the probe. The post acknowledges that real attacks adapt.
• Deployment decisions rest on a recall-escalation curve. Recall is measured on a small, carefully labelled set, and escalation rate on a large representative set.
• No code, weights, or thresholds are released.