Training activation classifiers for safety and alignment
Training activation classifiers for safety and alignment describes how OpenAI trains lightweight probes on a model's residual stream activations to flag misaligned or policy-violating completions. Probes use an MLP plus QV-attention architecture with an exponential moving average and a rolling softmax objective. They run in streaming mode on the inference engine, have run on all assistant completions since GPT-5.6, and transfer well across post-training checkpoints but less across pre-training checkpoints.
ai-safety, research· Published and added 2 Nov 2026







































.jpg)

















































































%20(1).png)








































































































































