Skip to content

Back to the catalog

Online Safety Monitoring for LLMs
arxiv.orgpaper

Online Safety Monitoring for LLMs

This arXiv preprint studies online safety monitoring for deployed LLMs: a real-time monitor converts a verifier signal from an external model into an alarm decision via thresholding, with the threshold calibrated using risk control methods. Tested on mathematical reasoning and red-teaming datasets, the simple thresholded design performs competitively against more complex sequential-hypothesis-testing monitors, suggesting simple calibrated alarms are a viable practical approach to catching unsafe generations post-alignment.

TL;DR A simple risk-controlled threshold monitor for catching unsafe LLM outputs online matches more complex sequential-testing approaches.
Open original ↗
Also seen from another angle🛡️ Security →
Document

What it claims

The paper's central practical result is one the summary omits: a single-threshold monitor not only matches the more complex "e-valuator" sequential-hypothesis-testing baselines on power, it flags unsafe outputs earlier, at roughly half the sequence length versus the e-valuators' much later detection. It also establishes when the approach fails — the monitor is only as good as its verifier signal — and quantifies the cost of swapping the external verifier for a free internal one. Code is at https://github.com/monasch/llm-monitor. It is an ICML 2026 workshop paper (Hypothesis Testing workshop).

How it works

The monitor watches a stream of safety signals s_1:t (e.g. an external verifier's predicted probability of a safe token) and raises an alarm at the first step where s_k falls below a single time-invariant threshold λ. The novelty is entirely in how λ is calibrated on a labeled held-out set, with two variants:

  • CRC (conformal risk control): controls false-alarm risk in expectation over calibration draws; less conservative.
  • UCB (Hoeffding-Bentkus upper bound, per Bates et al. 2021): controls risk with high probability (1−δ); more conservative, needs a larger calibration set, preferred when a single violation is costly.

The baselines (e-valuators, Sadhuka et al. 2025) instead train T per-step density estimators feeding an e-process for a sequential test. On an alarm, downstream actions include halting generation, escalating to a stronger verifier, or human review.

Evidence

Three use cases, three metrics (false alarm rate, power, detection delay):

  • MATH factuality: generators Claude Haiku 4.5 (90% correct) and Mistral-7B-Instruct-v0.3 (26% correct); labels from o3-mini vs ground truth; signal is Qwen2.5-Math-PRM-7B step probabilities. UCB, e-valuator-anytime, e-valuator-PAC control false-alarm rate; e-valuator-anytime violates the bound at ε∈{0.05,0.1} on Mistral. CRC/UCB flag after ~half the sequence; e-valuators detect more but much later.
  • Harmlessness: Anthropic Red Teaming (multi-turn, Llama Guard signal) and FineHarm (single-turn, a Qwen2.5-1.5B "SCM" verifier fine-tuned for token-level harm). On FineHarm the fine-tuned verifier gives power near 1; on Red Teaming Llama Guard's power stagnates below 0.8. e-valuator-PAC wins power on Red Teaming; CRC/UCB win on FineHarm.
  • Signal ablation (Mistral, MATH): replacing the PRM with the generator's own per-step-minimum token log-probability (free, no extra forward pass) yields substantially lower power at matched false-alarm rates across all four monitors — e.g. near ε=0.3 the PRM variants beat log-prob clearly.

Limits

The monitor caps at its verifier's quality: a generic safeguard (Llama Guard on Red Teaming) underperforms a task-fine-tuned one. The cheap internal log-prob signal that would avoid a second forward pass per step is markedly weaker. UCB's stronger guarantee costs a more conservative threshold and larger calibration set. Results are workshop-scale; the missed-detection-risk methodology is deferred to an appendix (not in the fetched text).

Techniques named here

  • Online safety monitoring via thresholded verifier signal
  • Sequential hypothesis testing monitor

Related in ai-safety