Skip to content

Back to the catalog

Online Safety Monitoring for LLMs
arxiv.orgpaper

Online Safety Monitoring for LLMs

This paper proposes an online safety monitor for deployed LLMs that converts a verifier model's signal into a real-time alarm via a threshold calibrated using risk control, flagging when the model's outputs can no longer be trusted as safe. Tested on mathematical reasoning and red-teaming datasets, the simple thresholded verifier approach matched more complex monitors built on sequential hypothesis testing.

TL;DR A calibrated-threshold verifier monitor for catching unsafe LLM outputs at inference time performs on par with sequential hypothesis testing methods.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

"Online Safety Monitoring for LLMs" (Schirmer et al., ICML 2026 Workshop on Hypothesis Testing). The paper's practical payload is a decision-rule recipe plus three findings the summary's "matched more complex monitors" phrasing hides. Code: https://github.com/monasch/llm-monitor.

How it works

The monitor is a single time-invariant threshold λ applied to a streaming safety signal s_t (a verifier's per-step safe-token probability). It raises an alarm at the first step where s_t < λ: Φ_t = 1{∃k ≤ t : s_k < λ}. The engineering choice is how to calibrate λ on a labeled held-out set, and the paper gives two options with different guarantees:

  • CRC (conformal risk control): controls the false-alarm risk in expectation over calibration draws. Less conservative, higher power, needs fewer calibration samples.
  • UCB (Bates et al. 2021, using the Hoeffding–Bentkus bound): controls risk with high probability (1−δ). More conservative threshold, needs a larger calibration set. Prefer when a single violation is costly.

What the experiments establish

  • Earlier detection is the real win. Against e-valuator baselines (Sadhuka et al. 2025, which train T per-step density estimators for a sequential test), CRC/UCB match power but flag incorrect sequences after ~half the generation, while e-valuators detect the same failures much later. Lower detection delay limits user exposure and token cost.
  • The monitor is only as good as its signal. On FineHarm, an SCM verifier (Qwen2.5-1.5B fine-tuned for token-level harmfulness) gives power near 1; Llama Guard on Anthropic Red Teaming stagnates below 0.8. If you deploy this, budget for a verifier trained for your specific detection task.
  • Free internal signals underperform. Substituting the generator's own per-step minimum token log-probability for an external PRM (Qwen2.5-Math-PRM-7B) avoids a second forward pass but yields substantially lower power at matched false-alarm rates (e.g. near ε=0.3). The cheap signal is a real downgrade, not a free lunch.

Setup detail worth noting: on MATH, generators were Claude Haiku 4.5 (90% correct) and Mistral-7B-Instruct-v0.3 (26%), with o3-mini providing ground-truth labels.

Caveats

This is a workshop paper on a simple baseline, not a deployment-hardened system. Guarantees require a calibration set drawn exchangeably from the deployment distribution — distribution shift breaks the risk control. e-valuator-anytime violated its bound at ε ∈ {0.05, 0.1} on Mistral, a reminder that "controlled" holds only under the stated assumptions.

Technologies named here

Related in AI Security