OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning

What it claims

OwlSight is a fine-tuned small LLM that acts as a binary classifier. It reads a natural-language Factor, the task context and a reasoning trace, then outputs the probability that the Factor is "Applied." The authors are Yan, Zhu, Zhang and Shao of Shanghai AI Laboratory and Beihang University. The paper is under review at ICLR 2027 and was posted in October 2026. On a benchmark of 32,222 Factor-conditioned queries over 15,971 trajectories, OwlSight scores a macro-average G-Mean² of 69.57 with fine-grained Factors and 66.80 with coarse-grained Factors.

How it works

A Factor has four parts: a target description q, a relation ρ, the criteria for Applied, and the criteria for Not Applied. A Factor counts as Applied only when the target "substantively influences or shapes" the reasoning or behavior. A trace that merely mentions or recognizes the target is labelled Not Applied. The monitor sees only the Factor, the task context and the trace. Final answers, tool outputs and environment outcomes are held out and used for supervision and diagnostics only.

The model is a pretrained causal LM trained with SFT. A linear head on the final non-padding hidden state produces a sigmoid probability, and training uses BCE loss per trajectory-level bag.

Long traces use multiple-instance learning. Each trace is split into overlapping windows, and the Factor and context are repeated in every window. Training samples at most four windows per long trajectory. Early epochs take windows at the start, one-third, two-thirds and end of the trace. Later epochs take the two highest-scoring windows plus two under-visited ones. The trajectory score is the mean of the top two window probabilities. Gradients flow only through those two windows.

flowchart TD
  Q["Factor + context + CoT"] --> L{"CoT fits budget?"}
  L -- yes --> S["Single window"]
  L -- no --> W["Overlapping windows, Factor repeated"]
  S --> M["Shared monitor + linear head"]
  W --> M
  M --> A["Top-2 mean of window probs"]
  A --> B["BCE vs trajectory label"]

Training data has two branches. Factors are built from task metadata (context-anchored) or extracted from unlabelled traces and then de-specified (trajectory-induced). Every Factor is frozen before labelling. Labels come from final outputs when the criteria can be checked objectively there. Otherwise they come from reading the full trace. For example, a Sabotage step is flagged even when the attempt fails.

The data covers 11 factor types in three domains:

• Capability: reasoning method, skill use, instruction following.
• Safety: deception, sycophancy, reward hacking, sabotage, sandbagging.
• Trustworthiness: hallucination, privacy, jailbreak.

Evidence

The headline margins are 1.32 to 30.57 points over general-purpose LLMs at fine granularity and 11.71 to 27.81 points at coarse granularity. Figure 1 compares OwlSight per factor against two groups:

• General-purpose LLMs: DeepSeek-v4, GPT-5.6-Luna, Grok-4.6 and Qwen3.8-Flash.
• Guard models: QwenGuard-8B, LlamaGuard-4, ShieldGemma-9B and gpt-oss-safeguard.

The extracted figure text does not reliably map individual per-factor numbers to models. The paper also reports state-of-the-art results on OpenAI's CoT monitorability benchmark (Guan et al., 2025), but the fetched text gives no figure for it.

How to use it

Code is at an anonymized review link: https://anonymous.4open.science/r/papercode-5F78/. To use the monitor, write a Factor with explicit Applied and Not Applied criteria, then pass it with the task context and the CoT. The output is a probability, not a free-text verdict. Input formatting and optimization settings are in Appendix B, which is not part of the fetched text.

Limits

• The fetched text says only "small-scale LLM." It does not give the backbone model, parameter count, weights licence or training compute.
• The text cuts off partway through the method section. The robustness results (rephrasing, compression, obfuscation) and the unseen-model results are claimed but not shown.
• The monitor is a binary classifier per Factor, so it gives no explanation or rationale.
• Labels depend on the authors' annotation rules (Appendices A.3 and A.4, not fetched).
• The paper is under review, and the code link is anonymized rather than a permanent release.