Agentic Ensemble with Vision-Language Models for Visual Anomaly Detection

What it claims

AnomalyAgent (Sun, Luo, He, Lee and Lu; NTU Singapore and Hyundai Motor Company; arXiv October 2026) fuses the outputs of fixed, pretrained anomaly detectors without labels or retraining. A vision-language model (VLM) is called only on samples where the detectors disagree. Even then, it can only pick one score operator and one map operator from a fixed menu. It cannot draw a mask or emit a score, so every final prediction is a deterministic transformation of detector outputs. The authors report gains over equal-weight averaging in all six MVTec AD and VisA benchmark–pool configurations. They also report transfer to seven more public benchmarks (including SDD/KSDD, MVTec 3D-AD and Kvasir-SEG) and score-level evaluation on Real-IAD.

How it works

1. Normalization. Each detector's image score and anomaly map is scaled by percentile-clipped min–max within an evaluation group (object category or benchmark subset). This is transductive: it uses the test-set outputs and group metadata, not labels.
2. Evidence state. Per image, it computes a pixel disagreement map R (max minus min across detectors). It also computes a top-q mean T_q that resists isolated spikes, and candidate high-response regions.
3. Conflict score. A weighted sum of four cues: the std of detector scores, T_q of R, and the deviation of a preliminary fallback fusion from equal-weight fusion at both map and score level. The cues measure ensemble inconsistency, not anomaly likelihood. A confident anomaly the detectors agree on skips the VLM.
4. VLM selection. Above a fixed threshold τ, the VLM sees the image, the per-detector maps, the disagreement and region overlays, plus a text summary. It returns action identifiers and an optional rationale.

flowchart TD
  A[Detector scores + maps] --> B[Group-wise normalization]
  B --> C[Evidence state + conflict score]
  C -->|below threshold| D[Fallback fusion]
  C -->|above threshold| E[VLM picks score + map action]
  E -->|invalid output| D
  E -->|valid IDs| F[Apply chosen operators]
  D --> G[Final score and map]
  F --> G

The score actions are mean, median, weighted sum, and score–map consistency, which is (1−α)Σw_k s_k + α·T_q(M). The map actions are mean, median, weighted sum, and geometric mean exp(mean log(M_k+ε)). A further map action is disagreement correction, clip(M ± ηR, 0, 1). Weights are nonnegative and sum to one. The parser accepts only the predefined IDs. Anything else reverts to the fallback action.

How to use it

Two inputs are required: detector outputs that include dense maps, and group metadata for the normalization step. Each high-conflict image costs one VLM call. Hyperparameters to set are the four λ weights, τ, q, α, η, ε, and the fallback actions.

Limits

The fetched text breaks off in the method section. It contains no results tables, so this document gives no AUROC or AP figures, no margins over baselines, and no list of compared fusion methods. It also does not state which VLM is used, which detectors form the pools, the τ and λ values, the fraction of samples that reach the VLM, inference cost, or a code release. Because normalization is per group over the test set, it assumes batch access and category labels at inference. Streaming single-image use is not addressed.