Two Conformal Constructions for Adaptive Within-Document AI-Text Screening

What it claims

The paper gives two ways to keep a document-level false-alert rate at or below α when a screener inspects growing prefixes of one document, switches between detectors (for example DetectGPT, Fast-DetectGPT or Binoculars scores), and may stop early. The concrete deliverable is the calibration count each construction needs before it can reject at all:

These counts only make rejection possible on the rank grid. They say nothing about power. The authors do not claim to originate conformal AI-text detection or sequential-testing theory. Their contribution is specifying these two constructions for selecting detectors within one document.

How it works

Construction A (weighted Bonferroni over conformal ranks):

1. Using development data only, fix a finite family of (token budget, detector) actions and weights w_a with Σw_a ≤ 1.
2. Each action gets a conservative conformal p-value against m human documents, using rank (1 + #ties-or-greater)/(m+1). An action rejects if p_a ≤ α·w_a.
3. At deployment, which actions run and in what order may depend on observed scores or ranks.
4. Requirement: m ≥ ⌈1/(α·w_a)⌉ − 1. An action with w_a = 0 can never reject.

Construction B (one rank on the complete-path maximum):

1. Using development data only, freeze an adaptive route generator π and per-action transformations g_a.
2. For each calibration document, run the complete route with alert-stopping disabled and record the maximum transformed score.
3. At test time, compare the running partial maximum against those complete maxima. Because partial max ≤ complete max, every partial rank is at least the terminal rank, so stopping early cannot inflate error.
4. Requirement: m ≥ ⌈1/α⌉ − 1.

flowchart TD
  A[Next action from frozen route] --> S[Score, update running max]
  S --> R{Rank vs complete calibration maxima ≤ α?}
  R -- yes --> F[Flag for review]
  R -- no --> C{Budget / futility stop?}
  C -- no --> A
  C -- yes --> N[No alert at this budget]

Failure handling applies to both constructions. A failed action gets −∞ before any transform, which yields p = 1. If a tail-rank transform maps a failure to 0 instead, then with 99 empty calibration routes and one failed test action the rank comes out as 1/100 rather than 1. That is a spurious alert.

Choosing between them

• A lets action selection depend on calibration ranks. Its costs are a large m and weights that must be fixed in advance.
• B needs about 12× fewer calibration documents in the example above, but has three costs:
• Every calibration document must be run through its complete route, which costs offline compute.
• The route generator must never see calibration documents or summaries of them.
• A broader route can raise the calibrated maximum and cost power.

If a policy is tuned on the calibration set, it needs fresh calibration data. Calibrating against matching prefix maxima instead of complete maxima breaks the proof.

Limits

• The guarantee is marginal. It is not conditional on the realized calibration set, document length, subgroup or chosen route, and it does not control multiplicity across many screened documents.
• Weights must not be chosen after seeing ranks. With two independent null p-values, putting all the weight on the smaller one gives error 1 − (1−α)² > α.
• Prefixes of the same document cannot serve as separate calibration units.
• An alert means "differs from the human reference distribution". It does not establish AI authorship or misconduct. No alert does not certify human authorship.
• The scope is passive detection of unwatermarked text. Neither proof builds an e-process or justifies multiplying conformal ranks.
• There are no detector experiments, no power results and no measurements of compute savings.
• The oracle Hoeffding budget, n ≥ (2/Δ²)·max(log 1/α, log 1/β), assumes independent bounded observations with known means. The authors stress that tokens within a document do not count as independent replicates.
• The fetched text cuts off partway through Section 6, so the distribution-shift and independent-audit bounds were not seen.