Six defects in one signature verification tool, found from outside in six rounds

What it is

A reproducible, round-by-round record of six specific defects found in nsgoods Workbench's verify_signature tool (offline signature verification over the x402 payment catalogue, operated by Nikolaos Dimitriadis), each tied to a dated test run between 18-22 September.

How it works

The method: write predictions to a file before each batch of calls, so results can't be graded after the fact. Six rounds, each probing a different failure mode:

• Run 1 (Sep 18): verify_signature returned valid: false on the operator's own correctly-signed preview. Offline recomputation using the manifest's stated recipe recovered the correct signer — the tool was stripping a fixed field list that included fields this service actually signs.
• Run 2 (Sep 19): That false negative was fixed, but an injected field from a different service (same signer) still verified as valid. Separately, submitted reproduction attestations returned a generic "failed verification" rather than "unsupported format" — indistinguishable from a real failure to an outside observer.
• Run 3 (Sep 19): A new service parameter, meant to remove guesswork, let the caller choose which fields were excluded from the signed message — so a tampered body verified if you named the right service. Predicted at ~60% likelihood; reproduced 2/2.
• Run 4 (Sep 20): Closed via a schema gate, confirmed across a full wrong-service matrix (30/30 off-diagonal cells correctly refused, including 5 that would have verified on signature alone). But schema-rejection and signature-failure returned byte-identical error text, so callers can't distinguish which check failed.
• Run 5 (Sep 22): Signer identity anchored to a public registry record; the shared refusal text from Run 4 persisted.
• Run 6 (Sep 22): Distinct status codes added, with an invariant that checked is true only if recovery actually ran. Tested across 106 calls; one exception found — non-object JSON input returns a status with no checked key at all (rather than false).

A separate disclosure (Sep 25) surfaced a logging/privacy issue: the operator's log had stored the first 80 characters of string arguments, session IDs, and caller IPs (Sep 17–26), contradicting the stated privacy page. Fixed same day: logging switched to argument names/lengths plus a daily-keyed IP hash, old-format records deleted by Oct 26, and a redaction rule extended to strip whole query strings containing wallet addresses (applied retroactively across 51 files, verified against all 108 log files).

Preconditions / Affected

Applies to anyone relying on nsgoods Workbench's verify_signature output, or more generally any boolean-returning verification/attestation service where callers trust the verdict without having observed it fail. The author notes four analogous defects found independently in their own repository via the same methodology.

What to do

No commands given. The actionable item is the methodology itself: before testing a verifier, write down predictions for what should fail; treat "no reported error" as non-evidence; and build regression tests that capture an actually-reproduced failing run (not a fabricated one) alongside the passing run — which the companion tool exactzk-verification-guards does. The author explicitly asks whether others have run predictions-first testing against attestation/signature tooling (e.g., ERC-8004) and invites comparable findings.

Caveats

Everything about the tool's behavior was independently verified by the author. The logging/privacy remediation is the operator's own account of his own systems — the author states explicitly there is no external way to verify it, and credits it mainly because the operator volunteered two details the author couldn't have discovered independently.