Understanding the Impact of LLM Watermarking on AI Agent Behavior

What it claims

Watermarking a model with SynthID-Text is not behaviorally free even when it is "non-distortionary": the guarantee holds only in expectation over the watermark randomness, while any fixed key changes token selection during generation. Because agents act on the same tokens the watermark biases, this "sampling drift" changes both which tool gets called (and with what arguments) and whether a model refuses harmful requests. The effect is model- and key-dependent, and net accuracy/refusal scores hide it because opposite-direction changes cancel.

How it works

Paired design: each item generated with and without SynthID from the same seed, batch composition, and order at each temperature — the watermark processor is the only difference in a pair. Config: HuggingFace's unmodified SynthIDTextWatermarkLogitsProcessor, 30 Tournament layers, n-gram length 5, sampling table 2^16, context history 1,024. Tool calling on BFCL v4 single-turn AST; refusal on 200 HarmBench harmful behaviors + 100 benign JailbreakBench controls, tested bare and under one fixed prompt-injection technique (adversarial "safety filter disabled" instruction appended as retrieved content). The reported metric is churn = share of items whose pass/fail verdict flips between the two runs, reported alongside net change.

Evidence

• Tool calling: accuracy drops on 6 of 7 models (significant on 4). Churn dwarfs net change: phi-4 at T=1.0 shows 16.8% verdict churn vs 2.87-point net loss; Llama-3.1-8B 9.9% churn vs 0.87-point net loss. Across 21 model-temperature combos, churn averages 6.5%, bootstrap interval excludes zero in every case.
• Error profiles differ: Llama-3.1-8B loss dominated by wrong arguments (−3.48) then wrong tool (−1.84); phi-4 and Granite-3.2-8B dominated by malformed output (−5.96, −4.36). Similar aggregate loss, different failure modes.
• Refusal under injection: gemma-3-27b churn rises 6.0% → 23.5% and net compliance −1.0 → +12.5 points; gemma-3-12b 7.5% → 11.0% and −0.5 → +9.0. Watermark-induced churn under injection exceeds temperature-induced churn (T=0.001→0.7) on 4 of 6 models; Granite watermark churn 21.5% vs 15.5% temperature.
• Key sensitivity (T=0.7, study key + 10 more): for Llama-3.1-8B study key +3.5 attack-success points, other ten average +4.4, range −4.5 to +14.5. Keys can move Granite in both directions.

Limits

• Measures model-level refusal, not end-to-end agent behavior; the combined injection→compliance→tool-action failure mode is argued but not tested directly.
• phi-4 and Qwen3-4B barely move under any condition — but they over-refuse even on benign controls, so their stability is not evidence watermarking is safe on them.
• Prompt injection is a single fixed technique. Refusal correctness judged by the study's own labeling.
• Practical caveat the authors stress: where the provider controls the key/config (e.g. Anthropic applying watermarking at the model level across API and cloud), drift can change outside the agent developer's control. Re-run red-teaming under the deployed watermark config, with paired comparisons and under injection.