Here’s a Way to Predict When AI Chatbots Will Turn Bad

What it claims

Neil Johnson and Frank Yingjie Huo (George Washington University) give a formula for n, the number of good tokens a model emits before its first bad one. Two outcomes follow from it. If accumulated context already leans toward the bad cluster, the model tips immediately and n = 0. If it leans good, the model produces a run of acceptable tokens and then flips. The authors frame this as the same mechanism jailbreakers exploit through long conversations and that system prompts try to counter.

How it works

The page gives only a high-level account:

• Mechanism: the attention head is pulled toward one answer cluster or another as context grows.
• Earlier version: a prior paper by the same pair (covered April 2025) modelled a single, deliberately simplified attention head. That paper found polite words like "please" and "thank you" are orthogonal to a request's substance and so have negligible effect on output.
• Proposed monitor: a low-cost process runs in parallel with the model and raises a warning when n* falls below a safety threshold.
• Mitigation: inject content into the conversation so that n* lands beyond the length of the response, meaning the tipping point is never reached within that output.

flowchart TD
  C[Accumulated context] --> L{Leans which way?}
  L -->|bad| Z["n* = 0: tips immediately"]
  L -->|good| D["Run of good tokens, then flips"]
  D --> M{Monitor: n* below threshold?}
  M -->|yes| W[Flag / inject content to push n* past response length]
  M -->|no| OK[Continue]

Evidence

• Preprint result: the formula classified immediate vs delayed tipping correctly in 15 of 16 clear-cut cases (94%).
• Models tested: six open-weight models from OpenAI, EleutherAI and Meta, between 124M and 410M parameters.
• Settings: a 300-token window. No baseline predictor is reported.
• Published paper: reportedly extends testing to seven models of up to 12B parameters. The page gives no results for that extension.

Limits

• Precision: predictions can be off by one output.
• Scope: only small models and a 300-token window were tested. "Clear-cut" cases were selected, so behaviour on ambiguous prompts is unreported.
• Alignment training: the authors say it can shift or suppress tipping for specific prompts but cannot remove the underlying mechanism. Do not assume a safety-tuned model is exempt.
• Missing details: the page gives no formula, code, monitor cost figure or threshold value. To implement the monitor you need the paper in Patterns.
• Source: this is secondary news coverage. The 12B-model extension is "reportedly" and unverified here.