Can You Check That? The Checkability Boundary for Local LLM Network Automation

What it claims

Touchstone is a local-first pipeline for network automation. Small local models are trusted only when an intrinsic check can reject wrong answers. An intrinsic check is a deterministic predicate Q(x, y, s) over the input, the candidate output and optional network state. It is a necessary condition for correctness, never a sufficient one. A failed check proves the answer wrong. A passed check proves nothing on its own, so each check must be characterised by its measured false-accept rate (FAR). Checks use no ground truth, no LLM, no learned verifier and no remote API.

On conflict detection, the pipeline reports 98.6% end-to-end accuracy with 16% escalation and zero false accepts.

How it works

Seven off-the-shelf SLMs run locally: Llama-1B, Qwen-1.5B, SmolLM-1.7B, Qwen-3B, Llama-3B, Qwen-7B and Llama-8B.

• Offline profiling. Each model is scored on a dev split of 100–150 examples, which is 23.1–25.0% of each task's data. Its vote weight is w_m = max(0.02, (acc_m − chance)^1.5). Chance is 0.5 for conflict detection and 0 for generation tasks, so near-random models get almost no weight.
• Aggregation. The method depends on the task:
• Conflict detection and log parsing: a skill-weighted vote.
• Intent translation: specs are split into rule "atoms" such as ("reach", "paris", "100.0.1.0/24"). An atom is kept if its supporters hold at least half the total weight. The assembled spec is then compared against the best single-model spec, and the higher grounding-and-coverage score wins. This guards against incoherent recombination.
• Routing code: no vote. Every function that passes execution is retained.
• Escalation. Confidence starts as the weighted family agreement and is adjusted by the check. The input escalates to a frontier LLM if conf(x) < τ = 0.6.

flowchart LR
  X[Input] --> S[7 local SLMs]
  S --> A[Weighted aggregation]
  C[Intrinsic check] --> A
  A --> G{conf < 0.6?}
  G -- no --> L[Return local answer]
  G -- yes --> F[Frontier LLM]

The checks per task:

Checks are written once per task by the system designer. Before deployment, each is evaluated on the dev set for FAR and for unnecessary-escalation rate, meaning correct candidates that were rejected.

Evidence

The baseline sends every input to the frontier LLM. The page does not name the frontier model.

Log parsing and routing code are among the evaluated tasks, but their numbers are not in the fetched text.

How to use it

To apply the pattern, find necessary conditions that any valid output must satisfy. Examples: no prefixes absent from the request, waypoints preserved, the template regenerates the line. Write each as a small deterministic refuter. Then measure FAR and unnecessary escalation on a held-out set before trusting it.

The page gives no code release, licence or install instructions.

Limits

• Necessary, not sufficient. A grounded, complete-looking intent spec can still misread the operator's intent.
• Weak log-parsing check. The authors flag the invertibility check as having a larger FAR, because over-general templates still regenerate the line.
• Execution is not new. Execution checking for code is already standard practice. The authors include it only as the strongest example.
• No-check tasks don't benefit. For TeleQnA-style tasks, model diversity plus escalation does not reach frontier accuracy.
• Untuned threshold. τ = 0.6 is fixed; the text does not report sensitivity to it.
• Small dev sets. Profiling uses 100–150 examples per task.
• Truncated text. The fetched page cuts off in Section 3, so the per-task FAR measurements (§4.1), the log-parsing and routing results, and any limitations section were not available.