Qwen3.8 27B addition in words

What it claims

With reasoning enabled, a locally run Qwen3.8-27B-Q4_K_M.gguf answered 167 of 169 "add these two numbers and give the sum in words" prompts correctly. Each pair was tried only once. The task comes from an experiment Colin Frasier ran on GPT-4o more than two years earlier. In that run, GPT-4o got many sums wrong as the numbers grew. Willison treats those errors as evidence that GPT-4o did not use a calculator.

How it works

• Willison pasted Frasier's results chart into a Codex Remote session running GPT-6 Astra. The agent rebuilt the experiment against the Qwen model on a DGX Spark. The harness was generated by the agent and is not published as code on the page.
• Reasoning disabled: 30 attempts per number-size combination, shown as an accuracy heatmap.
• Reasoning enabled: one attempt per combination, because each pair took much longer. Each heatmap cell is therefore either 100% or 0%.
• The published reasoning traces show the model doing grade-school column addition. It aligns the digits, then works right to left with explicit carries, for example "6 + 9 = 15, write 5, carry 1". It also corrects itself ("Wait, let me redo this more carefully"). The quoted example adds two 13-digit numbers: 4,299,366,105,622 + 6,088,794,067,970.

Evidence

• The only number stated in the text is the reasoning-enabled result: 167/169, one sample per pair.
• The reasoning-disabled accuracies exist only in the heatmap image. The page text gives no figures for them, so you cannot directly compare the two modes from the text.

Limits

• The reasoning-enabled result rests on a single sample per cell. Willison says a second run would likely give different results, so 167/169 is not a stable accuracy estimate.
• The page gives no timing or token cost for reasoning mode beyond "a lot longer per pair".
• Only one quantisation (Q4_K_M) on one machine was tested. There is no unquantised or other-quant comparison.
• The page does not list the exact number ranges or prompt wording; you would have to read them off the charts or the linked report.
• The GPT-4o baseline is Frasier's chart from two years ago, not a rerun under the same harness, so the two models are not compared head to head.