Don't be fooled–LLMs don't reason

What it claims

This is an opinion essay, not a paper or a system release. Graepel argues that chain-of-thought adds no reasoning mechanism separate from the network: "the same next-token prediction process, iterated for longer". He says scaling cannot fix this: "Scale sharpens intuition, but it does not make intuition more deliberative."

His evidence from AlphaGo is concrete. The policy network gave Move 37 roughly a 1-in-10,000 chance of being played by an expert human. Search over a game tree with thousands of branches chose it anyway. He says he left Google DeepMind to pursue the alternative described below.

How it works

The essay goes further than "evidence-backed moves" in several ways.

• What the epistemic state holds. It is the analogue of AlphaGo's game tree, where each position is annotated with network judgments and later synthesized into a decision. Here it records what is settled, what is doubted, what is ruled out, and which questions remain open.
• Moves. A move can be any of these:
• deducing consequences
• decomposing a problem
• choosing the next question to ask, calculation to perform or experiment to run (he calls this the crucial one)
• Role of LLMs. They are components, not the reasoner. They propose approaches given what is known and what resources are available. They call tools via APIs or code, and they help judge whether a claim is supported by evidence.
• Honesty mechanism. An independent part of the system scores each move by how much uncertainty it actually resolves. Beliefs are updated only when the evidence backs the change. With those rules enforced, the system builds up "certified knowledge" and improves its reasoning policy from past reasoning episodes.

flowchart TD
    S[Epistemic state] --> P[LLM proposes move<br/>deduce / decompose / query / experiment]
    P --> T[Tools, code, evidence]
    T --> E{Independent evaluator:<br/>uncertainty resolved,<br/>evidence-backed?}
    E -- yes --> U[Update beliefs,<br/>certified knowledge]
    E -- no --> R[Reject update]
    U --> S
    R --> S
    U --> L[Learn reasoning policy<br/>from past episodes]
    L --> P

Limits

• No implementation or results. The essay describes no system, benchmarks, or experiments for the proposal. It is a research direction only.
• Open-world difficulty. The author concedes that open-world reasoning is harder than Go in three ways:
• the state is only partially known
• the set of actions is large and changes
• the consequences of actions are stochastic or unknown
• No method for the evaluator. The essay does not say how the independent evaluator would measure "uncertainty resolved" or decide what counts as evidence.
• Unnamed source on unfaithful chains. The claim that chains of thought are often made up after the fact rests on "research has demonstrated", and no study is named.
• Concession on chain-of-thought. He grants that its gains are real, especially in mathematics and coding.