LLMs are not superhuman at reading and writing neural network weights

What it claims

Sean Goedecke argues that a recent paper overstates GPT-6 Astra's weight-level abilities. The paper calls them "well past what we'd expect from humans or conventional algorithms." Goedecke holds that a competent ML researcher could do each task. He says the only superhuman part is mental arithmetic: Astra computed weight norms for the toy networks without running Python. His forecast is that LLMs will help AI research by improving training, not by editing weights directly.

How it works

The game-identification task uses a small network of about 17k weights. Its labeled input channels (for example "agent" and "moving objects") each read one grid cell. Taking the L2 norm of each input across the 32 first-layer units gives a per-channel spatial heatmap. Goedecke reproduces it in three lines:

d = np.load("a9f535cc.npz")
W1 = d["actor/dense1/kernel"]
norms = np.linalg.norm(W1.reshape(10,10,5,32).transpose(2,0,1,3), axis=-1)  # 5 x 10 x 10

He says the heatmaps give away at least the first three games by eye. In Breakout the player sits in a horizontal line at the bottom. In Pac-Man the moving objects trace the maze.

For the Qwen 0.8B fix, Astra had Python access. It followed what Goedecke calls the Heretic abliteration playbook:

1. Write four pairs of reasoning texts, one looping and one not looping.
2. Take the difference between the looping and non-looping residuals.
3. Measure how much each weight contributes to that difference.
4. Trim the top weights and retest until looping stops.

Evidence

The "improved at mathematics" result is narrower than it sounds. The original Qwen model solved every problem that needed fewer than 32k tokens. It failed only on longer problems, because it fell into loops after 32k tokens. The patch removed the looping, so the model could work past 32k tokens. The post does not report benchmark scores before or after the patch.

Limits

This is a rebuttal blog post, not a replication. The heatmap code is the only result Goedecke reproduces himself. He does not name the paper's authors or link its benchmark numbers. He covers only two of the paper's examples. He skips Astra hand-writing a neural network from scratch, arguing that any task expressible as a simple program can be expressed as a network, even by humans. The prose says the inputs read an 8x8 grid, but the code reshapes them to 10x10 with 5 channels, so check the grid size against the paper.