GPT-6 Astra, Looped Transformers, and Hidden Reasoning

What it claims

Raschka makes two separate arguments. First, he thinks GPT-6 Astra probably uses some looped-transformer variant. The evidence is only The Information's unconfirmed report, and he credits training recipes and data, not looping, for most of the gain. Second, he argues looping is not what hides chain-of-thought. OpenAI has hidden most traces from users since o1. Astra's shorter traces fit the general pattern that more capable models backtrack less. OpenAI Chief Scientist Jakub Pachocki said Astra's computation-graph depth is "within a factor of two of GPT-4". He also said CoT monitorability is "trending in a negative direction, for reasons not contingent on architecture changes."

How it works

A looped transformer passes hidden states through the same blocks more than once, reusing the weights. For engineers, the cost accounting is the useful part:

• Parameters. Weights are saved. Nanbeige4.2-3B runs 22 blocks twice, giving 44 block applications with 22 blocks' worth of weights. Embedding and output layers are about 25% of its 3B parameters and are not affected by looping.
• Compute. Compute is not saved. Forward and backward passes cost about the same as 44 distinct blocks.
• KV cache. The KV cache is not saved either. Each pass sees different inputs, so it needs its own K/V entries. Nanbeige tried sharing the cache across passes: this halved its size but hurt quality, and the released model does not do it.

The variants differ in how they choose the loop count:

The "Latent Reasoning" recurrent-depth design (3.5B parameters, trained on 800B tokens) is drawn below. It samples the loop count randomly during training, and the operator sets a budget (e.g. 8, 32 or 64) at inference.

flowchart TD
  A[Embeddings] --> B[2 initial blocks]
  B --> C[Concat + linear projection]
  S[Previous loop state] --> C
  C --> D[4 shared blocks]
  D --> E{KL between successive<br/>next-token dists < threshold?}
  E -- no --> S
  E -- yes --> F[2 final blocks]
  F --> G[Next token]

Evidence

• Nanbeige: two loops was the best trade-off. Three loops gave small gains, but slowed training and made it less stable. Training from scratch beat upcycling a pretrained transformer.
• Mixture-of-Recursions: at the smallest scale (135M), the vanilla transformer wins. At larger sizes MoR catches up and often wins, especially at smaller training budgets. At the largest budget the curves converge. MoR also processes more tokens for the same compute.
• Latent Reasoning: HellaSwag plateaus after about 8 loops, while GSM8K and HumanEval keep improving with more.
• Beyond Parameters (June 2025): looping at a fixed parameter count leaves memorization capacity almost unchanged, but improves multi-step math.
• SMELT (Sept 2026): this MoE model loops the middle half of its blocks twice. It narrows the hidden dimension, adds experts back, and adjusts attention heads to match compute per token, non-embedding parameters and KV cache against a baseline. It scales to 54B non-embedding parameters. Fitted curves estimate 6.8–18% less training compute to reach the same validation loss.
• Full-bandwidth transformer (Aug 2026): a 1B base model feeds the previous token's final hidden state back in through a gate. It produced shorter MATH500 traces at equal or better accuracy, but the effect disappears after instruction tuning.
• Token efficiency: at fixed accuracy, Astra uses fewer output tokens than GPT-5.6 Sol, though not across every effort level. GPT-5.6 Luna uses about 80% more tokens than Sol at similar performance.
• Astra benchmarks: 99.9% on ARC-AGI-3, against 7.8% for GPT-5.6 Sol. On the independent Artificial Analysis indices, Astra is at the frontier but "doesn't pull ahead by leaps and bounds." Those use shared harnesses (Stirrup, Terminus 2, τ-Bench), which may understate Astra's performance in its primary harness.
• Computer-use training: the reported Mac Minis and Mac Studios serve as RL environments. The loop is: screenshot, model predicts mouse/keyboard actions, harness executes them, verifier gives reward. The model itself runs on NVIDIA GPUs. NVIDIA's CEO is quoted as saying about 100,000 Grace Blackwell GPUs were used.

Limits

• Astra's architecture is unconfirmed. Pachocki's quote is equally consistent with simply having twice as many ordinary blocks.
• Astra's system card does report a monitorability regression versus Sol, tied to shorter, less informative traces. Raschka argues this does not show looping is the cause, but he offers no evidence either way.
• The Full-bandwidth paper did not compare against conventional scaling (adding more blocks instead of looping), and it did not test whether its shorter traces are less faithful.
• Ouro's Hugging Face implementation computes all 4 passes before choosing an exit, so it gets no adaptive compute savings in practice.
• The Nanbeige paper gives few details.
• Practical aside: Raschka suggests pruning or regenerating old AGENTS.md and SKILL.md files, since newer models may do better with less hand-holding.