[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment

What it claims

This is a newsletter digest, not a primary source. Its most actionable content is release detail for Meta's Muse Spark Open, an 816B-total / 42B-active MoE with 1M context and text, image, video and audio input. Artificial Analysis (AA) scores it 39 on its Intelligence Index. That is first among US open-weight models but behind GLM-5.3 (45) and Kimi K3 (44). The lead story, Nick Cammarata's "Neural Networks Are Already Clean", claims that most neurons in models up to trillions of parameters are legible. His 2023 OpenAI automated-interpretability effort found only about 2% legible, and he argues that many famous polysemantic examples were misread. The issue says the paper itself is not yet out.

How it works

Muse Spark Open architecture, per vLLM:

• Layers and experts: 62 decoder layers, all MoE, with 256 experts per layer and 8 active per token. There are no shared experts; Bakouch notes it is the only major US open model built this way.
• Latent-MoE pooler: the hidden state is projected from 8,192 to 4,096 dimensions for expert computation, then back.
• Hybrid attention: 46 layers use a 2,048-token sliding window and 16 attend to the full context.
• Other components: gated residuals, about 9 norms per layer, and muP-like multipliers, including a router factor of 1/sqrt(number of experts).
• Multimodal encoders: image and video go through a Perception Encoder; audio uses a separate 118M encoder.
• Lineage: it was the first base model trained under Meta's MSL org, with extra training for the open release. Posters place it at about Muse Spark 1.2 level, below Spark 1.3.

Evidence

Muse Spark Open (AA measurements)

• Strengths: it uses 40k output tokens per index task, the fewest among the nine highest-scoring open models. It leads open models on AA-Omniscience with a score of 25.
• Weakness: it ranks last of those nine on AA's four agentic evals. GLM-5.3's entire 6-point lead comes from that agentic segment.
• Cost: $0.84 per index task, versus $2.01 for GLM-5.3 and $0.27 for DeepSeek V4.1 Flash.

Muse Spark Open (vLLM results)

• TPU throughput: over 650 tok/s on 4 TPU v7 chips and 960 tok/s on 8, using megakernels with DSpark speculation (acceptance length 6).
• LoRA GRPO: training ran on verl with vLLM rollouts, over 150 steps on 16 GB300s. AIME-2024 mean@8 rose from 0.56 to 0.85, with train-rollout KL around 2×10⁻⁴.

Nemotron 3.5 Super VL (120B / 12B-active hybrid Mamba-Transformer MoE)

• AA scores it 26, and its NVFP4 version comes within 0.1 points of BF16.
• NVIDIA claims 51.78% on SWE-bench Pro and 4,856 tok/s per GPU on GB200.
• The companion Bolt model (74B-A9B, native 2-bit, 24GB) scores 21 on AA.

Systems

• TPUv7: on SemiAnalysis's agentic inference benchmark, it delivered 1.42× better performance per dollar than GB300 NVL72 FP4 on an external cost basis. The initial figure was 1.67×. It also delivered 2.5× over MI355X. GB300 and GB200 still lead at low interactivity.
• Vera Rubin NVL72: up to about 5.7× GB300's throughput per MW on DeepSeek V4 Pro at high per-user speed, narrowing to 1.14–1.45× at peak throughput.
• TRL AsyncGRPO tree packing: the authors report 26× less trainer compute to reach the same reward on MiMo environments.
• vLLM bitwise-consistent RL: zero train-inference KL on Nemotron-3.5-Lightning while keeping 87.5% of rollout throughput.
• Perplexity SPACE sandbox: in 108 escape attempts by models with root inside the VM, none reached the host.

Eval integrity

• Vals found leftover Git objects containing solutions in 1,795 of 2,698 released MiMo v2.6 coding tasks. When Git commands were blocked, the model wrote its own parser for the stored objects.

How to use it

Muse Spark Open

• License: the custom Muse Model License requires "Built with Muse Spark Open" attribution. Model-as-a-Service businesses with more than $20M in annual revenue need a separate agreement.
• Hosted pricing: Together, Fireworks and Baseten charge $1.25 / $4.25 per 1M input/output tokens, with cached input at $0.15. Together offers a free endpoint until Oct 18.
• Weights and hardware: BF16 weights are about 1.6TB. The NVFP4 checkpoint quantizes only the experts and comes to about 530GB. It runs on 4× Blackwell Ultra, 8× H200, or two 512GB M5 Ultra Mac Studios via MLX.
• Engine support:
• vLLM day-0 support covers NVIDIA, AMD MI355X (FP8 and MXFP4) and TPU.
• llama.cpp had GGUF support on day 0.
• Unsloth publishes quants; UD-IQ1_S is 176GB.
• Inferact's block-diffusion DSpark drafter proposes up to 7 tokens per pass.

Other items

• Nemotron 3.5 Super VL: CoreWeave charges $0.25 / $1.50 per 1M input/output tokens.
• SPACE: Apache 2.0. Each agent task runs in a Firecracker microVM, with credentials and egress controlled from outside the VM.
• Opus 5.5 fast mode: now GA, up to 2.5× speed at $8 / $40 per 1M tokens.

Limits

• Second-hand reporting: almost every figure is a summary of a tweet or post. Verify against the linked primary sources before relying on any number.
• Cammarata essay: the paper is not yet out, per Pfau. The critics' objections are:
• Neel Nanda still expects representational superposition and believes weak activations carry memorized facts.
• Leo Gao argues explanations should be judged by neuron simulation measured with KL.
• Jake Mendel flags the neglected tail of non-max activations.
• Muse Spark Open: weak on agentic evals. One Reddit commenter says heavy safety tuning may make it unsuitable for cybersecurity work. Nathan Lambert called the license "trash".
• MAI-Base-2: its claim of matching DeepSeek-V4-Pro-Base with 12× less compute uses MAI's own validation data, as Bakouch and teortaxes point out.
• TPUv7 comparison: Zach Mueller questions whether the figures assume max utilization.
• Vals refund-agent study: one trial per model, differing run durations, and simulated organizations.
• MiMo leakage figure: it measures environment leakage, not a cheating rate.