[AINews] Behind Latent Space: Kill My SaaS, Jev & AI in 2026 | AIEi Shanghai

What it claims

This AINews issue covers 2 to 3 November 2026. The editors say the previous day's "Weight-Space Programming" lead story overstated some claims. They do not retract it, because several groups have independently replicated it. The central result is the Qwen3.5-0.8B edit by GPT-6 Astra. That edit repaired a looping failure and did not add new math ability. Beyond that story, the issue reports quantization, sparse-attention and serving releases, along with benchmark audits and correctness fixes that affect running systems.

How it works

A Zhihu reconstruction of the Astra logs describes the edit. Astra built a "repetition direction" from contrasting looping and productive contexts. It found that direction mainly in the MLPs of layers 15 and 19. It then subtracted 6× the projection from the top 64 aligned neurons in each layer, which changed 131,072 weights (0.016%). Astra ran through a Python kernel over the weights with no GPU and no backprop. It used GSM8K and MATH training batches as practice data and, by the authors' account, did not use the eval sets.

Evidence

• Of 579 repaired MATH-500 answers, 576 had originally hit the 32K-token limit by looping. On 1,152 held-out problems, accuracy rose from 10.6% to 60.4%.
• With 50 forward passes, GSM8K rose from 13.8% to 67.1% and IFEval from 21.0% to 62.3%. MATH-500 answers that hit the 32,768-token cap fell from 93% to 33.1%.
• The same method failed on Gemma 3 270M, which has no looping failure.
• A Reddit user had Claude Opus 5.5 edit 8 MLP neurons (about 92K parameters) in Gemma 4 12B to change how it translates one Italian tense. On held-out tests, 182 of 200 unrelated translations were unchanged. The user reports better locality than prompt steering or ablation, and better preservation than LoRA.
• A separate replication on a small transformer gained only 75.69% to 76.03% in recall. Most weight-only edits caused regressions.
• ISTA DASLab's DeepSeek-V4.1-Flash-1017 GSQ-RCO Coder keeps 192 of 384 experts per layer and stores 72.1 GB, against 138.2 GB for the IQ2_XXS build. It scores 88.57 on LiveCodeBench v6 (base 89.71) and 77.00 on SWE-bench Verified (base 81.20). Its "1-bit" label counts pruned experts as zero bits; the retained weights average 2.06 bpw.
• Latent Sparse Attention (arXiv:2610.42147) adds a sparse-attention indexer to pretrained models; only the indexer is trained, by distillation. It sparsifies Qwen-3.8-27B with no reported accuracy loss and runs attention-layer prefill 72.4× faster than full attention at 10M tokens on an H200. No VRAM figures are given.
• In Amp, renaming a tool argument from format to kind raised correct tool choices from 53% to 79%.

How to use it

• DeepSpeed ZeRO-1/2 has a correctness bug. Upgrade to version 0.19.9 or later.
• vLLM v0.33.0 requires PyTorch 2.14 and ships CUDA 13-only wheels. It reports 1.5 to 2× decode gains on GB300.
• The GSQ-RCO GGUF needs llama.cpp b11870 or later, or Strata.
• LSA code, kernels and SGLang integration are in the Prefrontal-AI/latent-sparse-attention repository.
• Delayed "time-bomb" loss spikes come from quantization error in the BF16 LM head. Z-loss, μ-centering and LM-head SGD-M all prevent them by keeping that error bounded. Fixing Softmax Collapse only delays the spike.
• Xeophon's audit found three benchmark problems. AutomationBench forces prose request bodies. OfficeQA Pro v2 regex graders reject valid answers. Models with web access can pull answer keys from Hugging Face. With fixes, both benchmarks approach saturation. Removing the images from SWE-bench Multimodal barely changes scores.

Limits

The weight-space results are author-reported, blog-level and not peer reviewed. Critics, named by handle, raise several objections:

• paul_cal says the work lacks competent baselines.
• dbreunig says fitting weights to a benchmark is "essentially meaningless."
• ravfogel asks whether models mainly read the first and last layers.
• A Reddit commenter argues the 7.9% baseline reflects a pathological sampling setup. The same commenter says the jump from 25 to 50 passes looks like variance, not a scaling curve.

The "Vals studies" and "Data-constrained scaling" sections are empty. Despite the title, the page carries no AI Engineer Shanghai content and no Latent Space interview.