arxiv.orgpaper

Graph Machine: Towards Better Pretraining via Edges

The Graph Machine (GM) is a neural architecture that maintains an O(n)-sized state accessed via sparse, dynamic routing using differentiable "edges" — pointer-like objects updated through a pointer-chasing-like referral mechanism. The authors replace 75% of dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens, showing that retrieving only 2–4 of 4,096 tokens per KV head per sparse layer preserves or marginally improves loss compared to the dense baseline. This suggests a path toward more efficient pretraining by reducing dense attention computation while retaining large effective context/state.