Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

What it claims

Olmo-core 3 is Allen AI's open MoE training stack. Its central claim is that it keeps sparse models cheap as the expert pool grows. In one benchmark, experts went from 8 to 128 with top-4 routing, so active parameters per token stayed at about 3.2B. Total parameters grew from 4.6B to 47B, and training throughput fell by less than 5%. Against Allen AI's own earlier FSDP-based MoE implementation, the new stack is about 2.7× faster on the same 47B model. The same infrastructure has been benchmarked at over one trillion total parameters.

How it works

• No more per-batch weight gathering. The old implementation used FSDP, which gathered and resharded weights for every micro-batch. The new one uses DDP: experts stay resident on their GPUs and tokens are routed to them.
• Three ways of splitting the model. Expert parallelism shards the expert pool across GPUs. Pipeline parallelism splits the layers across GPU groups. A distributed optimizer shards the optimizer state.
• Routing and compute optimizations.
• Rowwise expert parallelism writes routed tokens directly into expert input buffers, avoiding a separate rearrangement step.
• GPU-resident routing metadata means the CPU can queue work without waiting for a device-to-host copy.
• Grouped GEMM batches many small expert matmuls together.
• MXFP8. Lower precision is applied selectively, only in the parts of the system where the savings outweigh the cost of converting formats.
• DeepEP v2. An alternative expert-communication backend, tried experimentally.

The page names Megatron-Core as the established alternative, but gives no head-to-head comparison with it.

Evidence

Most of the MXFP8 gain came from feed-forward compute and inter-expert communication, not from attention.

Limits

• Systems throughput only, not model quality. The trillion-scale runs used random routing.
• The 2.38T result is not sustained training. It was a short-capacity test.
• The MXFP8 numbers assume a perfectly balanced load. Work was spread uniformly across experts, so real routing imbalance may reduce the gain.
• The 2.7× figure is preliminary, comes from eight GPUs, and is measured against Allen AI's own old stack rather than Megatron-Core.
• Only NVIDIA B300 hardware is reported.

Pitfalls the authors report, which matter if you build on this stack:

• Token gerrymandering: the load-balancing score can improve while the actual per-expert workload becomes less balanced. Check the real per-expert token counts, not just the auxiliary metric.
• Lowering expert learning rates (on the reasoning that experts see fewer tokens) did not help in the model family they tested.
• GPU kernel timing depends on the input values, not just on matrix shapes. Benchmarks need matching inputs to be comparable.
• Overlapping communication and computation on separate streams sometimes slowed end-to-end training.

How to use it

The post links to a GitHub repository and a technical report but gives no install commands, configuration examples, licence, or minimum hardware requirements. Those have to come from the repo itself.