Skip to content

Back to the catalog

whitecircle/halo
github.comrepo

whitecircle/halo

Halo is White Circle's open-source training framework that runs the same codebase from a single GPU to multi-node clusters, adding Expert/Context/Tensor/Expert-Tensor parallelism (EP/CP/TP/ETP) directly onto native Hugging Face models via FSDP2, DTensor and DeviceMesh rather than a separate distributed model implementation. It covers the full training lifecycle — pre-training, SFT, preference optimization, distillation, and fully asynchronous multi-turn RL with a Transformers-side trainer pushing weights over NCCL to vLLM/SGLang rollout servers — and reports up to 2.8x the throughput of stock TRL for gpt-oss-20b on 8x B300 GPUs. It ships as prebuilt Docker images for Blackwell and Hopper GPUs with FlashAttention-4, DeepEP V2, DeepGEMM, and a bf16 AdamW optimizer integrated, supports 15 MoE families, and includes LoRA/QLoRA paths that fit consumer GPUs.

TL;DR White Circle's open-source LLM training framework: HuggingFace-native pre-training/SFT/RL with EP/CP/TP/ETP parallelism, async multi-turn RL via vLLM/SGLang, and 2.8x TRL throughput on MoE models.
Open original ↗
Document

What it claims

Halo is a training framework that keeps models as plain Hugging Face objects: trainers subclass HF/TRL trainers, checkpoints stay standard SafeTensors, and from_pretrained still works. White Circle states that adding a new model family typically takes under 140 lines of integration code. Release 1.0.0 (2026-08-20) is the first public release.

How it works

  • Parallelism wraps the existing model in place. EP wraps MoE blocks, using DeepEP for all-to-all. CP wraps attention with Ulysses attention. TP shards attention weights via DTensor, and ETP shards expert FFNs. Each degree is set independently, and EP combines with CP, TP or ETP.
  • Pre-training, SFT, preference optimization, distillation and RL share one parallelism and checkpointing stack.
  • AdamWBF16 uses stochastic rounding to hold weights plus optimizer state in 6 bytes/param, versus 12 with FP32 master weights. Muon and FlashAdamW are also available.
  • FP8/FP4 MoE training runs through fake-quant QAT and DeepGEMM, with mxfp8/mxfp4/nvfp4 export.
  • Packing is padding-free and boundary-aware, using cu_seq_lens.
  • Async RL has no Megatron backend and no veRL dependency:
flowchart LR
  T[Transformers trainer] -- NCCL weight push --> S[vLLM / SGLang server]
  R[Ray actor env rollouts] --> S
  S --> R
  R --> Q[Prefetch queue]
  Q --> T

Evidence

All figures are on 8× B300. The baseline is trl.SFTTrainer with Transformers v5 and FSDP2 ZeRO-3, using the same model, data, bf16, FA4, Liger and grouped GEMM.

  • Throughput: gpt-oss-20b reaches up to ~2.8× stock TRL.
  • Matched sharding: with both sides on ZeRO-3, it is 2.7× at 25% lower peak memory.
  • QLoRA memory:
  • Qwen3-4B peaks at 7.9 GB (batch 1, 4k context).
  • Qwen3-8B at 32k context fits in 20.9 GB with the fused loss.

How to use it

The images can be pulled anonymously. There is no latest tag, so pin versioned tags (:blackwell-1.0.0, :hopper-1.0.0). The RL inference images are :vllm-0.26.0 and :sglang-0.5.17.

docker pull public.ecr.aws/whitecircle/halo:blackwell   # B200/B300 — FA4 + FA2 + DeepEP
docker pull public.ecr.aws/whitecircle/halo:hopper      # H100/H200 — FA2 + FA3 + DeepEP

The launcher picks python, torchrun or accelerate on its own. Extra flags override the YAML config:

halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml \
    -n 8 \
    --expert_parallel_size=8 \
    --learning_rate=1e-5 \
    --max_length=32000
halo launch --list
halo launch sft config.yaml --dry-run

Checkpoint tooling:

halo run merge-ep-shards --input_dir <ep-checkpoint-dir> --output_dir <merged-dir>
halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4

The parallelism flags are --expert_parallel_size, --context_parallel_size, --tensor_parallel_size and --expert_tensor_parallel_size.

  • Multi-node: works with shared (NFS/Lustre) or node-local filesystems.
  • Data: datasets can load from s3:// with a local cache.
  • Agent support: images ship agent skills for Claude Code and Codex. Build with --build-arg INSTALL_CLAUDE_CODE=1 to bake in the CLI.
  • Docs: human-docs/ has the task guides, and agent-docs/ has the full technical reference.

Limits

  • ETP: marked experimental.
  • Ampere/Ada GPUs: single-GPU LoRA/QLoRA runs from halo:blackwell using FA2/SDPA, but that path is not validated.
  • Benchmark scope: published only for 8× B300 SFT against TRL. The page gives no Hopper or multi-node comparison numbers.
  • Licence: the Halo License is Apache 2.0 modified by Supplemental Terms, so it is not plain Apache.
  • Use, modification, self-hosting, and shipping trained models are free at any scale.
  • Offering Halo's training functionality to third parties as a service is free up to US $20M in such revenue per 12 months. Above that you need a commercial agreement.
  • If you have over $20M in annual revenue, models you train with Halo must carry a "Trained with Halo" credit.
  • Vendored code: DeepEP and Flash Attention are vendored as submodules under their own upstream licences.

Organisations named here

Technologies named here

Techniques named here

Also named here

  • Tri Dao

Related in training