Aleph Alpha Kolibri-1 (78B MoE, Apache 2.0)

What it claims

Kolibri-1 is a 78B mixture-of-experts reasoning model trained from scratch on a German/English corpus. The card does not state the active-parameter count. Training covered 20T pre-training tokens, 3.44T mid-training tokens and 201B long-context tokens. Knowledge cutoff is June 18, 2026. Aleph Alpha estimates training energy at 9.5×10² MWh. That figure covers pre-, mid- and long-context training including PUE, and excludes SFT, RL and proxy models. The model is meant to run with a human reviewing its output. The card explicitly positions it as advisory in decision-support systems, not autonomous.

How it works

• Attention: most layers are sliding-window, and a few attend globally. Positional encoding is applied only in the sliding-window layers, so context extends past 262,144 tokens without position scaling. The trade-off the card names is memory: all experts must be resident even though few are active per token.
• Tokenizer: UniBPE uses greedy bottom-up BPE merges, chosen with the Unigram objective. Vocabulary is 128,000. It reaches 4.7 bytes/token on German and 4.2 on English. German is 23.9% of the training data.
• Optimizer: Nesterov Muon (lr 0.001) for 2-D backbone weights, and AdamW for embeddings, router, LM head and 1-D parameters. The learning rate is held constant after a ~100B-token warmup. Training ran on 768 GPUs with TorchTitan.
• Data mix: pre-training mixes were searched with thousands of 30M-parameter proxies trained on 3B tokens each. Mid-training and SFT mixes used MergeMix: domain-specialist continued training, then weight-averaging. For SFT, 69 Dirichlet-sampled weightings were scored on 16 benchmarks. Eight were trained at proxy scale and six of those beat the hand-tuned mix. The long-context mix is 34% OCR'd PDFs, 33% optimal mid-training mix and 33% long-document-weighted pool.
• Post-training:
• SFT: 4,000 steps at 262k sequence length, then a soup of two checkpoints trained on different mixes.
• RL: asynchronous, 1,000 steps, quantization-aware training so the model can serve in low precision.
• German: a language-consistency reward applies to both the reasoning trace and the answer, so the model reasons in German.
• Agentic SFT: erroneous tool calls were masked out of the loss.
• Merlin-Arthur hallucination RL:

flowchart LR
  Q[Question + context] --> M[Merlin hides parts<br/>answer more likely]
  Q --> G[Morgana hides parts<br/>answer less likely]
  Q --> O[Original sample]
  M --> A[Train: answer correctly]
  O --> A
  G --> B[Train: abstain]

Evidence

• Post-training benchmarks: Kolibri runs at reasoning effort high. Most benchmarks use Aleph Alpha's eval-framework, and SWE-bench and Terminal-Bench use Harbor. Comparison is limited to MoE models, and dense models such as Apertus are greyed out.
• Base-model comparisons: these use temperature 0.6, top_p 0.6 and 65,536 max context. Qwen3.5 35B-A3B Base is run with static YaRN (factor 4).
• Long-context comparisons: at 512k and 1M tokens, both Kolibri Base and Qwen3.5 run beyond their trained window. Nemotron 3 Nano natively supports 1M.
• Missing comparison values: the fetched text does not include any competitor scores, so the head-to-head claims can't be checked from it.
• PII redaction: an ablation on a 750M-row sample showed no clear performance difference between redacted and unredacted data.
• Decontamination: contamination in the mid-training pool measured between 0 and ~1e-4 of documents per dataset. RULER and MRCR were excluded from the overlap check.

How to use it

The model requires the aleph-alpha-inference vLLM plugin. You can either use the container ghcr.io/aleph-alpha/aleph-alpha-inference or install the package:

pip install 'aleph-alpha-inference>=1'
vllm serve Aleph-Alpha/Kolibri-1 --kv-cache-dtype fp8 \
  --reasoning-parser kolibri1 \
  --tool-call-parser kolibri1 \
  --enable-auto-tool-choice

For contexts beyond 262,144 tokens, add:

--max-model-len 1048576 --hf-overrides '{"max_position_embeddings": 1048576}'

Usage details:

• Sampling: recommended settings are temperature=1.0, top_p=0.97, top_k=128.
• API: the server exposes an OpenAI-compatible API.
• Reasoning effort: set it via extra_body={"chat_template_kwargs": {"reasoning_effort": "low|medium|high", "enable_thinking": True}}. Use reasoning_effort: none or enable_thinking=false for a direct answer.
• Tool calling: pass schemas in the standard tools field. Tool calling can be combined with reasoning.
• Other checkpoints: a Kolibri-1-BF16 repo is listed as the base model in the model tree.

Limits

• Licence scope: Apache 2.0 covers only the weights and config files in the repo. The card says it explicitly does not extend to code, architecture, parameter settings or training methods, and Aleph Alpha retains rights to those.
• Context length: for latency- or throughput-sensitive deployments and for complex tasks, the card recommends staying at or below 262,144 tokens. The 1M figure is "validated", not trained.
• Memory: the full 78B model must be held in memory.
• Languages: the model covers German and English only, by design.
• Safety filtering:
• Built-in refusals do not replace application-level filtering.
• CSAM was addressed only through URL and keyword filters, with no dedicated detection.
• Keyword filtering for explicit content applied to English data only.
• Political bias: training data includes output from Chinese LLMs. This bias was mitigated but not eliminated.
• Responsible-use terms: the card asks users to avoid EU AI Act Article 5 practices, military and nuclear uses, and sanctions or export violations.