tokenizers v1: encode, decode and scaling, measured

What it claims

tokenizers v1 (Rust release candidate on crates.io) encodes 3–30x faster than v0.23 single-threaded and produces identical token IDs. The API, vocabularies and merge ranks are unchanged, and v1 loads everything v0.23 loaded. The range has specific ends: t5-base is the low end at 3x and gpt2 is the high end at 30x. Batch encoding scales at 76% of linear across eight workers.

How it works

flowchart TD
    A[Input text] --> B{Pre-tokenizer pattern recognised?}
    B -- GPT-2 / cl100k / o200k / Tekken / DeepSeek --> C[bitcannon SIMD split]
    B -- other pattern --> D[regex split, no speed-up]
    C --> E{Pre-token in thread-local cache?}
    D --> E
    E -- hit --> G[Token IDs]
    E -- miss --> F[Allocation-free merge loop] --> G

• The split is the deciding step. bitcannon treats the input bytes as parallel bit streams and resolves 64 bytes per register operation, the same approach Parabix and simdjson use. It is hand-written for each known grammar. Any tokenizer whose pattern is not one of the five listed keeps the regex path, which is why the gains vary so much.
• Merge loop. A scratch buffer owned by the caller replaces the per-call allocations and the per-pre-token priority queue. Symbols sit in a flat array and are linked by index. Each candidate pair is packed into a 64-bit value with the rank in the high bits, and "no merge" is the maximum value, so selecting the next merge needs no branch. Multiple pre-token spans go through one model call.
• Decoding writes into a reusable buffer, supports buffered streaming and decodes batches in parallel.
• Crate structure. The crate is split into tk-encode, tk-serialize, tk-convert and tk-train.

Evidence

The setting is single-threaded on an Apple M4 Max across ten model families: eight BPE, one WordPiece and one Unigram. The headline figures use a stream of distinct documents from a corpus too large to fit in the cache. They do not come from re-encoding one document, which the authors say can be much faster and is often also called "warm". The comparison engines include tiktoken and other fast tokenizers, run through the tokbench repository. The page gives no per-competitor numbers in its text. To reproduce the cache effect:

tokbench measure prefix-sharing \
  --engine pipeline \
  --engine hf-tokenizers \
  --compare-to pipeline-no-cache \
  --corpus agentic_swe

How to use it

cargo add tokenizers --pre

Training is on by default and pulls in a C++ dependency. For an encode-only build:

cargo add tokenizers --pre --no-default-features --features http

Existing calls work unchanged:

let tokenizer = Tokenizer::from_pretrained("deepseek-ai/DeepSeek-V4-Flash", None)?;
let encoding = tokenizer.encode("The tokenizer is no longer the bottleneck.", false)?;
let encodings = tokenizer.encode_batch(documents, false)?; // the call that scales across cores

The release also adds Node.js bindings and role_to_token support.

Limits

• Python numbers are not measured. Every figure comes from the Rust crate. The Python bindings wrap the same code but add per-call overhead that none of the measurements include. Python users and transformers users should not expect the headline gains yet, because integration into transformers comes after the release candidates stabilise.
• Unlisted pre-tokenizer patterns get no split speed-up.
• The word cache can cost time. On input with few repeated pre-tokens, lookups add overhead without producing many hits.
• The release is pre-1.0. Still pending:
• more model families moved onto the new merge loop
• optional offsets and masks
• a normalizer rework
• spm precompiled support
• simplified Python bindings, including free-threaded CPython
• C/C++ bindings for ExecuTorch and llama.cpp
• a single encode path shared by training and inference
• GPU encode and decode (tok-devices) is only planned for after 1.0.0.
• The post may change. It is generated from tokbench results and will be updated, so the numbers on the page can shift.