ttok 1.0

What it claims

ttok 1.0 changes the default tokenizer from GPT-4's to the one shared by the GPT-5/GPT-6 family. The default change is the only reason for the 1.0 version number. The release came right after 0.4, which still defaulted to GPT-4. The author found that problem by piping a file through the freshly upgraded 0.4. If you installed 0.4, your counts are GPT-4 counts unless you chose a tokenizer yourself.

Evidence

The new default rests on an experiment in a commit by William Liu, not on any statement from OpenAI. Seven models were tested:

• GPT-5.5
• GPT-5.6 Sol, Terra and Luna
• GPT-6 Astra, Sol and Luna

Across 31 fixtures, every model reported a total of 44,794 tokens, and all seven matched on each individual fixture. The commit's conclusion: "GPT-6 introduces no input-count change on this corpus."

How to use it

The page shows the upgrade path the author used for an existing uv tool install:

uv tool upgrade ttok

The page does not give flags or syntax for choosing a non-default tokenizer.

Limits

• No official confirmation. OpenAI has not confirmed that GPT-6 uses the same tokenizer as GPT-5. There is an open, contested issue about it. The default is an inference from the experiment.
• Narrow evidence. The check covers 31 fixtures and only input-token counts. It does not show that output tokens, other content types or text outside that corpus tokenize the same way.
• Only seven models tested. Other GPT-5/GPT-6 variants, if they exist, were not checked.
• What to do. If exact counts matter, for budget caps or context-limit truncation, check a sample against the API's reported usage instead of relying on the default.