tiktoken
Named by 5 kept items in this workspace.
4 items name it under frameworks and tools and 1 under classification.technologies.
Search every workspace for tiktoken. It matches the name as written, so it can find other things that share it.
The items that name it 5
OpenAI's open source library used by ttok to count tokens
OpenAI's tokenizer library, used by the author to count tokens for the o200k_base tokenizer (GPT-4o/GPT-5).
third-party fast tokenizer library that influenced ideas in tokenizers v1
Counts tokens for token-budget chunking.
All technologies in this workspace.