arxiv.orgpaper

Objective vs. Search: Decomposing What Makes a Good Tokeniser

This arXiv paper disentangles the two axes that distinguish BPE and UnigramLM tokenisers—optimisation objective (compression vs. log-likelihood) and search procedure (bottom-up merging vs. top-down pruning)—by introducing two new hybrid algorithms, BottomUpLL and TopDownComp, that complete the 2x2 design space. Training language models across varying sizes, vocabulary sizes, and domains, the authors find that search procedure, not objective, is the dominant factor for bits-per-byte performance, with bottom-up tokenisers consistently outperforming, though no consistent effect emerges on the BLiMP linguistic benchmark. The work offers concrete guidance for more principled tokeniser construction.