ttok 1.0: Token Counter Switches Default to GPT-5/GPT-6

Seven GPT models now share the same tokenizer, counting 44,794 tokens across identical test fixtures. This finding prompted the release of ttok 1.0, a command-line tool for counting and truncating text based on tokens.

Default Tokenizer Change

The tool previously defaulted to the GPT-4 tokenizer. After upgrading from version 0.4, developer Simon Willison discovered this default no longer made sense given the current model landscape. The new 1.0 release switches to the GPT-5/GPT-6 tokenizer family as the default.

OpenAI has not officially confirmed that GPT-6 uses the same tokenizer as GPT-5 models, and an issue tracking this question remains open. However, experimental evidence suggests the tokenizers are identical across the GPT-5 and GPT-6 families.

Tokenizer Verification

Developer William Liu ran experiments comparing tokenization across multiple models. According to his commit, all seven tested GPT models produced matching results: GPT-5.5, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-6 Astra, GPT-6 Sol, and GPT-6 Luna. Each model reported 44,794 tokens when processing 31 test fixtures, with every token matching across all models.

The findings indicate GPT-6 introduces no changes to input token counting compared to the GPT-5 family. This consistency across model versions makes the shared tokenizer a stable default choice for the tool.

Installation and Usage

The tool can be upgraded using uv tool upgrade ttok. It accepts text input through pipes, making it straightforward to count tokens in files or text streams. The command-line interface allows developers to both count total tokens and truncate text to fit within token limits.

Token counting matters for API usage because language models charge based on token consumption rather than character or word count. Different tokenizers split text into tokens differently, so using the correct tokenizer ensures accurate cost estimation and prevents requests from exceeding model context limits.

Version Milestone

Switching the default tokenizer provided justification for the 1.0 version number. The change represents a breaking modification for users who relied on the previous GPT-4 default, though the underlying functionality remains focused on the same core tasks: counting and truncating text based on token boundaries.

The tool addresses a common developer need when working with language model APIs. Knowing token counts before making API calls helps estimate costs, ensure prompts fit within context windows, and truncate content appropriately when needed.

For applications integrating multiple OpenAI models, the tokenizer consistency across GPT-5 and GPT-6 families simplifies token counting. Developers can use a single tokenizer reference across these model versions rather than maintaining separate counting logic for each model variant.