Why AI Struggles to Count Letters in a Word

Language models can produce fluent writing and follow multi-step instructions, yet they often give wrong answers to questions that look trivial, such as how many letters appear in a short word. Part of the explanation lies not in the model’s reasoning but in how text is broken up before the model ever sees it.

How Words Become Tokens

Before a language model processes text, the text is split into units called tokens. A common method for doing this is byte-pair encoding (BPE), described in detail in the Hugging Face NLP course at https://huggingface.co/learn/nlp-course/chapter6/5. According to that material, BPE does not simply keep words intact or split them into individual letters. Instead, it learns a vocabulary through a training process.

The process starts with a base vocabulary of individual characters. From there, the algorithm learns merges, which are rules that combine two existing vocabulary elements into a new one. At each step it finds the most frequent adjacent pair of tokens and merges them. Early merges produce two-character pieces, and as training continues, longer subwords emerge. In the course’s worked example, the pair “u” and “g” merges into “ug”, then “un” forms, and then “hug” appears as a first three-letter token.

The result is that a real word is often represented as a handful of subword chunks rather than as a clean sequence of letters. The model operates over these chunks, not over the characters a person would count by eye.

Why Subwords Instead of Characters

The course explains the tradeoff behind this design. A vocabulary made only of characters runs into a problem: any character that never appeared in the training corpus is converted to an unknown token. To balance coverage against efficiency, models add merged subwords until the vocabulary reaches a desired size.

Some models go further with byte-level BPE. As the course notes, GPT-2 and RoBERTa work at the level of bytes, which keeps the base vocabulary small, around 256 entries, while still allowing every possible character to be represented. This avoids unknown tokens without forcing the model to treat every word as a long string of single characters.

What This Means for Simple Tasks

When new text arrives, it is normalized, pre-tokenized, and split into characters, after which the learned merge rules are applied in order. The course shows how an unseen word like “bug” is rebuilt as the pieces “b” and “ug” rather than left as raw characters. A task such as counting how many times a specific letter appears in a word asks the model to work at the character level, but the input it receives has already been reassembled into subword units.

This mismatch helps explain why a model can handle a demanding writing or coding request while stumbling on a letter-counting question. The difficulty is rooted in representation rather than in the conceptual hardness of the task.

Reading Model Behavior With This in Mind

Understanding tokenization sets more realistic expectations for where these systems are dependable and where they need help. Tasks that align with how the model represents text tend to go smoothly, while operations that require precise character-by-character handling are a poor fit for the same pipeline. Knowing the mechanism behind that gap makes a surprising failure easier to interpret, and it points toward giving the model the right kind of task rather than assuming a single answer reflects a single kind of intelligence.