Token

A token is the basic unit a large language model operates on: a piece of text, often a word, part of a word, or punctuation mark, produced by breaking input into smaller chunks in a step called tokenization. "Analytics" might be one token or split into pieces like "analy" and "tics," depending on the model's tokenizer and vocabulary.

Models do not read raw characters; they convert text into a sequence of token IDs, process that sequence, and then convert the predicted token IDs back into text. This differs across models and languages: tokenizers are trained on specific text distributions, so the same sentence can use noticeably more tokens in one language or domain than another, and code or rare technical terms often tokenize less efficiently than everyday prose.

Tokens matter practically because they set the limits and cost of working with a model: a context window is measured in tokens, and most commercial LLM pricing charges per token processed and generated. Practitioners writing prompt engineering for cost- or latency-sensitive applications need to estimate token counts, and a common pitfall is assuming "words" and "tokens" are interchangeable when budgeting context or cost.