Context window

The context window is the maximum span of text, counted in tokens, that a large language model can take into account at one time, including the prompt, any supplied documents or conversation history, and the response it generates. Anything beyond that limit is truncated or must be dropped before the model sees it.

Context window sizes vary widely across models, from a few thousand tokens in earlier systems to hundreds of thousands or more in current ones, and a larger window is not automatically better: models can still pay less attention to information placed in the middle of a very long context than to the beginning or end, an effect sometimes described informally as "lost in the middle."

Context window size directly shapes system design: it limits how much conversation history a chat application can retain, how many documents a retrieval-augmented generation system can pass in per query, and how much a single prompt engineering template can include before needing summarization or chunking. A common pitfall is assuming a large context window makes retrieval unnecessary; even with room to spare, retrieving only the most relevant passages generally produces more accurate answers than stuffing in everything available.