Understanding Context Windows: Managing Input and Output Sizes

In previous lessons, you learned how large language models (LLMs) generate text one token at a time and how different model versions affect your results. Now, let's focus on a key concept that shapes what you can do with LLMs: the context window.

A context window is the maximum amount of information (measured in tokens) a model can consider simultaneously. This includes your input (the prompt) and the model's output (the response). If you try to give the model more information than fits in its context window, some of it will be ignored or cut off.

Understanding context windows is important because it helps you design prompts that fit within these limits, ensuring the model can "see" everything it needs to give you a good answer.

Context windows have changed a lot as LLMs have improved. Early models could only handle short prompts and responses, while newer models can work with much more information at once.

Here's a simple table showing how context window sizes have grown over time:

As you can see, newer models can handle much larger context windows. This means you can give them longer prompts or get longer responses, but there is always a limit.

The models available to you span a wide range. Here is how the four compare:

Notice that even within a single environment, the limits differ by roughly eight times. If a long prompt works with one model but gets truncated with another, the context window is often the reason. This is worth remembering when a practice asks you to switch models.

Keep in mind that a large context window is a ceiling, not a guarantee. Models tend to answer more accurately when you give them the few thousand tokens that matter rather than pasting in everything you have — which is exactly what the strategies below are for.