Understanding Context Windows: Managing Input and Output Sizes

In previous lessons, you learned how large language models (LLMs) generate text one token at a time and how different model versions affect your results. Now, let's focus on a key concept that shapes what you can do with LLMs: the context window.

A context window is the maximum amount of information (measured in tokens) a model can consider simultaneously. This includes your input (the prompt) and the model's output (the response). If you try to give the model more information than fits in its context window, some of it will be ignored or cut off.

Understanding context windows is important because it helps you design prompts that fit within these limits, ensuring the model can "see" everything it needs to give you a good answer.

Context windows have changed a lot as LLMs have improved. Early models could only handle short prompts and responses, while newer models can work with much more information at once.

Here's a simple table showing how context window sizes have grown over time:

As you can see, newer models can handle much larger context windows. This means you can give them longer prompts or get longer responses, but there is always a limit.

Let's look at how the context window shapes what you can do with an LLM.

The context window is shared between your input and the model's output. For example, if a model has a 4,096-token context window and your prompt is 2,000 tokens, it can only generate up to 2,096 tokens in its response. If your prompt is too long, the model's response will be shorter or cut off part of your input.