How LLMs Generate Text: Predicting the Next Token

This course was developed in conjunction with Amazon and AWS.

Welcome to the first lesson of this course on prompt engineering. In this lesson, I will introduce a key principle behind how large language models (LLMs) generate text. Understanding this principle is important because it will help you write better prompts and get more accurate and helpful responses from LLMs.

LLMs, like the ones you will use in this course, generate text by predicting one piece at a time. These pieces are called tokens. The model looks at the text you have already given (the context) and tries to guess what comes next, one token at a time. This process is repeated until the model finishes its response.

Let's break down what a token is and how this prediction process works.

A token is a small chunk of text. Depending on the language model, it can be a word, part of a word, or even just a character. LLMs do not see text as whole sentences or paragraphs. Instead, they break everything down into tokens.

For example, let's look at the phrase:

A bottle of water

Depending on the model, this might be split into tokens like:

• A
• bottle
• of
• water

Or, in some cases, bottle might be split into bott and le if the model uses smaller pieces. For this lesson, you can think of tokens as words or short word parts.

Tokens are important because the model predicts text one token, not one word or sentence at a time.

Now, let's see how LLMs generate text step by step. The model always looks at the context — the already written tokens — and predicts what comes next.

Let's start with a simple prompt:

A bottle of

At this point, the model has three tokens: A, bottle, and of. It now needs to predict the next token. The model looks at the context (A bottle of) and tries to guess what is most likely to come next.

Common next tokens might be:

• water
• wine
• milk

The model chooses the most likely one based on its training. If you continue, the process repeats. For example, if the model predicts water, the new context is:

A bottle of water

Now, the model can predict what comes after water, such as a period or another word.

This process — predicting one token at a time — continues until the model decides the response is complete.