Explaining the LLM Prediction Game Idea

Welcome to the first lesson of our course on building the LLM Prediction Game! In this course, you will learn how to create a fun, interactive word game powered by a Large Language Model (LLM), such as GPT-4. The main idea is simple: you will try to predict which word the LLM will say next in response to a prompt.

This lesson will give you a clear understanding of how the game works, what makes it unique, and the key concepts behind its design. By the end, you’ll be ready to start building the game step by step in the following lessons.

Before we dive into the game, let’s quickly recall what a Large Language Model (LLM) is. If you’ve seen this before, consider this a quick refresher.

A Large Language Model is a type of artificial intelligence that can generate human-like text. It does this by predicting the next word in a sentence based on the words that came before. For example, if you give an LLM the prompt:

The capital of France is

The model will likely predict the next word as “Paris.” LLMs are trained on huge amounts of text data, so they can answer questions, write stories, and even play games.

Now, let’s look at the main idea of the LLM Prediction Game.

Each day, the game presents you with a new prompt and a simulated assistant’s response. The response is shown one word at a time, just like watching someone type. At three points during the response, the game pauses and asks you to guess what the next word will be.

For example, imagine the prompt is:

User: What is the capital of France?
Assistant: The capital of ____ is Paris

As the assistant’s answer is revealed word by word, you might see:

The capital of 

At this point, the game pauses and asks you to guess the next word. You might guess “France” The game then checks how close your guess is to the actual next word and gives you a score.

This process repeats for three different points in the response. At the end, your total score is shown, and you can see how you compare to other players on a leaderboard.

Let’s break down how a typical game session works, step by step:

1. Start Screen and Prompt Reveal

   • You begin on a start screen and click “Play.”
   • The game shows you the system prompt (which sets the assistant’s behavior) and the user’s question.

2. Animated LLM Response with Pause Points

   • The assistant’s response appears one word at a time.
   • At certain points (called breakpoints), the animation pauses.

3. Player Guesses and Scoring

   • When the animation pauses, you are asked to guess the next word.
   • You type your guess and submit it.

4. Final Score and Leaderboard

   • After three guesses, your scores are added up.
   • You can enter your name and see how your score compares to others on the leaderboard.

This flow makes the game both challenging and fun, as you try to “think like the AI” and predict its next move.

A unique part of this game is how your guesses are scored. Instead of just checking if your guess matches the exact word, the game uses something called semantic similarity. This means your guess can still earn points if it’s close in meaning to the correct word.

For example, suppose the correct word is “Paris” but you guess “London” Both are capital cities, so your guess is similar in meaning, even though it’s not exactly right. The game uses spaCy to measure how close your guess is to the actual word.

Here’s a simple example:

• If you guess “Paris,” you get a perfect score.
• If you guess “London,” you get a high score because it’s another capital city.
• If you guess “Apple,” you get a low score because it’s not related.

This scoring system makes the game more interesting and fair, rewarding you for thinking in the right direction.

In this lesson, you learned what the LLM Prediction Game is, how it works, and the key idea of semantic similarity scoring. You also saw how a typical game session flows from start to finish.

In the next sections, you’ll get hands-on practice and start building the game step by step. You’ll learn how to set up the game data, handle prompts and responses, and implement the scoring system. Get ready to put these ideas into action!