Welcome back! In our previous lesson, we explored the basics of chunking and storing text for efficient processing with Large Language Models (LLMs). We learned how breaking down large text into manageable chunks is crucial for handling text data effectively. Today, we'll dive deeper into advanced chunking techniques, focusing on recursive character-based and token-based methods. These techniques will help you optimize text processing for AI models, making your applications more efficient and effective.

Overlapping chunks involve ensuring that consecutive chunks share some common content. This shared content helps maintain continuity and context across chunks, which is crucial for AI models to understand and process text effectively.

Imagine a chatbot answering questions about an article. If chunks don’t overlap, the model might lose track of key details, leading to incomplete or incorrect responses. Overlapping chunks help maintain continuity by ensuring key phrases appear in consecutive chunks. This is especially important for AI applications like summarization, search indexing, and document processing. Overlap ensures that information flows smoothly across chunks, preserving context and enhancing the model's understanding.

To achieve effective overlapping, we can utilize Recursive Character-Based Chunking. This technique involves breaking down text into smaller pieces based on characters while preserving context. It respects natural boundaries like sentences and paragraphs, ensuring that chunks are readable and maintain logical structure. We'll use the RecursiveCharacterTextSplitter from the langchain library to implement this method.

1. Define a Maximum Chunk Size – Set a limit for each chunk (e.g., 100 characters).
2. Choose Separators – These define where text should be split, such as:
   • Paragraphs (\n\n)
   • Sentences (.)
   • Spaces ( ) as a last resort
3. Recursive Splitting:
   • The text is first split using the largest separator (paragraphs).
   • If any chunk is too long, it is further split using the next separator (sentences).
   • If necessary, it continues down to spaces to ensure all chunks fit within the limit.
4. Apply Overlap – Ensures that some characters from the end of one chunk appear at the start of the next chunk to maintain context.

First, we need to import the required libraries. We'll use RecursiveCharacterTextSplitter from langchain.text_splitter.

Imagine we loaded some data from text.txt that we'll use for chunking.

Now, we'll set up the RecursiveCharacterTextSplitter. We'll define parameters like chunk_size, chunk_overlap, and separators to control how the text is split.

• chunk_size=100: This sets the maximum size of each chunk.
• chunk_overlap=20: This ensures that each chunk overlaps with the next by 20 characters, preserving context.
• separators=["\n\n", ".", " "]: These define the hierarchy of splitting: paragraphs → sentences → spaces.

The splitter works by first attempting to break the text at double newlines (\n\n, indicating paragraphs); if a chunk is still too long, it then tries to split at sentence breaks (.), and if necessary, finally at spaces ( ). This approach ensures logical splitting while maintaining readable chunks.

Finally, we'll use the splitter to break the text into chunks and print the results to see how the text has been chunked.

This will output the text in manageable chunks, each preserving the context from the original text.

Now, let's explore token-based chunking. Token-based chunking breaks text into tokens, the smallest units of meaning, using a tokenizer. This method is effective for LLMs that process tokenized input. The process involves:

1. Tokenization: Convert text into tokens using a tokenizer, which identifies meaningful units like words or subwords.
2. Chunk Size and Overlap: Define the maximum number of tokens per chunk (chunk_size) and the overlap between chunks (chunk_overlap) to maintain context.
3. Splitting: Split the tokenized text into chunks based on the defined size and overlap, optimizing for LLM processing.
4. Encoding: Specify an encoding, such as OpenAI's cl100k_base, to determine tokenization.

This method ensures compatibility with LLM token windows, improving processing efficiency. We will see step-by-step implementations next.

First, import the required libraries for token-based chunking.

Before chunking, let's see how tokens differ from characters and words.

Output:

As we can see, tokens don’t always match character or word counts. For example, the sentence above has 51 characters, 8 words, but 10 tokens. This is because tokenizers may split words into smaller subword units, especially for uncommon words or punctuation. This is why token-based chunking is more effective for LLMs that rely on tokenized input rather than raw text.

Next, we'll set up the TokenTextSplitter using OpenAI's tokenizer.

• encoding_name="cl100k_base": This specifies the encoding to use for tokenization.
• chunk_size=40: This sets the maximum number of tokens per chunk.
• chunk_overlap=10: This ensures that each chunk overlaps with the next by 10 tokens.

Now, we'll use the token splitter to break the text into token-based chunks and print the results.

This will output the text in token-based chunks, each optimized for precise text processing.

Both recursive character-based and token-based chunking have their advantages and limitations. Recursive character-based chunking is useful for preserving context in a more flexible manner, while token-based chunking offers precision by focusing on meaningful units of text. Depending on your specific NLP task, you may choose one method over the other.

"Understanding these trade-offs helps us choose the right method for specific AI applications."

In this lesson, we explored advanced chunking techniques for LLMs, focusing on recursive character-based and token-based methods. Overlapping chunks help maintain context, enhancing model understanding. Recursive character-based chunking respects natural text boundaries, while token-based chunking aligns with token limits.

As you prepare for practice, consider how these methods can improve AI applications and think about when to use each technique. Familiarize yourself with the langchain and tiktoken libraries, as you'll apply these concepts in the upcoming exercises.