Introduction to Document Processing in Go

Welcome to the first lesson of Document Processing and Retrieval with LangChain in Go! In this course, you'll learn how to work with documents programmatically, extract valuable information from them, and build systems that can intelligently interact with document content.

Document processing is a fundamental task in many applications, from search engines to question-answering systems. The typical document processing pipeline consists of several key steps: loading documents from various sources, splitting them into manageable chunks, converting those chunks into numerical representations (embeddings), and finally retrieving relevant information when needed.

In this lesson, we'll focus on the first two steps of this pipeline: loading documents and splitting them into appropriate chunks. These steps are crucial because they form the foundation for all subsequent document processing tasks. If your documents aren't loaded correctly or split effectively, the quality of your embeddings and retrieval will suffer.

By the end of this lesson, you'll be able to:

• Load documents from different file formats using LangChain
• Split documents into manageable chunks for further processing
• Understand how to prepare documents for embedding and retrieval

Let's get started with understanding document loaders in LangChain.

LangChain for Go provides document loaders that simplify the process of reading and parsing different file formats. These loaders handle the complexities of extracting content from various document types, allowing you to focus on working with the content.

Let's look at how to use the text loader, which is one of the most common document loaders:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/tmc/langchaingo/documentloaders"
)

func main() {
    // Open the text file
    file, err := os.Open("data/sample.txt")
    if err != nil {
        log.Fatalf("failed to open file: %v", err)
    }
    defer file.Close()

    // Create a new Text loader
    loader := documentloaders.NewText(file)

    // Load the document
    ctx := context.Background()
    documents, err := loader.Load(ctx)
    if err != nil {
        log.Fatalf("failed to load document: %v", err)
    }

    // Process the loaded documents
    for i, doc := range documents {
        fmt.Printf("Document %d:\n%s\n---\n", i+1, doc.PageContent)
    }
}

In this example, we:

1. Open a text file using Go's standard os package
2. Create a new text loader using LangChain's documentloaders.NewText function
3. Load the document using the loader's Load method
4. Process the loaded documents, which are returned as a slice of Document objects

Each Document object contains the content of the document in its PageContent field, along with metadata about the document.

LangChain for Go supports various document types. Here's how you might load a PDF document:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/tmc/langchaingo/documentloaders"
)

func main() {
    // Open the PDF file
    file, err := os.Open("data/document.pdf")
    if err != nil {
        log.Fatalf("failed to open PDF file: %v", err)
    }
    defer file.Close()

    // Obtain the file size
    fi, err := file.Stat()
    if err != nil {
        log.Fatalf("failed to stat PDF file: %v", err)
    }
    size := fi.Size()

    // Create the PDF loader
    loader := documentloaders.NewPDF(file, size)

    // Load the document
    ctx := context.Background()
    documents, err := loader.Load(ctx)
    if err != nil {
        log.Fatalf("failed to load PDF document: %v", err)
    }

    // Process the loaded documents
    for i, doc := range documents {
        fmt.Printf("Document %d:\n%s\n---\n", i+1, doc.PageContent)
    }
}

Note on PDF Parsing: It's important to understand that PDFs can be lossy to parse. Unlike plain text files, PDFs are primarily visual formats designed for rendering, not for extracting structured text. This means you may encounter issues with spacing, text ordering, and layout preservation. For example, multi-column layouts might have text extracted in an unexpected order, and tables or figures might not parse cleanly. Keep these limitations in mind when working with PDFs, and always verify the extracted content quality for your specific use case.

LangChain also provides loaders for other formats such as CSV, JSON, and more, each designed to handle the specific characteristics of that format.

While loading documents is an essential first step, most documents are too large to process as a single unit, especially when working with language models or embedding techniques. This is where document splitting comes into play. Document splitting involves breaking down a large document into smaller, more manageable chunks. These chunks can then be processed individually, making it easier to work with large documents and improving the quality of embeddings and retrieval.

LangChain provides text splitters that make it easy to divide documents into appropriate chunks. Let's explore how to use them.

LangChain's textsplitter package offers various strategies for splitting documents. One common approach is to use the RecursiveCharacter splitter, which recursively splits text by different separators:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/tmc/langchaingo/documentloaders"
    "github.com/tmc/langchaingo/textsplitter"
)

func main() {
    // Open the text file
    file, err := os.Open("data/sample.txt")
    if err != nil {
        log.Fatalf("Failed to open file: %v", err)
    }
    defer file.Close()

    // Create a new Text loader
    loader := documentloaders.NewText(file)

    // Initialize the text splitter
    splitter := textsplitter.NewRecursiveCharacter(
        textsplitter.WithChunkSize(500),
        textsplitter.WithChunkOverlap(100),
    )

    // Load and split the document
    ctx := context.Background()
    documents, err := loader.LoadAndSplit(ctx, splitter)
    if err != nil {
        log.Fatalf("Failed to load and split document: %v", err)
    }

    // Process the resulting chunks
    for i, doc := range documents {
        fmt.Printf("Chunk %d:\n%s\n---\n", i+1, doc.PageContent)
    }
}

In this example, we:

1. Create a text loader as before
2. Initialize a RecursiveCharacter splitter with a chunk size of 500 characters and an overlap of 100 characters
3. Use the loader's LoadAndSplit method to both load and split the document in one step
4. Process the resulting chunks, which are returned as a slice of Document objects

The RecursiveCharacter splitter tries to split on paragraph breaks first, then sentence boundaries, and finally on individual characters if needed, ensuring that chunks don't exceed the specified size.

Effective chunking is a balance between chunk size and overlap. Too small a chunk size may fragment important ideas, while too large a size may exceed the token limit of embedding models. Overlap helps preserve context at chunk boundaries, since chunks are embedded/retrieved independently—without overlap, a sentence or key detail split between two chunks may be missing from each chunk on its own, which can hurt embedding quality and retrieval.

Similarly, too much overlap can introduce redundancy. For most applications, starting with a chunk size of 500–1000 characters and an overlap of 50–100 characters is a reasonable default, but you may need to adjust these parameters based on your specific documents and use case.

The optimal chunking strategy often depends on:

• The nature of your documents (technical papers vs. narrative text)
• The specific requirements of your downstream tasks
• The token limits of the embedding or language models you're using

Don't be afraid to experiment with different chunking parameters to find what works best for your particular application.

In this lesson, you've learned how to load documents using LangChain's document loaders and how to split documents into manageable chunks. These are the first two steps in the document processing pipeline and form the foundation for more advanced tasks like embedding and retrieval.

Let's recap what we've covered:

1. We explored how to use document loaders in LangChain to load text and PDF files.
2. We discussed the importance of document splitting and how it helps in processing large documents.
3. We used LangChain's text splitters to divide documents into manageable chunks with overlap to maintain context between chunks.

In the next lesson, we'll explore how to convert these document chunks into vector embeddings, which will allow us to perform semantic search and retrieval. You'll learn how embedding models work and how to use them effectively. The document loading and splitting techniques you've learned here are essential prerequisites for these more advanced operations, as they ensure that your documents are properly prepared for embedding and retrieval.