Introduction to Document Processing with Java

Welcome to the first lesson of Document Processing and Retrieval with LangChain in Java! In this course, you'll learn how to work with documents programmatically, extract valuable information from them, and build systems that can intelligently interact with document content using Java and LangChain4j.

Document processing is a fundamental task in many applications, from search engines to question-answering systems. The typical document processing pipeline consists of several key steps: loading documents from various sources, splitting them into manageable chunks, converting those chunks into numerical representations (embeddings), and finally retrieving relevant information when needed.

In this lesson, we'll focus on the first two steps of this pipeline: loading documents and splitting them into appropriate chunks. These steps are crucial because they form the foundation for all subsequent document processing tasks. If your documents aren't loaded correctly or split effectively, the quality of your embeddings and retrieval will suffer.

By the end of this lesson, you'll be able to:

• Load documents from different file formats using LangChain4j
• Split documents into manageable chunks for further processing
• Understand how to prepare documents for embedding and retrieval

Let's get started with understanding the document loaders available in LangChain4j.

LangChain4j provides several document loaders to simplify document processing by offering tools to handle different file formats. For PDF files, we can use the FileSystemDocumentLoader along with ApachePdfBoxDocumentParser The ApachePdfBoxDocumentParser is a parser that leverages the Apache PDFBox library, a popular Java tool for handling PDF files. It's used to extract text, process layout, and retrieve metadata from PDFs. It's ideal for efficiently and accurately processing PDFs in LangChain4j.

Here's how to load a document using a concrete example of a Sherlock Holmes story in PDF format:

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.loader.FileSystemDocumentLoader;
import dev.langchain4j.data.document.parser.apache.pdfbox.ApachePdfBoxDocumentParser;

public class DocumentLoader {
    public static void main(String[] args) {
        // Define the file path to our Sherlock Holmes story
        String filePath = "src/main/resources/data/the_adventure_of_the_blue_carbuncle.pdf";
        
        // Create a PDF loader for our document
        Document document = FileSystemDocumentLoader.loadDocument(filePath, new ApachePdfBoxDocumentParser());
        
        // Print the document content
        System.out.println("Document content: " + document.text());
    }
}

This code snippet demonstrates how to load a PDF document and extract its text content using LangChain4j. The loadDocument method reads the file and returns a Document object that contains both the content and metadata of the document.

Once we've loaded our document, we need to split it into smaller, more manageable chunks. LangChain4j offers various document splitters to handle different text structures:

• DocumentByParagraphSplitter - Splits on paragraph breaks (double newlines)
• DocumentByLineSplitter - Splits on single line breaks
• DocumentBySentenceSplitter - Splits text into natural sentences
• DocumentByWordSplitter - Splits into word groups
• DocumentByCharacterSplitter - Splits by character count
• DocumentByRegexSplitter - Splits using custom regex patterns
• DocumentSplitters.recursive() - Applies multiple separators in sequence until desired chunk size is reached

We will be using the DocumentByParagraphSplitter, as it is particularly well-suited for documents with clear paragraph boundaries, like articles, essays, or books. This helps preserve the semantic structure of the content, ensuring each chunk remains meaningful:

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.splitter.DocumentByParagraphSplitter;
import dev.langchain4j.data.segment.TextSegment;
import java.util.List;

public class DocumentSplitter {
    public static void main(String[] args) {
        String filePath = "src/main/resources/data/the_adventure_of_the_blue_carbuncle.pdf";
        Document document = FileSystemDocumentLoader.loadDocument(filePath, new ApachePdfBoxDocumentParser());
        
        // Split the document into chunks
        DocumentByParagraphSplitter splitter = new DocumentByParagraphSplitter(500, 5);
        List<TextSegment> chunks = splitter.split(document);
        
        // Print the number of chunks and first chunk content
        System.out.println("Number of chunks: " + chunks.size());
        System.out.println("First chunk: " + chunks.get(0).text());
    }
}

The DocumentByParagraphSplitter takes two parameters:

1. maxSegmentSize: The maximum size of each chunk (500 in our example)
2. maxOverlap: The number of characters that overlap between chunks (5 in our example)

After splitting the document, we can inspect both the content and metadata of the chunks:

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.loader.FileSystemDocumentLoader;
import dev.langchain4j.data.document.parser.apache.pdfbox.ApachePdfBoxDocumentParser;
import dev.langchain4j.data.document.splitter.DocumentByParagraphSplitter;
import dev.langchain4j.data.segment.TextSegment;
import java.util.List;

public class DocumentInspector {
    public static void main(String[] args) {
        String filePath = "src/main/resources/data/the_adventure_of_the_blue_carbuncle.pdf";
        Document document = FileSystemDocumentLoader.loadDocument(filePath, new ApachePdfBoxDocumentParser());
        
        DocumentByParagraphSplitter splitter = new DocumentByParagraphSplitter(500, 5);
        List<TextSegment> chunks = splitter.split(document);
        
        // Print the number of document chunks loaded
        System.out.println("Loaded " + chunks.size() + " document chunks");
        
        // Print the content of the first chunk
        System.out.println("\nFirst 200 characters of the first chunk:\n" + 
            chunks.get(0).text().substring(0, 200));
        
        // Print the metadata of the first chunk
        System.out.println("\nMetadata of the first chunk:\n" + chunks.get(0).metadata());
    }
}

This code demonstrates how to:

1. Load a PDF document using LangChain4j
2. Split it into chunks using the paragraph splitter
3. Inspect the content and metadata of the resulting chunks

In this lesson, you've learned how to:

1. Use LangChain4j's document loaders to load PDF files
2. Split documents into manageable chunks using the DocumentByParagraphSplitter
3. Access and inspect document content and metadata

These fundamental operations form the basis for more advanced document processing tasks. In the next lesson, we'll explore how to convert these document chunks into vector embeddings, which will allow us to perform semantic search and retrieval.