Building a Chat Engine with Conversation History in Java

Welcome to the second unit of our course on building a RAG-powered chatbot! In the previous lesson, we built a document processor that forms the retrieval component of our RAG system. Today, we'll focus on the conversational aspect by creating a chat engine that can maintain conversation history.

While our document processor is excellent at finding relevant information, a complete RAG system needs a way to interact with users in a natural, conversational manner. This is where our chat engine comes in. The chat engine is responsible for managing the conversation flow, formatting prompts with relevant context, and maintaining a history of the interaction.

The chat engine we'll build today will:

1. Manage interactions with the language model
2. Optionally maintain a history of the conversation for display or logging
3. Format prompts with relevant context from our document processor
4. Provide methods to reset the conversation history when needed

By the end of this lesson, you'll have a fully functional chat engine that can be integrated with the document processor we built previously to create a complete RAG system. Let's get started!

Let's begin by setting up the basic structure of our ChatEngine class in Java. This class will encapsulate all the functionality needed for managing conversations with the language model.

import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;
import dev.langchain4j.model.openai.OpenAiChatRequestParameters;
import dev.langchain4j.data.message.SystemMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.data.message.ChatMessage;
import java.util.ArrayList;
import java.util.List;
import java.time.Duration;

public class ChatEngine {
    private ChatLanguageModel chatModel;
    private String systemMessage;
    // Optionally keep conversation history for display/logging only
    private List<ChatMessage> conversationHistory;

    public ChatEngine() {
        // Initialize the chat model using LangChain4j
        this.chatModel = OpenAiChatModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .baseUrl(System.getenv("OPENAI_API_BASE"))
            .modelName("gpt-5.6-luna")
            .defaultRequestParameters(OpenAiChatRequestParameters.builder().reasoningEffort("none").build())
            .temperature(0.7)
            .maxCompletionTokens(500)
            .timeout(Duration.ofSeconds(60))
            .build();
            
        this.systemMessage = "You are a helpful assistant that ONLY answers questions based on the " +
                "provided context. If no relevant context is provided, do NOT answer the " +
                "question and politely inform the user that you don't have the necessary " +
                "information to answer their question accurately.";

        // Optionally, keep conversation history for display/logging only
        this.conversationHistory = new ArrayList<>();
    }
}

Key points in this initialization:

1. Chat Model: We initialize chatModel using LangChain4j's OpenAiChatModel to create an instance for generating responses.
2. System Message: We define strict instructions that guide the AI's behavior, telling it to only answer questions based on the provided context and to politely decline if no relevant context is available.
3. Conversation History: We initialize an empty list to optionally keep track of the conversation for display or logging purposes. This history is not sent to the model in this implementation.

This structure ensures our chat engine can properly communicate with the language model while optionally maintaining a record of the conversation.