Beyond Basic RAG: Enhancing RAG Systems with Constrained Generation in JavaScript

Introduction

Welcome to the first lesson of the " Beyond Basic RAG: Improving our Pipeline " course, part of the " Foundations of RAG Systems " course path! In previous courses, you delved into the basics of Retrieval-Augmented Generation (RAG), exploring text representation with a focus on embeddings and vector databases. In this course, we'll embark on an exciting journey to enhance our RAG systems with advanced techniques. Our focus in this initial lesson is on constrained generation, a powerful method to ensure that language model responses remain anchored in the retrieved context, avoiding speculation or unrelated content. Get ready to elevate your RAG skills and build more reliable systems!

Theoretical Foundations of Constrained Generation

When employing large language models (LLMs) in real-world applications, accuracy and fidelity to a trusted dataset are paramount. Even advanced LLMs can produce incorrect or fabricated information — often termed “hallucinations.” This is where constrained generation becomes indispensable. In essence, it is a form of advanced prompt engineering : we carefully craft instructions so the LLM only responds using the retrieved information or provides disclaimers when insufficient data is found. By shaping the prompt and enforcing rule-based fallback mechanisms, we instruct the LLM to: Use only the data you supply (the “retrieved context”). Provide disclaimers or refusal messages when context is insufficient. Optionally cite which part of the content it used. The result is a system less prone to made-up facts and more consistent with the original knowledge source.

Why Constrained Generation Is Important

LLM hallucination can be quite misleading. Imagine a scenario where your application confidently presents policies or regulations not present in your knowledge base. This can create confusion or even compliance issues. With constrained generation : The model remains grounded in the retrieved context only. Uncertain or unavailable information triggers a fallback message like “No sufficient data.” You can require the model to cite lines to verify the source of the answer, building trust with users.

Defining the Constrained Generation Function

We'll start by defining a function that enforces these constraints using JavaScript: JavaScriptfunction generateWithConstraints(query, retrievedContext, strategy = "base") { /** * Thoroughly enforce model reliance on 'retrievedContext' when answering 'query.' * * The 'strategy' parameter allows for different prompt template variations: * 1) Base approach: Provide context, instruct LLM not to use outside info. * 2) Strict approach: Provide context with explicit disclaimers if the answer is not found. * 3) Citation approach: Provide context, then request the LLM to cite the relevant lines. * * Robust fallback: * - If 'retrievedContext' is empty, respond with an apology or neutral statement. * - Optionally log each stage for debugging or performance analysis. */ // Provide a safe fallback if no context is retrieved if (!retrievedContext.trim()) { return ["I'm sorry, but I couldn't find any relevant information.", "No context used."]; } // Choose a prompt template based on strategy let prompt; if (strategy === "base") { // Base approach: instruct to use the context and not rely on external info prompt = `Use the following context to answer the question in a concise manner.\n\nContext:\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`; } else if (strategy === "strict") { // Strict approach: explicitly disallow info beyond the provided context prompt = `You must ONLY use the context provided below. If you cannot find the answer in the context, say: 'No sufficient data'.\nDo not provide any information not found in the context.\n\nContext:\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`; } else if (strategy === "cite") { // Citation approach: require references to lines used prompt = `Answer strictly from the provided context, and list the lines you used as evidence with 'Cited lines:'.\nIf the context does not contain the information, respond with: 'Not available in the retrieved texts.'\n\nProvided context (label lines as needed):\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`; } // ... } Here's how it works: If no context was retrieved, the function immediately returns a fallback response. Different strategies (base, strict, cite) each construct a slightly different prompt. This lets you control how rigidly the model relies on the retrieved context: Base Approach: This strategy provides the retrieved context and instructs the LLM not to use any external information. It is a straightforward method that ensures the model focuses on the given context but allows for some flexibility in interpretation. Strict Approach: This strategy explicitly disallows the use of any information beyond the provided context. If the answer cannot be found within the context, the model is instructed to respond with "No sufficient data." This approach is ideal for scenarios where accuracy and adherence to the provided information are critical. Citation Approach: This strategy requires the model to answer strictly from the provided context and to list the lines used as evidence with "Cited lines:". If the context does not contain the necessary information, the model responds with "Not available in the retrieved texts." This approach is useful for applications where transparency and traceability of the information source are important.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

JavaScript

function generateWithConstraints(query, retrievedContext, strategy = "base") {
    /**
     * Thoroughly enforce model reliance on 'retrievedContext' when answering 'query.'
     *
     * The 'strategy' parameter allows for different prompt template variations:
     *   1) Base approach: Provide context, instruct LLM not to use outside info.
     *   2) Strict approach: Provide context with explicit disclaimers if the answer is not found.
     *   3) Citation approach: Provide context, then request the LLM to cite the relevant lines.
     *
     * Robust fallback:
     *   - If 'retrievedContext' is empty, respond with an apology or neutral statement.
     *   - Optionally log each stage for debugging or performance analysis.
     */

    // Provide a safe fallback if no context is retrieved
    if (!retrievedContext.trim()) {
        return ["I'm sorry, but I couldn't find any relevant information.", "No context used."];
    }

    // Choose a prompt template based on strategy
    let prompt;
    if (strategy === "base") {
        // Base approach: instruct to use the context and not rely on external info
        prompt = `Use the following context to answer the question in a concise manner.\n\nContext:\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`;
    } else if (strategy === "strict") {
        // Strict approach: explicitly disallow info beyond the provided context
        prompt = `You must ONLY use the context provided below. If you cannot find the answer in the context, say: 'No sufficient data'.\nDo not provide any information not found in the context.\n\nContext:\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`;
    } else if (strategy === "cite") {
        // Citation approach: require references to lines used
        prompt = `Answer strictly from the provided context, and list the lines you used as evidence with 'Cited lines:'.\nIf the context does not contain the information, respond with: 'Not available in the retrieved texts.'\n\nProvided context (label lines as needed):\n${retrievedContext}\nQuestion: '${query}'\nAnswer:`;
    }
    // ...
}

JavaScript

    // ...
    const response = await getLLMResponse(prompt);

    // Attempt to parse out 'Cited lines:' if present
    const segments = response.split("Cited lines:");
    if (segments.length === 2) {
        const [answerPart, usedContextPart] = segments;
        return [answerPart.trim(), usedContextPart.trim()];
    } else {
        return [response.trim(), "No explicit lines cited."];
    }

    // ...
    const response = await getLLMResponse(prompt);

    // Attempt to parse out 'Cited lines:' if present
    const segments = response.split("Cited lines:");
    if (segments.length === 2) {
        const [answerPart, usedContextPart] = segments;
        return [answerPart.trim(), usedContextPart.trim()];
    } else {
        return [response.trim(), "No explicit lines cited."];
    }

JavaScript

// Example demonstration of retrieval and constrained generation

// 1. Load and chunk a corpus
const chunkedDocs = loadAndChunkCorpus("data/corpus.json");

// 2. Build a collection in a vector database
const collection = await buildChromaCollection(chunkedDocs, "corpus_collection");

// 3. Run a sample query
const query = "Highlight the main policies that apply to employees.";
const retrievalResults = await collection.query({ queryTexts: [query], nResults: 2 });

// 4. Construct the retrieved context from top matches
let retrievedContext = "";
if (retrievalResults.documents[0]) {
    // Join the relevant chunks into one convenient string
    retrievedContext = retrievalResults.documents[0].map(docText => `- ${docText}`).join("\n");
}

// 5. Execute constrained generation function based on chosen strategy
const strategy = "strict";
const [answer, usedContext] = generateWithConstraints(query, retrievedContext, strategy);

// Example demonstration of retrieval and constrained generation

// 1. Load and chunk a corpus
const chunkedDocs = loadAndChunkCorpus("data/corpus.json");

// 2. Build a collection in a vector database
const collection = await buildChromaCollection(chunkedDocs, "corpus_collection");

// 3. Run a sample query
const query = "Highlight the main policies that apply to employees.";
const retrievalResults = await collection.query({ queryTexts: [query], nResults: 2 });

// 4. Construct the retrieved context from top matches
let retrievedContext = "";
if (retrievalResults.documents[0]) {
    // Join the relevant chunks into one convenient string
    retrievedContext = retrievalResults.documents[0].map(docText => `- ${docText}`).join("\n");
}

// 5. Execute constrained generation function based on chosen strategy
const strategy = "strict";
const [answer, usedContext] = generateWithConstraints(query, retrievedContext, strategy);

JavaScript

    // ...
    const response = await getLLMResponse(prompt);

    // Attempt to parse out 'Cited lines:' if present
    const segments = response.split("Cited lines:");
    if (segments.length === 2) {
        const [answerPart, usedContextPart] = segments;
        return [answerPart.trim(), usedContextPart.trim()];
    } else {
        return [response.trim(), "No explicit lines cited."];
    }

JavaScript

// Example demonstration of retrieval and constrained generation

// 1. Load and chunk a corpus
const chunkedDocs = loadAndChunkCorpus("data/corpus.json");

// 2. Build a collection in a vector database
const collection = await buildChromaCollection(chunkedDocs, "corpus_collection");

// 3. Run a sample query
const query = "Highlight the main policies that apply to employees.";
const retrievalResults = await collection.query({ queryTexts: [query], nResults: 2 });

// 4. Construct the retrieved context from top matches
let retrievedContext = "";
if (retrievalResults.documents[0]) {
    // Join the relevant chunks into one convenient string
    retrievedContext = retrievalResults.documents[0].map(docText => `- ${docText}`).join("\n");
}

// 5. Execute constrained generation function based on chosen strategy
const strategy = "strict";
const [answer, usedContext] = generateWithConstraints(query, retrievedContext, strategy);

Introduction

Theoretical Foundations of Constrained Generation

Why Constrained Generation Is Important

Defining the Constrained Generation Function

Join the 1M+ learners on CodeSignal

Generating the Final Response

Demonstration of Retrieval and Constrained Generation

Practical Example: A Policy FAQ Bot

Conclusion and Next Steps