Implementing Preprocessing Routines for Code Translation

Welcome to the third lesson of Laying the Foundations for Code Translation with Haystack! So far, you've learned the basics of Haystack and built a simple code translation pipeline. Now, we're ready to make our translator much more powerful by adding preprocessing routines. This is a key step that will help our system handle real-world, messy inputs — making it more reliable and user-friendly.

In this lesson, you'll learn how to create a custom Haystack component that extracts clean code from mixed input and automatically detects the programming language. By the end, you'll see how these improvements make your code translator smarter and more robust.

Let's take a moment to understand why preprocessing is so important for code translation. In real scenarios, users rarely provide perfectly formatted code. Instead, you might see:

• Code snippets mixed with explanations or natural language requests;
• Code embedded in markdown or copied from documentation;
• Unclear or missing information about the programming language.

If we try to translate such input directly, the results can be confusing or even incorrect. Preprocessing helps by:

• Extracting just the code, ignoring any extra text;
• Identifying the programming language automatically;
• Ensuring the code is in a consistent format for translation.

By handling these challenges upfront, we make the translation process smoother and more reliable for everyone.

Haystack makes it easy to extend pipelines with your own logic using the @component decorator. Let's start by outlining a custom component that will handle our preprocessing tasks.

from haystack import component
from haystack.components.generators.openai import OpenAIGenerator

@component
class CodePreprocessor:
    def __init__(self, model_name: str = "gpt-4o-mini"):
        # Initialize the LLM for use in preprocessing
        self.llm = OpenAIGenerator(model=model_name)

    @component.output_types(code_block=str, source_lang=str)
    def run(self, text: str):
        # Preprocessing logic will go here
        pass

Here's what's happening:

• The @component decorator tells Haystack this is a reusable pipeline component.
• We set up an LLM to help with code understanding.
• The run method will take in raw text and output both the extracted code and the detected language.
• The @component.output_types decorator specifies the names and types of the outputs produced by the run method, making it clear to Haystack (and to other developers) what data this component will return.

This structure lets us plug our preprocessor directly into any Haystack pipeline, making it easy to reuse and maintain.