Introduction

Welcome back to Mastering Advanced AI Tooling in Codex. You've mastered capturing test failures and consulting the model for fix suggestions. Now we'll take the next step: building a complete engineer loop that not only plans changes but actually applies them, verifies the results, and documents the outcome.

This transforms Codex from an advisor into an autonomous agent that can execute complete fix cycles suitable for continuous integration systems.

The Engineer Loop Architecture

A production-grade automation loop follows five distinct stages:

  1. Collect Context - Gather repository state and test results
  2. Generate Structured Patch - Get machine-readable fixes from the model
  3. Apply Patch Safely - Modify source code using validated diffs
  4. Verify with Tests - Confirm the fix actually works
  5. Generate Report - Document everything for audit and review

Each stage produces artifacts stored in a timestamped directory, creating a complete audit trail. Let's build this step by step.

Collecting Repository Context

Effective automation requires precise understanding of the current state. We need two pieces of information: uncommitted changes already in the working directory and the output of failing tests.

import subprocess
import os
import json
from datetime import datetime
from openai import OpenAI

MAX_CHARS = 12000


def collect_context():
    """
    Gather current repository state: uncommitted changes and test results.
    
    Returns:
        Dictionary with git_diff, test_output, and test_exit_code
    """
    # Capture any uncommitted changes
    git_proc = subprocess.run(
        ["git", "diff", "HEAD"],
        capture_output=True,
        text=True,
        cwd="."
    )
    
    # Run the test suite
    test_proc = subprocess.run(
        ["yarn", "test:app", "--run", "packages/math/tests/point.test.ts"],
        capture_output=True,
        text=True,
        cwd="."
    )
    
    return {
        "git_diff": git_proc.stdout,
        "test_output": test_proc.stdout + "\n" + test_proc.stderr,
        "test_exit_code": test_proc.returncode
    }

By combining git diff with test execution, we give the model complete visibility into what's already modified versus what needs fixing. This context enables more precise patch generation.

Generating Structured Patches

The key insight for automation is requesting structured outputs instead of freeform text. We instruct the model to return only valid JSON with exactly two fields—plan and diff—so the response is machine-readable and deterministic to parse. We then validate the response by running json.loads(), with a small best-effort recovery if the model accidentally wraps the JSON in extra text.

There's one more practical concern: the model has no way to know the exact layout of your repo. If it guesses the wrong path in the diff headers (e.g. packages/math/point.ts instead of packages/math/src/point.ts), git apply will reject the patch with No such file or directory. To avoid this, we read the source file from disk and inline it into the prompt, and we tell the model exactly which path to use in the diff headers.

import os
import json
from openai import OpenAI

from .openai_helpers import extract_response_text

# Maximum characters to send to API to avoid context limits
MAX_CHARS = 12000

# Path (relative to repo root) of the file under test
SOURCE_FILE = "packages/math/src/point.ts"


def generate_patch_plan(context):
    """
    Consult Codex for a structured fix plan with a git-compatible diff.
    
    This function uses structured JSON prompting and robust parsing to ensure 
    machine-readable output that can be handled deterministically. 
    The prompt enforces:
    - A 'plan' field containing a brief summary
    - A 'diff' field containing a unified diff that git apply can use
    
    Args:
        context: Dictionary from collect_context() containing:
                - git_diff: Current uncommitted changes
                - test_output: Test failure messages
    
    Returns:
        Dictionary with keys:
        - plan: Brief summary of the proposed fix
        - diff: Unified diff format that git apply can process
    """
    # Read the real source file so the model sees exact line content / numbering
    repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
    source_path = os.path.join(repo_root, SOURCE_FILE)

    try:
        with open(source_path, "r", encoding="utf-8") as f:
            source_content = f.read()
    except FileNotFoundError:
        source_content = ""

    # Combine all context information
    combined_input = f"""Repository State:
{context['git_diff'] if context['git_diff'] else 'No uncommitted changes'}

Source File ({SOURCE_FILE}):
{source_content}

Test Failures:
{context['test_output']}"""

    # Truncate to avoid API limits (12K chars gives ~3K tokens)
    if len(combined_input) > MAX_CHARS:
        truncated = combined_input[-MAX_CHARS:]
    else:
        truncated = combined_input

    client = OpenAI()
    
    try:
        response = client.responses.create(
            model="gpt-5.3-codex",
            instructions=(
                "You are a senior engineer. Generate a minimal fix for the failing tests. "
                f"The source file lives at '{SOURCE_FILE}' — diff headers MUST use this exact path "
                f"(e.g. 'a/{SOURCE_FILE}'). "
                "Return ONLY valid JSON with two fields: "
                "'plan' (brief summary) and 'diff' (unified diff format that git apply can use). "
                "Do not include markdown, code fences, or any other text."
            ),
            input=truncated,
        )
        
        # Parse the JSON response
        result_text = extract_response_text(response).strip()
        try:
            return json.loads(result_text)
        except json.JSONDecodeError:
            # Best-effort recovery if the model wrapped JSON with extra text.
            start = result_text.find("{")
            end = result_text.rfind("}")
            if start != -1 and end != -1 and end > start:
                return json.loads(result_text[start : end + 1])
            raise
        
    except Exception as e:
        return {
            "plan": f"Error generating patch: {str(e)}",
            "diff": ""
        }

This gives us two artifacts we can drive the rest of the loop with:

  • plan: a brief summary we can log into the report
  • diff: a unified diff we can feed directly into git apply for safe, auditable code changes
Understanding Unified Diff Format

Before we apply patches, it's critical to understand the format git expects. A unified diff shows changes between two versions of a file. Here's what one looks like:

diff --git a/packages/math/src/point.ts b/packages/math/src/point.ts
index 1234567..abcdefg 100644
--- a/packages/math/src/point.ts
+++ b/packages/math/src/point.ts
@@ -12,7 +12,7 @@ export function pointRotateRads(
   const dx = point.x - center.x;
   const dy = point.y - center.y;
   
-  const cos = Math.cos(angle);
-  const sin = Math.sin(angle);
+  const cos = Math.cos(-angle);
+  const sin = Math.sin(-angle);
   
   return {

The format breaks down as:

  • Header (diff --git...): Identifies which file changed
  • Index line: Git metadata about the change
  • File markers (--- old, +++ new): Show before/after filenames
  • Hunk header (@@ -12,7 +12,7 @@): Line numbers where changes occur
  • Change lines: Lines starting with - are removed, + are added, (space) are context

This format is what git apply expects. The model generates this automatically when we request a "unified diff format" in our instructions. The beauty of this approach is that git handles all the actual file modification logic—we just validate and execute.

Applying Patches with Precision

Once we have a diff, we use git apply to modify the actual source files. The critical safety measure is validating the patch before applying it to avoid corrupting the working directory.

In a real repo setup (like excalidraw/), it's also important to:

  • Save the patch into the artifacts directory using an absolute path
  • Ensure the artifacts directory exists
  • Run git from the repository root (not from wherever the script happens to be invoked)

LLM-generated diffs are also frequently cosmetically imperfect — miscounted hunk headers (@@ -12,7 +12,7 @@ when the hunk is really 8 lines) and stray whitespace are extremely common, and vanilla git apply rejects both with corrupt patch at line N. We pass two extra flags to make git apply tolerant of these defects without sacrificing safety:

  • --recount auto-corrects mismatched hunk line counts
  • --whitespace=fix auto-corrects whitespace drift

Git still requires context lines to match the real file content, so the patch can't silently apply somewhere it shouldn't — these flags just stop punishing the model for cosmetic mistakes that don't change the semantic patch.

import os
import subprocess


def apply_patch(diff_text, artifact_path):
    """
    Write the diff to a file and apply it using git.
    
    This function implements a two-step safety process:
    1. Validate the patch can apply cleanly using --check
    2. Only apply the patch if validation succeeds
    
    This prevents partial modifications that could corrupt the working directory.
    
    Args:
        diff_text: The unified diff string from generate_patch_plan()
        artifact_path: Directory to store the patch file for audit trail
    
    Returns:
        True if patch applied successfully, False otherwise
    """
    if not diff_text or not diff_text.strip():
        print("No diff to apply")
        return False

    # Write patch to artifacts dir
    patch_file = os.path.abspath(os.path.join(artifact_path, "changes.diff"))
    os.makedirs(os.path.dirname(patch_file), exist_ok=True)

    with open(patch_file, "w", encoding="utf-8") as f:
        f.write(diff_text)

    print(f"Patch saved to {patch_file}")

    # Always run git from repo root (excalidraw/)
    repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))

    # --recount: fix mismatched hunk line counts (common in LLM-generated diffs)
    # --whitespace=fix: auto-correct whitespace errors
    apply_flags = ["--recount", "--whitespace=fix"]

    check_result = subprocess.run(
        ["git", "apply", "--check", *apply_flags, patch_file],
        capture_output=True,
        cwd=repo_root,
    )

    if check_result.returncode == 0:
        apply_result = subprocess.run(
            ["git", "apply", *apply_flags, patch_file],
            capture_output=True,
            cwd=repo_root,
        )

        if apply_result.returncode == 0:
            print("Patch applied successfully")
            return True
        else:
            print(
                f"Failed to apply patch: {apply_result.stderr.decode(errors='replace')}"
            )
            return False
    else:
        print(
            f"Patch validation failed: {check_result.stderr.decode(errors='replace')}"
        )
        return False

This two-step process (--check then actual apply) ensures we never partially modify files. If validation fails, no changes occur. Running from a known repo root also prevents confusing failures caused by executing git in the wrong working directory.

Verifying Results with Tests

After applying the patch, we rerun the test suite to verify the fix worked. This is the critical validation step that closes the automation loop.

def verify_fix():
    """
    Run the test suite again to verify the patch fixed the issues.
    
    Returns:
        Tuple of (success: bool, output: str)
    """
    print("Running tests to verify fix...")
    
    result = subprocess.run(
        ["yarn", "test:app", "--run", "packages/math/tests/point.test.ts"],
        capture_output=True,
        text=True,
        cwd="."
    )
    
    output = result.stdout + "\n" + result.stderr
    success = result.returncode == 0
    
    if success:
        print("✓ Tests passed! Fix verified.")
    else:
        print("✗ Tests still failing after patch.")
    
    return success, output

The exit code provides an objective measure of success. Zero means the fix worked; non-zero means we need to try again or escalate to human review.

Handling Failure Scenarios

In this lesson, we focus on the success path where patches apply cleanly and tests pass. However, in production systems you'll encounter scenarios where:

  • Tests still fail after the patch is applied
  • The patch breaks other tests that were previously passing
  • The patch conflicts with uncommitted changes in the working directory

For these situations, you'd add rollback logic using git stash or work on isolated feature branches before merging to main. You might also implement retry logic with different prompts or escalate to human review after N failed attempts.

Unit 4 will cover these safety and recovery patterns in depth. For now, understanding the deterministic success path gives you the foundation to build robust error handling later.

Generating Comprehensive Reports

The final stage documents everything: what was attempted, what changed, and whether it succeeded. This creates the audit trail required for production automation.

def generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir):
    """
    Create a comprehensive Markdown report of the engineer loop run.
    
    Args:
        context: Original context from collect_context()
        plan: The generated patch plan dictionary
        patch_applied: Whether the patch was applied successfully
        tests_passed: Whether verification tests passed
        test_output: Output from verification test run
        run_dir: Path to artifact directory
    
    Returns:
        Path to the generated report
    """
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    
    # Capture final repository state
    final_diff = subprocess.run(
        ["git", "diff", "HEAD"],
        capture_output=True,
        text=True,
        cwd="."
    ).stdout
    
    # Truncate long outputs for report readability (keep full versions in artifact files)
    report = f"""# Engineer Loop Report

**Timestamp:** {timestamp}  
**Status:** {'✓ SUCCESS' if tests_passed else '✗ FAILED'}

## Summary

- Original test exit code: {context['test_exit_code']}
- Patch applied: {patch_applied}
- Tests passed after fix: {tests_passed}

## Plan

{plan.get('plan', 'No plan generated')}

## Changes Applied

{plan.get('diff', 'No diff generated')[:2000]}

## Verification Test Output

{test_output[:1000]}

## Final Repository State

{final_diff[:2000] if final_diff else 'No uncommitted changes'}

## Commands Executed

1. `git diff HEAD` (collect context)
2. `yarn test:app --run packages/math/tests/point.test.ts` (initial test run)
3. `git apply changes.diff` (apply patch)
4. `yarn test:app --run packages/math/tests/point.test.ts` (verify fix)

## Artifacts

- Full patch: `changes.diff`
- This report: `REPORT.md`
"""
    
    report_path = os.path.join(run_dir, "REPORT.md")
    with open(report_path, "w", encoding="utf-8") as f:
        f.write(report)
    
    print(f"Report saved to {report_path}")
    return report_path

By capturing both the planned diff and the final repository state, we create complete visibility into what the automation attempted and achieved.

Orchestrating the Complete Pipeline

Now we connect all stages into a single execution flow. Each stage feeds into the next, creating the deterministic loop that makes automation reliable.

def create_artifact_dir():
    """Create timestamped directory for this run's artifacts."""
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    run_dir = f"artifacts/{timestamp}"
    os.makedirs(run_dir, exist_ok=True)
    return run_dir


def engineer_loop():
    """
    Execute the complete engineer loop:
    collect → plan → patch → test → report
    """
    print("=== Engineer Loop Starting ===\n")
    
    # Stage 1: Collect context
    print("Stage 1: Collecting repository context...")
    context = collect_context()
    
    if context["test_exit_code"] == 0:
        print("All tests passing - no action needed")
        return
    
    print(f"Tests failing (exit code {context['test_exit_code']})\n")
    
    # Stage 2: Generate structured patch plan
    print("Stage 2: Generating patch plan...")
    plan = generate_patch_plan(context)
    print(f"Plan: {plan.get('plan', 'Error')}\n")
    
    # Create artifact directory for this run
    run_dir = create_artifact_dir()
    print(f"Artifacts will be saved to: {run_dir}\n")
    
    # Stage 3: Apply the patch
    print("Stage 3: Applying patch...")
    patch_applied = apply_patch(plan.get("diff", ""), run_dir)
    print()
    
    # Stage 4: Verify the fix
    print("Stage 4: Verifying fix...")
    if patch_applied:
        tests_passed, test_output = verify_fix()
    else:
        tests_passed = False
        test_output = "Patch was not applied - verification skipped"
    print()
    
    # Stage 5: Generate comprehensive report
    print("Stage 5: Generating report...")
    generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir)
    
    print(f"\n=== Engineer Loop Complete ===")
    print(f"Results saved to: {run_dir}")


if __name__ == "__main__":
    engineer_loop()
Example Output

When you run this complete pipeline, you'll see output like:

=== Engineer Loop Starting ===

Stage 1: Collecting repository context...
Tests failing (exit code 1)

Stage 2: Generating patch plan...
Plan: Fix the pointRotateRads function by correcting the rotation matrix formula

Artifacts will be saved to: artifacts/20260218_154312

Stage 3: Applying patch...
Patch saved to artifacts/20260218_154312/changes.diff
Patch applied successfully

Stage 4: Verifying fix...
Running tests to verify fix...
✓ Tests passed! Fix verified.

Stage 5: Generating report...
Report saved to artifacts/20260218_154312/REPORT.md

=== Engineer Loop Complete ===
Results saved to: artifacts/20260218_154312
Understanding Determinism in Automation

The key to reliable automation is determinism: given the same inputs, the pipeline produces consistent, predictable results. We achieve this through:

  • Structured outputs (structured JSON prompting and robust parsing eliminate ambiguity)
  • Validated patches (two-step git apply prevents partial modifications)
  • Exit codes (objective pass/fail signals)
  • Timestamped artifacts (isolated storage prevents run-to-run interference)

This design makes the loop suitable for CI/CD systems where failures must be reproducible and outcomes must be verifiable.

Conclusion and Next Steps

You've now built a complete engineer loop that autonomously:

  • Collects repository context (git state + test failures)
  • Generates structured patches using structured JSON prompting and robust parsing
  • Applies changes safely using validated git operations
  • Verifies fixes by rerunning tests
  • Persists comprehensive reports in timestamped directories

This is the foundation of autonomous development agents. In the practice section, you'll implement this pipeline yourself in the excalidraw repository, gaining hands-on experience with deterministic AI-driven workflows.

Previous
Next
Sign up
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal