Welcome back to Mastering Advanced AI Tooling in Codex. You've mastered capturing test failures and consulting the model for fix suggestions. Now we'll take the next step: building a complete engineer loop that not only plans changes but actually applies them, verifies the results, and documents the outcome.
This transforms Codex from an advisor into an autonomous agent that can execute complete fix cycles suitable for continuous integration systems.
The Engineer Loop Architecture
A production-grade automation loop follows five distinct stages:
Collect Context - Gather repository state and test results
Generate Structured Patch - Get machine-readable fixes from the model
Apply Patch Safely - Modify source code using validated diffs
Verify with Tests - Confirm the fix actually works
Generate Report - Document everything for audit and review
Each stage produces artifacts stored in a timestamped directory, creating a complete audit trail. Let's build this step by step.
Collecting Repository Context
Effective automation requires precise understanding of the current state. We need two pieces of information: uncommitted changes already in the working directory and the output of failing tests.
Python
import subprocessimport osimport jsonfrom datetime import datetimefrom openai import OpenAIMAX_CHARS = 12000def collect_context(): """ Gather current repository state: uncommitted changes and test results. Returns: Dictionary with git_diff, test_output, and test_exit_code """ # Capture any uncommitted changes git_proc = subprocess.run( ["git", "diff", "HEAD"], capture_output=True, text=True, cwd="." ) # Run the test suite test_proc = subprocess.run( ["yarn", "test:app", "--run", "packages/math/tests/point.test.ts"], capture_output=True, text=True, cwd="." ) return { "git_diff": git_proc.stdout, "test_output": test_proc.stdout + "\n" + test_proc.stderr, "test_exit_code": test_proc.returncode }
By combining git diff with test execution, we give the model complete visibility into what's already modified versus what needs fixing. This context enables more precise patch generation.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Generating Structured Patches
The key insight for automation is requesting structured outputs instead of freeform text. We instruct the model to return only valid JSON with exactly two fields—plan and diff—so the response is machine-readable and deterministic to parse. We then validate the response by running json.loads(), with a small best-effort recovery if the model accidentally wraps the JSON in extra text.
There's one more practical concern: the model has no way to know the exact layout of your repo. If it guesses the wrong path in the diff headers (e.g. packages/math/point.ts instead of packages/math/src/point.ts), git apply will reject the patch with No such file or directory. To avoid this, we read the source file from disk and inline it into the prompt, and we tell the model exactly which path to use in the diff headers.
import osimport jsonfrom openai import OpenAIfrom .openai_helpers import extract_response_text# Maximum characters to send to API to avoid context limitsMAX_CHARS = 12000# Path (relative to repo root) of the file under testSOURCE_FILE = "packages/math/src/point.ts"def generate_patch_plan(context): """ Consult Codex for a structured fix plan with a git-compatible diff. This function uses structured JSON prompting and robust parsing to ensure machine-readable output that can be handled deterministically. The prompt enforces: - A 'plan' field containing a brief summary - A 'diff' field containing a unified diff that git apply can use Args: context: Dictionary from collect_context() containing: - git_diff: Current uncommitted changes - test_output: Test failure messages Returns: Dictionary with keys: - plan: Brief summary of the proposed fix - diff: Unified diff format that git apply can process """ # Read the real source file so the model sees exact line content / numbering repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) source_path = os.path.join(repo_root, SOURCE_FILE) try: with open(source_path, "r", encoding="utf-8") as f: source_content = f.read() except FileNotFoundError: source_content = "" # Combine all context information combined_input = f"""Repository State:{context['git_diff'] if context['git_diff'] else 'No uncommitted changes'}Source File ({SOURCE_FILE}):{source_content}Test Failures:{context['test_output']}""" # Truncate to avoid API limits (12K chars gives ~3K tokens) if len(combined_input) > MAX_CHARS: truncated = combined_input[-MAX_CHARS:] else: truncated = combined_input client = OpenAI() try: response = client.responses.create( model="gpt-6-luna", instructions=( "You are a senior engineer. Generate a minimal fix for the failing tests. " f"The source file lives at '{SOURCE_FILE}' — diff headers MUST use this exact path " f"(e.g. 'a/{SOURCE_FILE}'). " "Return ONLY valid JSON with two fields: " "'plan' (brief summary) and 'diff' (unified diff format that git apply can use). " "Do not include markdown, code fences, or any other text." ), input=truncated, ) # Parse the JSON response result_text = extract_response_text(response).strip() try: return json.loads(result_text) except json.JSONDecodeError: # Best-effort recovery if the model wrapped JSON with extra text. start = result_text.find("{") end = result_text.rfind("}") if start != -1 and end != -1 and end > start: return json.loads(result_text[start : end + 1]) raise except Exception as e: return { "plan": f"Error generating patch: {str(e)}", "diff": "" }
This gives us two artifacts we can drive the rest of the loop with:
plan: a brief summary we can log into the report
diff: a unified diff we can feed directly into git apply for safe, auditable code changes
Understanding Unified Diff Format
Before we apply patches, it's critical to understand the format git expects. A unified diff shows changes between two versions of a file. Here's what one looks like:
diff --git a/packages/math/src/point.ts b/packages/math/src/point.tsindex 1234567..abcdefg 100644--- a/packages/math/src/point.ts+++ b/packages/math/src/point.ts@@ -12,7 +12,7 @@ export function pointRotateRads( const dx = point.x - center.x; const dy = point.y - center.y;- const cos = Math.cos(angle);- const sin = Math.sin(angle);+ const cos = Math.cos(-angle);+ const sin = Math.sin(-angle); return {
The format breaks down as:
Header (diff --git...): Identifies which file changed
Index line: Git metadata about the change
File markers (--- old, +++ new): Show before/after filenames
Hunk header (@@ -12,7 +12,7 @@): Line numbers where changes occur
Change lines: Lines starting with - are removed, + are added, (space) are context
This format is what git apply expects. The model generates this automatically when we request a "unified diff format" in our instructions. The beauty of this approach is that git handles all the actual file modification logic—we just validate and execute.
Applying Patches with Precision
Once we have a diff, we use git apply to modify the actual source files. The critical safety measure is validating the patch before applying it to avoid corrupting the working directory.
In a real repo setup (like excalidraw/), it's also important to:
Save the patch into the artifacts directory using an absolute path
Ensure the artifacts directory exists
Run git from the repository root (not from wherever the script happens to be invoked)
LLM-generated diffs are also frequently cosmetically imperfect — miscounted hunk headers (@@ -12,7 +12,7 @@ when the hunk is really 8 lines) and stray whitespace are extremely common, and vanilla git apply rejects both with corrupt patch at line N. We pass two extra flags to make git apply tolerant of these defects without sacrificing safety:
--recount auto-corrects mismatched hunk line counts
--whitespace=fix auto-corrects whitespace drift
Git still requires context lines to match the real file content, so the patch can't silently apply somewhere it shouldn't — these flags just stop punishing the model for cosmetic mistakes that don't change the semantic patch.
import osimport subprocessdef apply_patch(diff_text, artifact_path): """ Write the diff to a file and apply it using git. This function implements a two-step safety process: 1. Validate the patch can apply cleanly using --check 2. Only apply the patch if validation succeeds This prevents partial modifications that could corrupt the working directory. Args: diff_text: The unified diff string from generate_patch_plan() artifact_path: Directory to store the patch file for audit trail Returns: True if patch applied successfully, False otherwise """ if not diff_text or not diff_text.strip(): print("No diff to apply") return False # Write patch to artifacts dir patch_file = os.path.abspath(os.path.join(artifact_path, "changes.diff")) os.makedirs(os.path.dirname(patch_file), exist_ok=True) with open(patch_file, "w", encoding="utf-8") as f: f.write(diff_text) print(f"Patch saved to {patch_file}") # Always run git from repo root (excalidraw/) repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) # --recount: fix mismatched hunk line counts (common in LLM-generated diffs) # --whitespace=fix: auto-correct whitespace errors apply_flags = ["--recount", "--whitespace=fix"] check_result = subprocess.run( ["git", "apply", "--check", *apply_flags, patch_file], capture_output=True, cwd=repo_root, ) if check_result.returncode == 0: apply_result = subprocess.run( ["git", "apply", *apply_flags, patch_file], capture_output=True, cwd=repo_root, ) if apply_result.returncode == 0: print("Patch applied successfully") return True else: print( f"Failed to apply patch: {apply_result.stderr.decode(errors='replace')}" ) return False else: print( f"Patch validation failed: {check_result.stderr.decode(errors='replace')}" ) return False
This two-step process (--check then actual apply) ensures we never partially modify files. If validation fails, no changes occur. Running from a known repo root also prevents confusing failures caused by executing git in the wrong working directory.
Verifying Results with Tests
After applying the patch, we rerun the test suite to verify the fix worked. This is the critical validation step that closes the automation loop.
def verify_fix(): """ Run the test suite again to verify the patch fixed the issues. Returns: Tuple of (success: bool, output: str) """ print("Running tests to verify fix...") result = subprocess.run( ["yarn", "test:app", "--run", "packages/math/tests/point.test.ts"], capture_output=True, text=True, cwd="." ) output = result.stdout + "\n" + result.stderr success = result.returncode == 0 if success: print("✓ Tests passed! Fix verified.") else: print("✗ Tests still failing after patch.") return success, output
The exit code provides an objective measure of success. Zero means the fix worked; non-zero means we need to try again or escalate to human review.
Handling Failure Scenarios
In this lesson, we focus on the success path where patches apply cleanly and tests pass. However, in production systems you'll encounter scenarios where:
Tests still fail after the patch is applied
The patch breaks other tests that were previously passing
The patch conflicts with uncommitted changes in the working directory
For these situations, you'd add rollback logic using git stash or work on isolated feature branches before merging to main. You might also implement retry logic with different prompts or escalate to human review after N failed attempts.
Unit 4 will cover these safety and recovery patterns in depth. For now, understanding the deterministic success path gives you the foundation to build robust error handling later.
Generating Comprehensive Reports
The final stage documents everything: what was attempted, what changed, and whether it succeeded. This creates the audit trail required for production automation.
def generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir): """ Create a comprehensive Markdown report of the engineer loop run. Args: context: Original context from collect_context() plan: The generated patch plan dictionary patch_applied: Whether the patch was applied successfully tests_passed: Whether verification tests passed test_output: Output from verification test run run_dir: Path to artifact directory Returns: Path to the generated report """ timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Capture final repository state final_diff = subprocess.run( ["git", "diff", "HEAD"], capture_output=True, text=True, cwd="." ).stdout # Truncate long outputs for report readability (keep full versions in artifact files) report = f"""# Engineer Loop Report**Timestamp:** {timestamp} **Status:** {'✓ SUCCESS' if tests_passed else '✗ FAILED'}## Summary- Original test exit code: {context['test_exit_code']}- Patch applied: {patch_applied}- Tests passed after fix: {tests_passed}## Plan{plan.get('plan', 'No plan generated')}## Changes Applied{plan.get('diff', 'No diff generated')[:2000]}## Verification Test Output{test_output[:1000]}## Final Repository State{final_diff[:2000] if final_diff else 'No uncommitted changes'}## Commands Executed1. `git diff HEAD` (collect context)2. `yarn test:app --run packages/math/tests/point.test.ts` (initial test run)3. `git apply changes.diff` (apply patch)4. `yarn test:app --run packages/math/tests/point.test.ts` (verify fix)## Artifacts- Full patch: `changes.diff`- This report: `REPORT.md`""" report_path = os.path.join(run_dir, "REPORT.md") with open(report_path, "w", encoding="utf-8") as f: f.write(report) print(f"Report saved to {report_path}") return report_path
By capturing both the planned diff and the final repository state, we create complete visibility into what the automation attempted and achieved.
Orchestrating the Complete Pipeline
Now we connect all stages into a single execution flow. Each stage feeds into the next, creating the deterministic loop that makes automation reliable.
def create_artifact_dir(): """Create timestamped directory for this run's artifacts.""" timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") run_dir = f"artifacts/{timestamp}" os.makedirs(run_dir, exist_ok=True) return run_dirdef engineer_loop(): """ Execute the complete engineer loop: collect → plan → patch → test → report """ print("=== Engineer Loop Starting ===\n") # Stage 1: Collect context print("Stage 1: Collecting repository context...") context = collect_context() if context["test_exit_code"] == 0: print("All tests passing - no action needed") return print(f"Tests failing (exit code {context['test_exit_code']})\n") # Stage 2: Generate structured patch plan print("Stage 2: Generating patch plan...") plan = generate_patch_plan(context) print(f"Plan: {plan.get('plan', 'Error')}\n") # Create artifact directory for this run run_dir = create_artifact_dir() print(f"Artifacts will be saved to: {run_dir}\n") # Stage 3: Apply the patch print("Stage 3: Applying patch...") patch_applied = apply_patch(plan.get("diff", ""), run_dir) print() # Stage 4: Verify the fix print("Stage 4: Verifying fix...") if patch_applied: tests_passed, test_output = verify_fix() else: tests_passed = False test_output = "Patch was not applied - verification skipped" print() # Stage 5: Generate comprehensive report print("Stage 5: Generating report...") generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir) print(f"\n=== Engineer Loop Complete ===") print(f"Results saved to: {run_dir}")if __name__ == "__main__": engineer_loop()
Example Output
When you run this complete pipeline, you'll see output like:
=== Engineer Loop Starting ===Stage 1: Collecting repository context...Tests failing (exit code 1)Stage 2: Generating patch plan...Plan: Fix the pointRotateRads function by correcting the rotation matrix formulaArtifacts will be saved to: artifacts/20260218_154312Stage 3: Applying patch...Patch saved to artifacts/20260218_154312/changes.diffPatch applied successfullyStage 4: Verifying fix...Running tests to verify fix...✓ Tests passed! Fix verified.Stage 5: Generating report...Report saved to artifacts/20260218_154312/REPORT.md=== Engineer Loop Complete ===Results saved to: artifacts/20260218_154312
Understanding Determinism in Automation
The key to reliable automation is determinism: given the same inputs, the pipeline produces consistent, predictable results. We achieve this through:
Structured outputs (structured JSON prompting and robust parsing eliminate ambiguity)
This design makes the loop suitable for CI/CD systems where failures must be reproducible and outcomes must be verifiable.
Conclusion and Next Steps
You've now built a complete engineer loop that autonomously:
Collects repository context (git state + test failures)
Generates structured patches using structured JSON prompting and robust parsing
Applies changes safely using validated git operations
Verifies fixes by rerunning tests
Persists comprehensive reports in timestamped directories
This is the foundation of autonomous development agents. In the practice section, you'll implement this pipeline yourself in the excalidraw repository, gaining hands-on experience with deterministic AI-driven workflows.
import osimport jsonfrom openai import OpenAIfrom .openai_helpers import extract_response_text# Maximum characters to send to API to avoid context limitsMAX_CHARS = 12000# Path (relative to repo root) of the file under testSOURCE_FILE = "packages/math/src/point.ts"def generate_patch_plan(context): """ Consult Codex for a structured fix plan with a git-compatible diff. This function uses structured JSON prompting and robust parsing to ensure machine-readable output that can be handled deterministically. The prompt enforces: - A 'plan' field containing a brief summary - A 'diff' field containing a unified diff that git apply can use Args: context: Dictionary from collect_context() containing: - git_diff: Current uncommitted changes - test_output: Test failure messages Returns: Dictionary with keys: - plan: Brief summary of the proposed fix - diff: Unified diff format that git apply can process """ # Read the real source file so the model sees exact line content / numbering repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) source_path = os.path.join(repo_root, SOURCE_FILE) try: with open(source_path, "r", encoding="utf-8") as f: source_content = f.read() except FileNotFoundError: source_content = "" # Combine all context information combined_input = f"""Repository State:{context['git_diff'] if context['git_diff'] else 'No uncommitted changes'}Source File ({SOURCE_FILE}):{source_content}Test Failures:{context['test_output']}""" # Truncate to avoid API limits (12K chars gives ~3K tokens) if len(combined_input) > MAX_CHARS: truncated = combined_input[-MAX_CHARS:] else: truncated = combined_input client = OpenAI() try: response = client.responses.create( model="gpt-6-luna", instructions=( "You are a senior engineer. Generate a minimal fix for the failing tests. " f"The source file lives at '{SOURCE_FILE}' — diff headers MUST use this exact path " f"(e.g. 'a/{SOURCE_FILE}'). " "Return ONLY valid JSON with two fields: " "'plan' (brief summary) and 'diff' (unified diff format that git apply can use). " "Do not include markdown, code fences, or any other text." ), input=truncated, ) # Parse the JSON response result_text = extract_response_text(response).strip() try: return json.loads(result_text) except json.JSONDecodeError: # Best-effort recovery if the model wrapped JSON with extra text. start = result_text.find("{") end = result_text.rfind("}") if start != -1 and end != -1 and end > start: return json.loads(result_text[start : end + 1]) raise except Exception as e: return { "plan": f"Error generating patch: {str(e)}", "diff": "" }
Diff
diff --git a/packages/math/src/point.ts b/packages/math/src/point.tsindex 1234567..abcdefg 100644--- a/packages/math/src/point.ts+++ b/packages/math/src/point.ts@@ -12,7 +12,7 @@ export function pointRotateRads( const dx = point.x - center.x; const dy = point.y - center.y;- const cos = Math.cos(angle);- const sin = Math.sin(angle);+ const cos = Math.cos(-angle);+ const sin = Math.sin(-angle); return {
Python
import osimport subprocessdef apply_patch(diff_text, artifact_path): """ Write the diff to a file and apply it using git. This function implements a two-step safety process: 1. Validate the patch can apply cleanly using --check 2. Only apply the patch if validation succeeds This prevents partial modifications that could corrupt the working directory. Args: diff_text: The unified diff string from generate_patch_plan() artifact_path: Directory to store the patch file for audit trail Returns: True if patch applied successfully, False otherwise """ if not diff_text or not diff_text.strip(): print("No diff to apply") return False # Write patch to artifacts dir patch_file = os.path.abspath(os.path.join(artifact_path, "changes.diff")) os.makedirs(os.path.dirname(patch_file), exist_ok=True) with open(patch_file, "w", encoding="utf-8") as f: f.write(diff_text) print(f"Patch saved to {patch_file}") # Always run git from repo root (excalidraw/) repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) # --recount: fix mismatched hunk line counts (common in LLM-generated diffs) # --whitespace=fix: auto-correct whitespace errors apply_flags = ["--recount", "--whitespace=fix"] check_result = subprocess.run( ["git", "apply", "--check", *apply_flags, patch_file], capture_output=True, cwd=repo_root, ) if check_result.returncode == 0: apply_result = subprocess.run( ["git", "apply", *apply_flags, patch_file], capture_output=True, cwd=repo_root, ) if apply_result.returncode == 0: print("Patch applied successfully") return True else: print( f"Failed to apply patch: {apply_result.stderr.decode(errors='replace')}" ) return False else: print( f"Patch validation failed: {check_result.stderr.decode(errors='replace')}" ) return False
Python
def verify_fix(): """ Run the test suite again to verify the patch fixed the issues. Returns: Tuple of (success: bool, output: str) """ print("Running tests to verify fix...") result = subprocess.run( ["yarn", "test:app", "--run", "packages/math/tests/point.test.ts"], capture_output=True, text=True, cwd="." ) output = result.stdout + "\n" + result.stderr success = result.returncode == 0 if success: print("✓ Tests passed! Fix verified.") else: print("✗ Tests still failing after patch.") return success, output
Python
def generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir): """ Create a comprehensive Markdown report of the engineer loop run. Args: context: Original context from collect_context() plan: The generated patch plan dictionary patch_applied: Whether the patch was applied successfully tests_passed: Whether verification tests passed test_output: Output from verification test run run_dir: Path to artifact directory Returns: Path to the generated report """ timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Capture final repository state final_diff = subprocess.run( ["git", "diff", "HEAD"], capture_output=True, text=True, cwd="." ).stdout # Truncate long outputs for report readability (keep full versions in artifact files) report = f"""# Engineer Loop Report**Timestamp:** {timestamp} **Status:** {'✓ SUCCESS' if tests_passed else '✗ FAILED'}## Summary- Original test exit code: {context['test_exit_code']}- Patch applied: {patch_applied}- Tests passed after fix: {tests_passed}## Plan{plan.get('plan', 'No plan generated')}## Changes Applied{plan.get('diff', 'No diff generated')[:2000]}## Verification Test Output{test_output[:1000]}## Final Repository State{final_diff[:2000] if final_diff else 'No uncommitted changes'}## Commands Executed1. `git diff HEAD` (collect context)2. `yarn test:app --run packages/math/tests/point.test.ts` (initial test run)3. `git apply changes.diff` (apply patch)4. `yarn test:app --run packages/math/tests/point.test.ts` (verify fix)## Artifacts- Full patch: `changes.diff`- This report: `REPORT.md`""" report_path = os.path.join(run_dir, "REPORT.md") with open(report_path, "w", encoding="utf-8") as f: f.write(report) print(f"Report saved to {report_path}") return report_path
Python
def create_artifact_dir(): """Create timestamped directory for this run's artifacts.""" timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") run_dir = f"artifacts/{timestamp}" os.makedirs(run_dir, exist_ok=True) return run_dirdef engineer_loop(): """ Execute the complete engineer loop: collect → plan → patch → test → report """ print("=== Engineer Loop Starting ===\n") # Stage 1: Collect context print("Stage 1: Collecting repository context...") context = collect_context() if context["test_exit_code"] == 0: print("All tests passing - no action needed") return print(f"Tests failing (exit code {context['test_exit_code']})\n") # Stage 2: Generate structured patch plan print("Stage 2: Generating patch plan...") plan = generate_patch_plan(context) print(f"Plan: {plan.get('plan', 'Error')}\n") # Create artifact directory for this run run_dir = create_artifact_dir() print(f"Artifacts will be saved to: {run_dir}\n") # Stage 3: Apply the patch print("Stage 3: Applying patch...") patch_applied = apply_patch(plan.get("diff", ""), run_dir) print() # Stage 4: Verify the fix print("Stage 4: Verifying fix...") if patch_applied: tests_passed, test_output = verify_fix() else: tests_passed = False test_output = "Patch was not applied - verification skipped" print() # Stage 5: Generate comprehensive report print("Stage 5: Generating report...") generate_report(context, plan, patch_applied, tests_passed, test_output, run_dir) print(f"\n=== Engineer Loop Complete ===") print(f"Results saved to: {run_dir}")if __name__ == "__main__": engineer_loop()
text
=== Engineer Loop Starting ===Stage 1: Collecting repository context...Tests failing (exit code 1)Stage 2: Generating patch plan...Plan: Fix the pointRotateRads function by correcting the rotation matrix formulaArtifacts will be saved to: artifacts/20260218_154312Stage 3: Applying patch...Patch saved to artifacts/20260218_154312/changes.diffPatch applied successfullyStage 4: Verifying fix...Running tests to verify fix...✓ Tests passed! Fix verified.Stage 5: Generating report...Report saved to artifacts/20260218_154312/REPORT.md=== Engineer Loop Complete ===Results saved to: artifacts/20260218_154312