Reviewing AI Generated Code
Introduction
Welcome to Review AI-Assisted Python Changes. This is lesson 1, in which we begin learning how to review code that may arrive faster than we can fully understand it. We will examine an AI-generated ticket lookup, compare its assumptions with the ticket ID contract, and choose checks that give us useful evidence. By the end, we will have a focused verification plan and a clear way to distinguish correctness risks from security risks.
The Verification Tax: Why Plausible AI Code Needs a Different Review Budget
AI can produce code quickly, but a quick implementation still takes time to understand and verify. That gap creates a verification tax: we must spend review effort checking whether the code’s assumptions match the requirements and whether its behavior holds at important boundaries.
A polished-looking change is not proof that it is correct. We can keep the review proportional by asking what could go wrong, how much it would matter, and what small set of checks would provide evidence. Here, we will focus on one concrete risk: a lookup that might mistake a valid ID for a missing one.
The Running Example: An AI-Generated Ticket Lookup and Its ID Contract
Our example stores tickets in a dictionary using integer IDs. The contract allows ID 0; it does not say that IDs must be positive. In app/tickets.py, Ticket holds an ID and a title, while create_ticket stores and returns the new ticket:
The dictionary key is the ticket ID, so creating a ticket with ID 0 stores it under key 0. The key question is whether the lookup respects that same contract, rather than adding an unstated rule about which IDs are valid.
Spotting the Unsupported Assumption: When Truthiness Hides Ticket Zero
A generated implementation might use truthiness to decide whether an ID was provided:
In Python, 0 is falsy, so this code returns None before checking whether ticket 0 is stored. That makes a valid ticket look absent. An unknown ID should return None, but a stored zero ID should return its ticket. Those are different outcomes, and our checks need to distinguish them.
Evidence Over Plausibility: Building a Focused Verification Plan
Instead of accepting a claim such as “IDs are positive” because it sounds reasonable, compare it with the actual contract. The project’s REVIEW.md records the claim, its impact, and the evidence needed:
This review note makes the unsupported assumption visible and ties it to a specific failure. It also keeps the review focused: we need evidence about zero and unknown IDs, not a rewrite of unrelated ticket behavior.
Choosing the Right Checks: Boundary Tests, the Entry Point, Flake8, and pytest
Focused tests check both sides of the boundary in tests/test_tickets.py:
The first assertion confirms that the lookup returns the same stored object; the second confirms that an absent ID returns None. The main.py entry point exercises the public helper, while solution.py calls main(). When the lookup is correct, its output is:
The review plan also names Flake8 and pytest. requirements.txt pins both tools, and .flake8 sets a 100-character line limit. The Run control executes solution.py, while the lint and test checks run from the terminal:
A plan lists the checks we intend to run. The review record should then report only results we actually observed, so a reader can tell planned evidence from executed evidence.
Classifying Risk: Correctness Boundary vs. Security Boundary
This lookup is correctness-sensitive: a truthiness check could hide a valid ticket. It is not, based on the described change, a security boundary. It does not handle authorization, executable input, credentials, secrets, sensitive data, or an external trust boundary.
That distinction helps us choose the review depth. We still verify the behavior carefully, but we do not label every bug a security issue. If a change does involve those sensitive areas, it needs specialized security review whether a person or an AI wrote it.
Reviewing Without Rewriting: Requesting Direct Lookup and Focused Tests
A reviewer’s job is to identify the evidence needed and explain unresolved concerns, not to silently take over the author’s work. The direct dictionary lookup matches this API: it returns the stored value for an ID or None when the key is absent.
If the proposed change used truthiness, a focused review comment could ask the author to preserve the integer-ID contract and add tests for both zero and an unknown ID. That request names the risk and the evidence needed without prescribing unrelated changes.
The same boundary applies to every revision that comes back. When the author fixes one case, we check the whole contract again rather than trusting that a passing test covers everything. Throughout the practices, we stay in the reviewer’s seat: the author supplies the code and its revisions, and we run the checks and record findings and requests in REVIEW.md.
Conclusion and Next Steps
The verification tax is the work of checking code that may be easier to generate than to understand. We compared the generated positive-ID assumption with the actual contract, identified why truthiness can hide ticket 0, and selected focused evidence: tests for zero and unknown IDs, the runnable entry point, and the Flake8 and pytest checks. We also classified this as a correctness risk while recognizing when specialized security review is needed. Next, we will put that approach to work in the practices, starting by running the focused tests against the generated lookup and recording what they show.
