How to review an AI-generated pull request

What’s different about reviewing AI-written code

Reviewing a colleague’s pull request relies on things that do not hold for a coding agent. A colleague usually knows the product rules, writes tests separately from the fix, and tells you when they are unsure. An agent produces more code, faster, in a uniformly confident style, and its pull request description is a claim about what it did, not evidence that it did it.

Three habits change as a result:

• Read what was removed as carefully as what was added. Agents simplify aggressively, and a deleted line is often a deleted safety check.
• Treat test changes as behavior changes. If the tests moved, the definition of correct moved with them.
• Verify, don’t trust, the summary. “All tests pass” and “no breaking changes” are statements to check, not conclusions to accept.

The review order that catches the most

1. Restate the intent. In one sentence, write what behavior this pull request should change and what it must not change. If you cannot, the agent could not either.
2. Review the tests first. Look for changed expected values, weakened assertions, new skips, deleted cases, and mocks of the code under test. See how agents change tests to make them pass.
3. Read every deletion. Removed if statements, early returns, throws, and where clauses deserve a reason.
4. Check trust boundaries by hand. Authentication, ownership and tenant checks, permission logic, input validation, and anything that touches secrets.
5. Confirm new dependencies are real. A study of 16 code-generating models found average hallucinated-package rates of at least 5.2% for commercial models and 21.7% for open-source models. Attackers register those invented names, a technique called slopsquatting.
6. Require evidence. A link to a CI run of the relevant tests, and for critical paths, a mutation report showing those tests catch real breakage.

Failure patterns that look correct

These pass a quick read because the code is clean and idiomatic. They are wrong because of what the code no longer does.

A reviewer checklist

[ ] I can state in one sentence what behavior this PR changes
[ ] Test diffs reviewed first; no unexplained expected-value changes
[ ] No new .skip / .only / deleted test cases
[ ] Every removed guard, condition, or throw has a reason
[ ] Auth, ownership, and permission checks still apply on every path
[ ] New dependencies exist, are maintained, and are the intended package
[ ] Unfamiliar APIs and options checked against real documentation
[ ] Relevant tests ran in CI (not just "the agent said so")
[ ] Critical paths have a mutation check or equivalent evidence
[ ] Any intended behavior change is approved by its owner

Where AI review bots and verification fit

AI code review bots are a useful first pass. They catch typos, obvious bugs, and missing error handling quickly. But they read the same diff the coding agent produced and have no record of what the product promised, so they cannot tell an intended rule change from an accidental one. The comparison of review tools and verification covers which layer catches what.

For the parts of this checklist that can be automated, MaruCheck checks the diff against reviewed requirements, runs only the linked tests, detects behavior drift, runs mutation checks on changed code, and posts a ship-or-block decision from a least-privilege GitHub Actions workflow. The reviewer still owns the decision; the evidence just arrives with the pull request.

Related guides