We’ve helped dozens of companies rebuild their review workflow after it collapsed under the volume of code their teams are producing. Trying to keep running things like it’s 2020 is DDoSing your team with code review. Installing an AI reviewer leaves several decisions unresolved: which findings must be fixed, who can make exceptions, and what happens when the reviewer cannot run.
Here’s the approach we recommend to our customers:
1. Give every check an explicit role#
Organize review into four categories:
- Automated checks that block: tests, build checks, etc., with defined pass/fail results.
- AI checks that block: review against written criteria, with thresholds for which findings stop a change.
- AI review that advises: findings for humans to consider, without giving the model veto power.
- Human judgment: final approval, disputed findings, and decisions to accept an exception.
AI review should cover at least six areas: security, performance, test coverage, backend architecture, frontend architecture, and documentation. This policy should be in your repo. The scope can also vary depending on your product’s needs at your current stage.
2. Give every finding an outcome#
Every review comment needs one of three outcomes:
- Fixed in this change.
- Declined, with a reason and supporting evidence.
- Deferred, with a linked task.
Keep a record of those decisions in the PR description for accountability. This record also helps improve the reviewer. With it you can tell whether a recurring comment is flagging a recurring problem, or if the reviewer is repeatedly getting something wrong.
3. Check what “required” actually enforces#
“Required” checks might report as passed if there is an outage in the model they use. GitHub accepts neutral as a successful outcome for required status checks. GitHub documents this behavior.
For a required review, an unavailable reviewer should leave the change blocked until the review completes or an authorized person records an explicit exception.
4. Make the checks available before submission#
These tests should be available for engineers and agents before a PR is ready for review. Otherwise, each fix requires submitting the change, waiting for the shared checks, discovering problems, and trying again.
Our local AI review uses the same written criteria and model tiers as the review on submitted changes.
Saved review results also need to be kept up to date. If the review criteria change, all earlier passes should expire. Including the exact criteria in the saved result’s identifier makes that automatic.