We ran AI code review across our engineering team for six months. Some of it became indispensable. Some of it made code review actively worse. Here's the honest breakdown.
By late 2025, every code review tool we used had added AI-powered review. PR-summarisation bots. Inline-comment bots. "Suggested change" bots. We turned them all on, because why not and within a fortnight our PR threads had become unreadable - hundreds of low-value comments per review, drowning the human discussion that actually mattered.
We rolled it back, picked one tool at a time and ran each through a six-month evaluation. The result is a much shorter list of things that actually earned their place.
We wanted AI review to do things humans were bad at or slow at - not to replicate things humans already did well. Specifically:
We didn't want AI to do the things humans should be doing - judging trade-offs, weighing design choices, deciding what's worth fixing now versus later.
The default behaviour of most AI review tools is to comment on everything they notice. That's the opposite of what makes code review valuable. A good human reviewer leaves three to five comments. A default-configured AI reviewer leaves thirty.
Comment volume isn't a neutral cost. It changes how the author engages with the review. When every PR has thirty AI comments, the author starts skimming them. When the author skims them, they miss the one that actually matters. The signal drowns in the noise it generated.
Three things, each scoped narrowly.
1. PR summarisation, gated to large diffs. For PRs over 400 lines changed, we auto-generate a structured summary - what changed, which subsystems are affected and which test files were touched. The summary goes in the PR description, not the comments. Reviewers told us this was the single most useful AI feature, because it lets them decide where to spend attention.
2. Security-focused review, run as a single bot with a single comment. One tool, configured to look for a specific list of issue classes (secrets in code, SQL injection patterns, unvalidated user input flowing to system calls). It posts one consolidated comment per PR, listing whatever it found. Zero findings means zero comments. This catches roughly two real issues per month across our team. It also catches false positives, but at one consolidated comment, the noise is bounded.
3. Convention enforcement as a pre-commit hook, not a reviewer. Things like naming, import order, error-handling style - these belong in an automated check that runs before the PR is even opened. Putting them in code review wastes everyone's time. We moved every "convention" rule out of the review tools and into pre-commit.
# pre-commit hook config excerpt
- repo: local
hooks:
- id: convention-check
name: Convention enforcement
entry: tools/convention-check
language: system
pass_filenames: true
Inline "suggested change" comments at scale. A bot suggesting a one-line rewrite to every function it thinks could be cleaner is exactly the noise we wanted out of the review. We disabled this across the board.
AI as an approving reviewer. Some tools offer to auto-approve PRs that the AI considers low-risk. We tried it on a sample of internal repos and reverted within a week. The AI's risk assessment is fine for what it is, but human approval is also a signal that someone took responsibility for the change. Removing that signal made ownership murky.
Architecture-level review. AI reviewers comment on local code patterns. They don't see the system the way a tech lead does - the historical context, the open RFCs, the things-we-tried-and-rejected. Asking AI to review architecture decisions in a PR generated either generic platitudes or confidently wrong objections. We left this to humans.
We would set the comment-volume budget explicitly from day one. Configure each tool to produce at most one comment per PR. Anything more, escalate inside the tool, don't post additional comments. This single constraint forced the tooling decisions in the right direction.
AI code review earns its place when it's narrow, consolidated and aimed at things humans are demonstrably bad at - security scanning, convention enforcement, summarisation of large changes. Anywhere it overlaps with judgement, taste, or system context, it makes review worse, not better. Configure for signal, not volume.