How the Claude Code Team Does Code Review When Agents Write the Code

Leafw included in AI

2026-10-18 3151 words 15 minutes

Contents

With coding agents like Claude Code and Codex, PRs get opened far faster than before. Agents keep working overnight, and you may wake up to several PRs waiting to be merged. The code gets written faster, but the people reviewing it are still the same few.

This week I read two pieces that are worth looking at together. One is How the Claude Code team does code review, published by the Claude Code team on the claude.dev blog and signed by Addy Osmani, Morgan Westlee Lunt, and Qing Wang. It describes how they do code review now that agents write most of the code. The other is an October 16 report from The Register: the release of PostgreSQL 19 has been delayed, and one of the reasons has to do with AI-assisted code review.

Both stories circle the same question: once AI makes both writing and checking code faster, where should human time go? Below I’ll first walk through Anthropic’s process in the order of the original post, then look at Postgres.

The Review Bottleneck

The post opens by repeating something they’ve said before: about 90% of Claude Code’s own code is written by Claude.

The volume of code has grown along with it. In Anthropic’s monorepo, the number of changes landed per day went from about 2,500 in May this year to more than 30,000 in September. Claude Code has its own separate repository outside that monorepo, but the post says it’s under plenty of pressure too. In their words, at this pace, line-by-line human review is “extremely difficult, if not impossible,” and review has become the bottleneck.

So they changed the question. “Has someone read this code?” is only a rough proxy for what they actually care about. What they really want to know is: what could go wrong with this change, and which check (automated or human) is best placed to catch it?

The timing of review changed too. Review used to be a gate right before the finish line: write the code, open a PR, wait for a colleague to look. Now review runs continuously, like a spell checker that checks as you write. Most changes have already been reviewed and fixed by Claude at least once before a human is asked to look.

The post says what’s scarcest now is human time and attention, and that it has to be counted on both ends: the time people spend reviewing, and the time people spend fixing broken code. If review time saved turns into time spent fixing bugs, nothing was saved. So the goal they set for Claude Code’s repositories (CLI, desktop and web, mobile) is that most PRs merge with at most one human involved, and that human can be the author, without rollbacks or incidents going up. They track human approvals per PR (median and 95th percentile), human comments, and how long a PR waits on a person. Rollback rate and incident rate are the guardrails: if they rise, human time has merely moved somewhere else.

How Many Layers a Change Passes Through

The post breaks the path of a change, from the author’s machine to users’ hands, into several layers: self-verification before opening a PR; CI, Claude review, and security review on every PR, all fully automated; then human approval based on risk; then the merge queue; after merging, the change sits behind a flag for internal use, then rolls out gradually, and finally someone keeps watch, ready to fix forward or roll back. Each layer has holes in different places, so a problem that slips through one is usually caught by a later one. The rare ones that get through every layer are backstopped by fast rollback at the end.

Prove It Before Opening a PR

Most review now happens before a PR even exists. They give Claude something it can actually run, such as tests, a build, or a screenshot to compare against the design, and require it to show the results instead of just saying “done.” Then they dispatch a subagent to review the diff in a fresh context. Many people also run a few internal skills before opening a PR, one each for finding bugs, simplifying the diff, and checking behavior end to end.

Boris Cherny, the creator of Claude Code, has said that giving Claude a way to verify its own work is probably the single most important thing for getting good results, and he estimates it makes the final result two to three times better.

If automated review keeps flagging the same kind of issue, they fix it at the source: change a skill, add a lint rule, or update the instruction files. CLAUDE.md describes the project’s stack, conventions, goals, and what not to do. It’s loaded on every review, so it has to be short. REVIEW.md is written by code owners and tells the reviewer what matters most in that part of the code. The post says good REVIEW.md files are all short: they spell out what “wrong” looks like in this repo (a doc is wrong when “it lies,” a config is wrong when “it routes requests incorrectly”), cap the number of nits, list paths that can be skipped, and require file and line numbers as evidence.

AI Review on Every PR

Every PR goes through a Claude review before a human looks at it. It reads the diff in the context of the whole repository and the intent stated in the PR, checking whether the change solves the problem, fits the system, and breaks anything else. The external version is called Code Review for Claude Code and is generally available on Team and Enterprise plans; you can run a lightweight version locally with /code-review. When the research preview launched, the share of internal PRs receiving substantive review comments rose from 16% to 54%.

Before review comments reach a human, they go through verification: several agents hunt for bugs in parallel, then another group of agents votes on each finding, and only findings that enough agents consider real get posted. Comments only about style, diagnostics, or code tidying are filtered out. The rest are sorted by severity and tagged with one of three labels: Important for significant issues, Nit for minor ones, and Pre-existing for bugs that were there before this PR, so they aren’t pinned on the author. Every comment must be addressed, either fixed or closed with a stated reason.

Security Review and CI Can Both Block

Security issues get their own automated review. As long as any comment from the bug review or the security review is still open, the PR can’t merge; false positives can be closed with a one-line reason. After that come lint, type checks, per-platform tests, and builds. A typical CLI PR may have run dozens of checks before anyone looks at it. The PR template asks how the change was tested and how to back it out if something goes wrong. Claude can fill it in, but the person proposing the change has to sign off on it.

Risk Decides How Many People Approve

Editing a doc and editing the auto-updater carry very different risks. The post gives three tiers:

Risk	Examples	Approvals required
Low	Docs, copy, internal-only features, incremental changes behind a flag	None once automated review and security checks are all green
Medium	Ordinary product changes, plus sensitive areas like caching, the auto-updater, and system prompts	One person
High	The riskiest code, plus a small set of paths that always need two people	Two people

The risk tier is determined by the content of the change, not by who wrote it: one classifier reads the change and assigns a tier, and a second classifier decides whether the change still needs an extra person even if its tier doesn’t require one.

A few more rules:

Only authors on a repo’s trust list can take the lightweight path; most of them are on the team that owns the repo. Everyone else’s changes need a second person’s approval;
Paths that code owners mark as sensitive, and all CI workflow files, always need a human to look;
An agent’s approval never counts toward a human slot. Automated review can let low-risk changes through, but it can’t fill a seat that requires a person;
For a PR written by an agent, the person directing that agent counts as one approver, the same as if they had typed the code themselves;
An approval applies to one specific commit. Any later commit the system can’t vouch for (such as code pushed by an agent) requires re-approval.

Merge Queue, Flags, and Rollback

The merge queue reruns every check with the change applied on top of the latest main, so that two PRs that each passed on their own don’t break main once combined. New features usually ship behind a flag to Anthropic employees first; public releases go to the latest channel first, with the stable channel following. Flags can be turned off without a redeploy. Claude Code has a weekly rotating on-call, and incident reviews ask which layer should have caught the problem.

The post is also upfront about what isn’t done yet: spotting a bad merge still depends on the author, the on-call engineer, and the rollback and incident dashboards. They don’t yet have a circuit breaker that automatically tightens approval requirements when metrics go bad, and the engineers want to build one before more code goes through zero-human merges.

The Reviewer Gets Tested Too

The team collected PRs that caused past incidents, along with their later fixes and rollbacks, and asked engineers familiar with that code to confirm each bug one by one. That became an eval set, which the reviewer runs against daily. Every time the reviewer’s prompt, tools, or model changes, it’s scored against the whole set, and the change is kept only if the score goes up.

Noise counts too. They look at how people respond to each comment: fixed it, said it was wrong, said the behavior was intentional, or didn’t care. They also run the reviewer on PRs known to be fine and count how often it nudges people toward a harmful change. Noise costs more when the author is an agent: a human can skim past a nit in seconds, but an agent will go fix it, which means another commit and another round of review.

Review also has to converge. One CLI PR went back and forth between review and fixes more than fifty times overnight until the author stopped it manually. They ran git blame on the code the reviewer kept flagging, and about 59% of it already existed at the reviewer’s first pass. The reviewer was rarely finding new bugs; each round just dug up a few more old ones. The current rules: at most three rounds, after which the author is asked to rethink the approach; after the first few rounds, review only the new diff; flag bugs by category so the author can fix similar issues together; never repeat a comment that’s already been made. In addition, a review agent sweeps the existing code every night looking for old bugs.

After these changes, the share of internal PRs that pass after one round of review plus fixes rose from 65% to 78%, the share needing five or more rounds dropped from 8.9% to 1.5%, and the average number of reviews fell from 2.08 to 1.32.

Where Humans Stay

The post says their work has shifted from reading diffs one by one to five kinds of decisions machines aren’t good at: whether something should be built at all; high-risk code like billing, the auto-updater, and CI workflows; architecture that others will copy; reading evidence, including test output, commands run and their results, screenshots, and review comments; and owning the outcome. One team declared its review bottleneck solved but still kept human review for changes submitted from outside the team, so that someone can ask “should we be doing this?”

Another job of review is helping the team understand the codebase, and they have no formal process for that. If you want to understand a part of it, you ask Claude to write an explainer on that part; it takes a few minutes and matches the current code. The team also has a convention: if a small or medium change breaks something after merging, the author usually fixes it. What holds this convention up: changes are small; everything can be backed out, and most mistakes can be undone within minutes; the bigger the blast radius, the stricter the requirements; and telemetry and internal use surface regressions quickly.

Two Sets of Numbers

The post contains two sets of numbers measured differently, and they should be read separately.

The earlier figures, from 2,500 to more than 30,000, refer to the whole Anthropic monorepo.

The second set covers only Claude Code’s own repository, from June to September: the number of merged PRs more than doubled, and the share of PRs requiring a rollback fell by more than 40%. The post presents these numbers alongside the review process described above.

The two sets describe two different repositories; the change volume from the first and the rollback share from the second can’t be combined.

PostgreSQL 19 Delayed

According to The Register, the PostgreSQL Global Development Group has delayed the release of PostgreSQL 19. The release was originally scheduled for late September or early October and is now expected around November 19. At the same time, 8 to 10 major features were pulled and pushed to next year’s PostgreSQL 20.

The person explaining the withdrawals is Peter Eisentraut, a vice president at EDB, a PostgreSQL support and consulting company, and one of PostgreSQL’s major committers. A committer is a core developer with the right to merge code into the main line. As relayed by The Register, he believes the withdrawals can be attributed at least in part to the LLM-based code review developers are using.

AI review is only one of the reasons. He wrote on his blog that some of the withdrawn features had serious architectural flaws that would be hard to fix “quickly and during beta.”

On the AI review side, what it turned up was a long list of issues that were “relatively harmless and involve various edge cases.” In the past, issues like these would mostly have been discovered and patched one by one over the five years after release. He gave an example: “With LLM-assisted code review, these kinds of issues can be found much faster, and then you’re looking at a list of forty defects. Even if each one takes on average only a three-line fix, there’s a fixed overhead to dealing with each one, and then you run out of time.” The forty is his illustrative example; the report doesn’t give an actual number of issues.

There was also a timing mismatch. When several big features were written, LLM-assisted review wasn’t as useful as it is now; in the months leading up to the release, people figured out how to use AI-assisted review better and it got more effective. Eisentraut said: “The actual quality bar was raised after the features were written and committed. Unless the capabilities of LLM-assisted review keep improving at the current pace, this should sort itself out in future development cycles.” He also made a point of saying the withdrawals have nothing to do with vibe coding, since most of the new features were written before that practice existed.

Among the pulled features is SQL/PGQ, the property graph query support that entered the SQL standard in 2023, used to explore relationships between nodes connected by edges. PostgreSQL contributor Tom Lane warned at the time: “I’d bet a dinner that if we ship this in v19, we’ll find bugs after release that can’t be fixed until v20.”

Eisentraut said 19 will still be “a great release,” including REPACK CONCURRENTLY, which reclaims disk space without the table-locking VACUUM FULL; “with all this extra scrutiny, it will be very solid.” A few months ago, developers thought PostgreSQL 20 would only fix bugs; now it will most likely take in the features pulled from 19.

My Takeaways

Count finding and fixing together. When Anthropic counts human time, it includes the time spent fixing broken code. By Eisentraut’s account, Postgres’s problem was on the fixing end: issues were found faster, but each one carried a fixed handling cost. My read is that before bringing in AI review, it’s worth estimating how much your team can actually fix.
When review happens matters a lot. Anthropic’s review runs while the code is being written. Postgres’s big features were written and committed before AI review became useful, and the bar was raised after the fact. The same batch of issues leaves very different amounts of time for people to deal with depending on whether it’s found during coding or right before release.
Rank findings by severity. Anthropic sorts comments into Important, Nit, and Pre-existing, filters out pure style, and leaves old bugs to the nightly sweep. What Postgres dug up was mostly relatively harmless edge cases. A long, unranked list eats up the most human time.
Review needs to be able to stop. A three-round cap, reviewing only the new diff, reporting bugs by category. After adding these, Anthropic’s average number of review rounds fell from 2.08 to 1.32.

What the Sources Don’t Say

Both outcome numbers for the Claude Code repo are relative: there’s no absolute count of merged PRs and no starting point for the rollback share.
The incident rate is listed as a guardrail, but its change isn’t published.
The post doesn’t break down how much of the June-to-September results came from this process.
Neither the accuracy of the risk classifier nor the share of PRs in each tier is given.
On the PostgreSQL side, the report doesn’t give the total number of issues AI review found, nor does it explain the reason for each withdrawn feature individually.

Final Thoughts

The post ends by saying that when review misses something, they ask the model what information it was missing at the time, then write the fix into CLAUDE.md, a skill, or a new check. The goal is less human time per change, with no more rollbacks than before.

The post also notes that the external Code Review product only gives comments, not approvals; approving a PR is still done by a person. In their internal repos, owners decide which low-risk changes can merge on automated checks alone. What external teams get isn’t quite the same as what they do internally.

If your team already has agents opening PRs, a few steps from the post’s list have a low barrier: cap the number of review rounds, take agent approvals out of the human slots, add “Testing,” “Risk,” and “How to back out” sections to the PR template, and write down your risk tiers, starting strict and loosening once you have evidence.