Automated Pull Request Review Agent Setup

Setting up an automated pull request review agent is an infrastructure problem. A useful reviewer needs persistent context, webhook-triggered scheduling, and a runtime that holds repository knowledge across reviews. Those foundations let it follow a change through revisions instead of leaving a single comment and disappearing.

Why a one-shot LLM call is the wrong foundation for PR review

The simplest setup takes a diff, sends it to an LLM, posts a comment, and exits. Its context window acts as per-conversation RAM, so whatever the model considered during that run disappears when the process ends. The next run starts without a record of the previous review. That makes it hard to tell whether a finding from last week was fixed or whether the current change repeats a known bad pattern. A comment may be useful on its own, but it does not establish continuity.

That gap matters because review is not limited to reading the change in front of the model. Teams accumulate conventions through earlier decisions, and later changes need to be judged against those decisions. A stateless reviewer cannot apply that history unless something supplies it again. Careful reasoning therefore requires a way to preserve and retrieve context.

The strongest objection is that LLM calls are fast and cheap, while infrastructure adds work. Yet putting the full repository history into every call is not a sound substitute for memory. Output quality degrades at very large context sizes, and the cost of repeatedly supplying that history grows with it. Using a dedicated memory layer to retrieve just what each call needs costs far less per call and scales as the repository grows.

The three operational requirements a PR review agent actually needs

The first requirement is persistent cross-session memory. Working context serves the current review, but durable memory carries findings and conventions into the next one. Without that distinction, the agent cannot tell that an author has already dealt with an earlier finding. Cross-PR learning requires a record that survives the session.

The second requirement is webhook-triggered scheduling. The agent must react to changes in the pull request lifecycle rather than poll for work or wait for someone to invoke it. Without a trigger for an updated PR, a review can remain attached to an earlier version of the change while new commits go unexamined. Event handling connects the reviewer to the moment its judgment is needed.

The third requirement is a runtime that holds and updates repository knowledge. It must scope tools to the right repository root, track file changes, and resume from its last known state after a restart. Otherwise, a failure halfway through a review can sever the connection between completed work and what happens next. These requirements solve different problems, so none substitutes for another. Managed tools make architectural choices around them because a reviewer needs continuity in its knowledge, its timing, and its execution.

How production PR review agents handle persistent memory

Memory design begins by separating three kinds of context that serve different purposes. In-context working memory contains what the agent reasons over during one review session. It is bounded by the context window and is temporary by design. Recall memory, by contrast, stores searchable review history across sessions. It lets the agent find earlier conversations and findings without loading the whole archive into every request.

That separation makes a concrete difference to follow-up work. With recall memory, an agent can connect a pattern flagged in PR #214 to a fix in PR #217. The important capability is retrieving the relevant relationship when a later change calls for it. A developer choosing a memory design therefore needs to distinguish a store of past material from a way to find the useful part of that material.

The third layer is archival or procedural memory, which holds the team's review conventions. Reusable prompt templates and instruction files turn those conventions into criteria that can be applied across agent instances. Distributing them uniformly gives a new repository access to the same encoded standards as a mature one. Claude Code Review illustrates this approach through project instruction files that customize domain-specific review criteria.

Durable memory also needs a clear policy for what to forget. Without a way to retire stale material, months of accumulated PR context can degrade retrieval quality. Scope is a separate concern again. MCP Roots can define a project root so tool calls stay within that directory, but MCP Roots do not provide file-change tracking or cross-restart state resumption. An infrastructure layer still needs to handle those tasks, even with the memory store and repository boundary in place.

Wiring webhook triggers and CI so the agent fires on the right events

Once relevant context can survive a run, the next task is to start reviews at the right time. The trigger layer should respond when a pull request is opened and when it is synchronized after new commits. This gives the agent a chance to act before a human reviewer reaches the PR. It also makes the author's next push part of the same review process.

PR-Agent's GitHub Action provides a minimal example of the connection. Its workflow uses pull_request as the event, with opened and synchronize as the selected event types. The action step references the-pr-agent/pr-agent@main and receives the environment variables OPENAI_KEY and GITHUB_TOKEN. These pieces connect a pull request event to an automated review invocation, establishing when the review runs.

The same event-driven approach works beyond GitHub, though how it connects differs by platform. PR-Agent documents webhook setup for GitLab and Azure DevOps, and pipeline or server-based integration for Bitbucket. GitHub Actions can be enough to provide the trigger, but each job starts fresh. Without external memory, even a job that runs at the right time has the same one-shot problem.

What a runtime substrate adds that a webhook plus an LLM call cannot

A trigger explains why an agent starts, and a memory store explains where its findings can persist. Neither, on its own, governs a sequence of tool calls or manages the working directory in which those tools operate. The runtime supplies that execution context. It keeps the review at the right repository root as the agent works through each step. It also provides the basis for recovering when execution stops before the review is complete.

Recovery is important because storing findings is not the same as recording execution state. A review may involve several tool calls before it produces comments, and a failure can interrupt that sequence. A recoverable runtime is meant to keep that interruption from leaving a half-written comment thread as the final result. Its checkpoint records where the work stood so a restart need not discard progress. Memory preserves what the agent learned, while execution state preserves where it stopped.

The useful analogy is an operating system for applications. A runtime abstracts the underlying execution machinery so review logic does not have to manage every primitive directly. A purpose-built agent runtime is built for work that runs for a long time, takes many steps, and uses many tools, rather than for the request-and-response patterns of a web service. Assembling those responsibilities from general-purpose cloud primitives leaves the developer to make the pieces behave as one recoverable system. A developer-first runtime should also be curl-installable, because the infrastructure should not demand a longer onboarding path than the agent it runs.

Architecting the review loop: from webhook event to inline comment to follow-up

A GitHub pull_request event reaches the runtime, which creates an agent session scoped to that PR's repository root. The session queries recall memory for prior findings on the PR and archival memory for the team's review conventions. Only what the current diff needs goes into working context. This handoff joins the event to both the right codebase and the right history.

With that context loaded, the agent runs its review tools against the diff and the surrounding code. PR-Agent has three automatic tools that run by default, named describe, review, and improve. Equivalent tooling can occupy the same place in the architecture without changing the loop.

Findings then return to the pull request as line-anchored comments. A summary comment can include a hidden marker that allows the interaction to be resumed. The Gemini pattern uses an interaction ID and an environment ID for that marker. This connection matters because the next event needs to lead back to the existing review. The comment is an output for the author and a point of continuity for later work.

After posting, the session writes its findings to recall memory, and the runtime checkpoints its execution state. When the author pushes a new commit, the next synchronize event brings the agent back to that stored work. It compares the new diff with earlier findings, reports what is fixed and what remains open, and raises new findings only for the delta. That differential follow-up is the payoff of the architecture.

The entry point can also come from an existing team discussion. CodeRabbit Agent for Slack demonstrates a pattern in which someone mentions the agent in a thread and it investigates from that conversation. The same webhook infrastructure that listens for PR events can listen for Slack events through the runtime. This changes where the loop begins while preserving the need for scoped execution and persistent context. The architecture remains centered on carrying relevant work forward from one interaction to the next.

Human oversight boundaries the architecture must enforce

Continuity makes an agent more capable, but capability does not settle who should authorize a production change. The architecture must distinguish reviewing a pull request from approving and merging it. An agent can review, comment, suggest changes, and commit fixes to a branch while the merge decision remains human-gated for changes with material production impact. That boundary belongs in the design from the start to make the system fit for production.

METR's February–March 2026 pilot with Anthropic, Google, Meta, and OpenAI gives this distinction a concrete basis. METR reported that agents often autonomously wrote complex PRs to fix bugs or add features at some participating companies. It placed approving and merging PRs for critical shared codebases without human review among the things it believed agents did not do autonomously at any participating company.

The boundary also has to hold through permissions. An agent with both repository write access and merge authority is a high-value target. Scoping access to comments and branch commits rather than merging is therefore a security posture as well as a review policy. The runtime's permission scope, GitHub branch protection rules requiring human approval, and audit trails recording what the agent did and when make the boundary concrete.

Measurement needs the same distinction. When agents review AI-generated PRs, recorded review activity can reflect automation-mediated interaction. Teams need to instrument review metrics so agent activity remains distinguishable from human oversight. Otherwise, more reviews in the record can make it harder to tell who actually checked the change.