PagerDuty Alert Triage Agent End-to-End Recipe

Building a PagerDuty alert triage agent end to end means connecting three jobs, namely pulling an alert, correlating telemetry through MCP, and producing a root cause hypothesis with next steps. The recipe closes the gap between detection and resolution by automating the first investigation.

Why alert triage is still the slowest phase of incident response

When an incident fires, the responder’s first job is often finding enough context to start thinking. That means locating a runbook, pulling logs from several observability tools, and checking whether the same issue has happened before. Modern alerting platforms handle detection and routing, but those capabilities do not assemble the evidence needed for a diagnosis.

The relevant data lives across PagerDuty, an observability platform, and a knowledge base, with no automatic connection between them. An on-call engineer who does not own the affected service also faces a cold start. Before searching, that responder must learn which signals matter.

What an alert triage agent actually does, versus a pager or a chatbot

A triage agent takes responsibility for that first pass through the evidence. Instead of stopping at routing, it pulls the alert, searches recent telemetry for correlated errors, and produces a timeline, a root cause hypothesis, and next steps. This work can happen before a human acknowledges the incident. PagerDuty’s named SRE Agent, Paige, belongs to this investigative category, rather than serving only as another way to deliver a notification.

A useful triage agent differs architecturally from a chatbot: it needs live data from across systems and must be able to link the steps from retrieval through correlation to synthesis. It also needs a stopping rule, because an investigation that keeps searching without reporting is not useful during an incident. Mezmo’s AURA recipe makes that boundary explicit through a default five-minute investigation window and a cap of 10 reasoning turns. The goal is a bounded first hypothesis.

The integration problem that MCP solves before you write a single line of agent logic

Before the agent can reason, it needs access to the systems that hold the evidence. Without a shared interface, connecting runtimes to PagerDuty, observability tools, and knowledge bases creates an N×M integration problem. Each additional pairing can require bespoke connector work. Model Context Protocol, or MCP, standardizes the interface between AI models and external tools and data sources. In the AURA recipe, PagerDuty and Mezmo connect through their official MCP servers without custom integration code.

PagerDuty also supports connectors to third-party sources including Grafana, New Relic, and Honeycomb through MCP or API. That shifts setup toward entering credentials and authorizing access. PagerDuty’s Spring 2026 release positions MCP as the connection to observability tools, internal developer platforms, and developer environments. MCP does not define a memory primitive or cover memory governance, so other parts of the architecture must handle both.

The agent stack for this recipe: runtime, protocol, and telemetry substrate

The stack is easiest to understand as three rows, each with a separate job. The first row is AURA, the agentic runtime that supplies API servers, state management, authentication, streaming, error handling, and tool integrations. It began as Mezmo’s internal harness for operating its SaaS and powers thousands of agent sessions per month in Mezmo’s hosted platform. AURA is open source under Apache 2.0 with no software license fee, although running it still involves model, infrastructure, integration, and engineering costs.

The second row is MCP, the interface that carries tool calls between the runtime and the systems it connects to. The third row is Mezmo, the telemetry analysis and pipeline monitoring layer reached through its official MCP server. PagerDuty supplies the incident context that starts the workflow.

Step 1 — Pulling the PagerDuty alert into the agent

The first step turns the alert into structured input rather than a free-form request to investigate a problem. At a minimum, the alert input has the service name, severity, timestamp, and any attached metadata. The service identifies the investigation’s subject, while the timestamp anchors the search for related signals. Severity and metadata carry the context already attached to the alert. No invented example values are needed to see the contract between this step and the next.

In the open AURA recipe, a request passed to the agent starts the workflow, and the alert data becomes its input context. PagerDuty’s own architecture offers an escalation-policy path in which Paige is added to an escalation level to investigate in parallel with on-call responders. That path is currently in Early Access. Both lead toward investigation, but the request is the starting mechanism described for AURA.

Step 2 — Searching telemetry for correlated errors across connected sources

With the alert in context, the agent searches Mezmo telemetry for correlated errors and anomalies. Unless the operator specifies another window, the recipe examines the last five minutes of telemetry and alerts. It also has a 10-turn reasoning cap. These defaults limit how many tokens the agent uses and make it report what it found.

When an investigation spans independent sources, the next concern is waiting time. The orchestrator/worker pattern described for this recipe delegates metrics work to a Metrics Worker through an MCP server connected to Datadog and log work to a Logs Worker through an MCP-connected Elastic instance. The queries are meant to work like parallel API calls, so each can start before the other finishes. Mezmo describes AURA’s planned multi-agent orchestration module as a Directed Acyclic Graph executor supporting dependency-aware parallel wave execution. That planned capability is separate from the basic PagerDuty-and-Mezmo connection and would let independent searches for evidence run together.

Step 3 — Generating the root cause hypothesis and next steps

After retrieval, the agent must turn correlated signals into something the responder can use. The AURA recipe defines the deliverable as “a timeline, a root cause hypothesis, and next steps, with timestamps and links back to the source data”. The timeline establishes the sequence, the hypothesis explains the suspected cause, and prioritized next steps guide further investigation. Timestamps and source links make the reasoning auditable without requiring the responder to repeat every search. Correlation supports a hypothesis, but it is not the same as a confirmed root cause.

AURA’s reasoning architecture has steps for planning, execution, synthesis, and self-evaluation. When confidence is low, the agent can replan and make another investigative pass within its turn budget rather than present uncertainty as a confident answer. Paige adds a separate capability through memory accumulated from incidents it participates in. Operators can also enhance that memory directly through its shared memory API. These are distinct mechanisms, but both address the same practical concern of grounding the answer in more than the current prompt while preserving a consistent output format.

Where human sign-off fits in the workflow and how to configure it

The recipe described here is investigative only, reading alerts and searching telemetry. Any extension that changes system state raises a different question about authorization. Autonomous investigation and human approval over action are separate design choices, and production systems with costly false positives need that distinction made explicitly. In AURA, tools capable of changing state can be placed behind a [hitl] approval block. Only tools that match a configured require_approval pattern need sign-off before they run.

PagerDuty likewise separates suggested remediations from permission to carry them out. Its SRE Agent presents recommendations in plain language, and operators can configure whether it acts or waits for approval. Responders can interact with Paige through the Incident Details Page, Operations Console or PD Reliability Console, Slack, and Microsoft Teams, which is currently in Early Access. Console access requires both AIOps and PagerDuty AI, while Slack and Incident Details Page access require PagerDuty AI. Slack also needs its integration configured and PagerDuty AI connected, and PagerDuty AI is included with PD Reliability Platform plans or available as an add-on with Enterprise, Business, and Professional plans.

Three production risks in this architecture and how the recipe addresses each

Silent context corruption can make an investigation appear complete even though findings have been dropped. The recipe’s bounded window and turn budget constrain how much material the agent processes, while source links leave an artifact the responder can inspect. Those measures reduce exposure but do not prove completeness. Teams using larger log windows should apply additional validation before treating the hypothesis as authoritative. The operator still has to decide whether the evidence at hand supports the conclusion.

Assuming that MCP supplies memory that lasts and has rules for how it is managed creates a second risk. It does not define a memory primitive, and wrapping a memory backend as MCP tools is lossy. AURA places state management at the runtime layer, while Paige exposes a shared memory API that operators can update directly. Operators still have to handle memory governance, even when a memory tool is easy to connect.

The third risk comes from tools that can change connected systems as well as read them. For that reason, the approval configuration must cover all state-changing tools if the intended design requires human authorization. The [hitl] block supplies the gate, and PagerDuty’s granular permissions control which data its agent can access. Neither replaces the operator’s responsibility to configure the relevant patterns and permissions. The production boundary for an agent using MCP is the combination of scoped access and explicit approval.

What teams actually get back and how to measure whether it is working

The immediate return is a first-pass investigation that an on-call engineer would otherwise assemble across tools. Its fixed output can go in the incident channel and postmortem without someone having to write a separate summary. Platform engineers get a reviewable configuration to extend, while SRE leads get a more consistent starting point across the rotation. Security and compliance reviewers get an agent that can run inside the team’s infrastructure, with model and tool calls traceable. These benefits concern the shape and visibility of the work.

Anaplan used AI-driven runbook automation to cut MTTR and removed unnecessary alerts through service-based, AIOps-driven alerting. Cloudflare reduced mean-time-to-action from multiple minutes to seconds with automated alert triage. ResultsCX reduced network failover resolution from tens of minutes to a matter of minutes after automating diagnostics and triage. These examples concern different approaches and outcomes, so they are not benchmarks for what this AURA recipe will deliver.

A pilot can instead track the AURA recipe’s three measures: how long it takes to get from alert to first hypothesis, how well the hypothesis matches the confirmed root cause, and what on-call feedback says about how useful the summary is. AURA’s open-source, curl-installable setup lets teams test it on real incidents before deciding to buy, though they still need to account for operating costs. A platform engineer can extend it with more workers, more sources, or tighter memory while keeping the same measurable contract between an alert and a useful first investigation.