A Better Benchmark Harness for SRE Agents

PagerDuty launched an automated evaluation framework designed to score site reliability engineering AI agents against synthetic, live-injected infrastructure outages. As detailed in pagerduty.com's report, the testing environment injects faults directly into a proprietary Kubernetes-based microservices architecture, requiring participating systems to pinpoint issues, determine root causes, suggest remediation steps, and present grounded evidence without relying on pre-baked training shortcuts.

This breakdown examines the technical architecture of the benchmark harness, its multi-model evaluation mechanics, and its initial comparative evaluation across seventeen operational scenarios. Engineering leaders, platform teams, and site reliability specialists evaluating autonomous agents will see how synthetic fault injection exposes model reasoning limitations and context constraints during complex distributed systems failures.

A Better Benchmark Harness for SRE Agents: what actually changed

PagerDuty shifted away from evaluating operational assistants through historical incident replays, establishing an automated fault injection harness built on a private microservices stack. Historical post-incident reviews often reflect human-focused write-ups that lack uniform telemetry, mask true system states, or suffer from hindsight leakage where postmortems, triage transcripts, and remediation pull requests contaminate the reference context. By causing active outages in a controlled environment, the harness ensures exact ground truth while evaluating agents through their native interfaces and standard SaaS integrations.

The evaluation process also moves away from public synthetic environments such as the OpenTelemetry Astronomy Shop or DeathStarBench. Because those open-source test applications often exist within common large language model pretraining datasets, evaluated agents can memorize failure patterns or recall fault locations directly from training corpora. PagerDuty's closed benchmark relies on a purpose-built financial application running on Kubernetes, preventing models from leaning on prior memorization during root cause analysis.

A Better Benchmark Harness for SRE Agents: how it works

The benchmark harness introduces four distinct fault injection techniques: Helm value overrides, Chaos Mesh disruptions against the Kubernetes substrate, live pull request merges, and changes enacted across the AWS control plane. Telemetry generated by the application flows into Grafana alerting rules, which subsequently dispatch alerts into PagerDuty or the agent platform under review. Agents receive standard access to metrics, system logs, alert streams, and raw source code repositories to carry out automated root cause identification.

Performance evaluation uses an automated panel of judges drawn from three distinct model providers to prevent single-family scoring bias. The judges score agent responses against predefined criteria established during scenario design across five dimensions: locating the fault, diagnosing root causes, recommending mitigations, grounding statements in evidence, and clarity of reasoning. A dedicated contamination scanner inspects sweeps for telemetry leakage, while neutral prompt templates ensure differences in output format, such as structured JSON versus chat responses, do not penalize evaluated agents.

A Better Benchmark Harness for SRE Agents: why it matters now

Evaluating autonomous operational tooling has become difficult due to the proliferation of general-purpose and specialized SRE assistants across modern engineering teams. Basic assistants can be assembled quickly by pairing commercial language models with observability endpoints through the Model Context Protocol, but measuring whether incremental complexity yields genuine problem-solving gains requires reproducible, objective quantification. Without isolated benchmarking environments, engineering organizations risk deploying operational agents based on synthetic demos that fail during unscripted production outages.

Initial data from seventeen benchmark scenarios showed that PagerDuty's SRE Agent outpaced competing systems across issue localization, diagnostics, evidence grounding, mitigation proposals, and explanation clarity. The report demonstrated that two architectural attributes primarily dictate operational performance: the underlying reasoning strength of the model and the total volume of relevant context the assistant can retrieve. These findings provide platform teams with concrete dimensions to optimize rather than adding superficial integrations.

A Better Benchmark Harness for SRE Agents: who is affected

Platform engineering groups, site reliability teams, and DevOps practitioners managing critical microservices infrastructure directly feel the impact of these findings. Engineering groups attempting to integrate AI agents into incident response loops can establish stricter procurement standards by demanding live fault-injection verification instead of relying on vendor-supplied postmortem replays. This evaluation standard clarifies whether an agent requires access to active source code, historical alerts, and telemetry pipelines to succeed.

Teams developing internal incident-response agents must also reconsider evaluation practices that depend on past incident tickets. Because production postmortems frequently record human consensus rather than clean operational signals, relying on historical records risks evaluating agents on incomplete or skewed data. Adopting controlled multi-vector injection allows teams to safely stress-test custom skill files, instruction sets, and subagents prior to granting operational privileges across production clusters.

A Better Benchmark Harness for SRE Agents: what to watch

The current benchmark run intentionally excluded several high-value context channels that operate in live production, including Scribe Agent incident call transcripts and persistent organizational memory systems. While conversational context between human responders assists complex triage, synthesizing realistic incident audio without exposing answers remains difficult. Future benchmark iterations will address the integration of historical memories and safe call-transcript feeds while accounting for customized subagents, skills, and specialized instruction files.

Although PagerDuty is currently keeping the financial microservices repository private to avoid training data contamination, the organization plans to make the codebase available to external researchers to validate initial comparative findings. Observers should track whether the benchmark expands beyond its four current injection vectors into multi-cloud failure modes, external third-party service outages, and security-related incidents, as well as how third-party SRE agents adapt to multi-model judge panels.