The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon

Amazon Web Services detailed an evaluation framework for artificial intelligence agents running on Amazon Bedrock AgentCore Evaluations, pairing native AWS evaluators with managed open-source frameworks. As outlined in AWS Machine Learning Blog's report, the architecture enables engineering teams to verify production agent execution across trace-level tool calls and complete conversational sessions. The system addresses critical blind spots where high resolution rates mask silent policy failures, task omissions, or data leaks.

This analysis details how development teams match specific agent failure modes to targeted automated judges across native AWS benchmarks, DeepEval, and AutoEval libraries. It provides cloud architects, machine learning engineers, and software operators with the operational rules required to evaluate agent reliability without building standalone evaluation pipelines. The blueprint covers behavioral translation, third-party managed scoring, custom judge definitions, and multi-dimensional scoring protocols.

01Business outcomes and behavioral mapping

Evaluation workflows begin by translating broad operational targets into concrete, observable runtime behaviors rather than picking arbitrary metrics from an available catalog. A customer support agent targeted at first-contact resolution must reliably achieve stated goals, maintain coherence across clarification turns, and call backend systems correctly. Research assistants require complete answers grounded entirely in retrieved documents, while content moderation agents must avoid false positives and false negatives.

Engineers separate these behaviors across two distinct execution tiers to ensure metrics measure the proper scope of activity. Trace-level metrics inspect granular per-turn interactions and specific tool calls, verifying whether the agent invoked backend tools with correct parameters. Session-level metrics assess end-to-end outcomes across entire dialogues, capturing whether the conversation completed the overarching task or preserved compliance boundaries throughout the session.

02Built-in catalog and trajectory checks

The native catalog includes fifteen pre-configured evaluators authored and benchmarked by AWS, accessible directly through resource identifiers like Builtin.GoalSuccessRate and Builtin.Helpfulness. Eight response quality evaluators score correctness, faithfulness, helpfulness, response relevance, conciseness, coherence, instruction following, and refusal behavior. Dedicated categories assess goal success rate across full sessions, tool selection accuracy, parameter formatting, and skill execution fidelity alongside safety checks for harmfulness and stereotyping.

Deterministic tool monitoring relies on three ground-truth trajectory evaluators that inspect execution order against predefined benchmarks. TrajectoryExactOrderMatch verifies that every tool call strictly mirrors an expected path, while TrajectoryInOrderMatch permits intermediate steps as long as required calls follow the right sequence. TrajectoryAnyOrderMatch confirms that an agent invoked all necessary tools regardless of intermediate order, allowing teams to audit workflow compliance against operational runbooks.

03Managed DeepEval and AutoEval metrics

AgentCore hosts open-source evaluators from DeepEval directly on managed infrastructure, eliminating the requirement to provision external evaluation clusters. DeepEval provides task completion, goal accuracy, and conversation completeness metrics alongside turn relevancy, knowledge retention, and summarization checks. It also supplies safety benchmarks covering bias, toxicity, and PII leakage, which catch agents that resolve user tickets while inadvertently exposing personal data in replies.

Specialized operational edge cases use AutoEval to identify behaviors standard quality metrics fail to register during session execution. The security evaluator flags attempts to inject malicious instructions or reveal underlying system prompts through user inputs. AutoEval also introduces a possible evaluator that detects over-cautious agents that incorrectly abandon valid requests, as well as humor evaluators for consumer-facing systems where conversational personality is essential.

04Custom judges and prompt templates

Development teams can implement custom model-based judges by modifying prompt templates, rubrics, and underlying foundation models to address proprietary requirements. These customized evaluators derive from built-in or third-party baselines while executing on dedicated model capacity specified by the development team. The prompt template library in the service documentation details the exact scoring rubrics, input formats, and execution levels required for custom configurations.

System operators retrieve available built-in options and registered custom evaluators through a unified ListEvaluators API call. When configuring evaluation suites, teams distinguish between reference-free evaluators that run against live production traffic and evaluators that require ground-truth references. Metrics requiring expected responses or assertions run strictly within offline batch runs, while reference-free evaluators monitor ongoing production interactions continuously.

05Multi-dimension evaluation for independent criteria

Multi-dimension evaluators replace blended aggregate scores by evaluating multiple criteria across a single judge execution. A customer service reply must resolve problems, adhere to refund regulations, and maintain appropriate tone, yet a single score obscures which requirement failed. Multi-dimension evaluators define distinct instructions and numerical or categorical scales for each dimension, allowing a single LLM call to return individual scores for every criterion.

Independent scoring prevents releases that improve conversational tone while secretly regressing data compliance from passing production deployment gates. Teams maintain separate evaluators when specific criteria require distinct foundation models, different execution levels, or offline ground-truth references. By keeping reference-free criteria isolated in their own evaluator configurations, production pipelines maintain automated quality gates without exposing live user interactions to unnecessary latency or compute overhead.