Use decision models with Amazon Bedrock AgentCore Evaluations

Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent on demand during development, in batch jobs against stored sessions, and online against production traffic. AWS Machine Learning Blog's report shows how Jev, the hosted decision model from TypeSafe, and open-source Strands Decider 2B connect through code-based evaluators. Each model takes a structured state and a fixed set of allowed answers and returns a calibrated probability over them. Free-form text has confidence that drifts from run to run.

Engineers evaluating agents on Amazon Bedrock can use this piece before a narrow check gates a release. It covers the HR Assistant path from evaluator input to a probability, plus prerequisites, evaluate.py steps, and the Jev and Strands Decider flags.

Use decision models with Amazon Bedrock: what actually changed

AgentCore Evaluations includes 15 built-in evaluators and custom evaluators defined as an LLM-as-a-judge or a code-based evaluator. A custom LLM-as-a-judge applies the team's instructions and rating scale with a model in Amazon Bedrock. A code-based evaluator runs the team's logic in an AWS Lambda function, which can call an external system, model, or API, including a decision model. An informed person could decide these at a glance: did this response stay on policy, is this tool call safe to approve, and does this answer match the retrieved context?

Jev and Strands Decider can each answer several questions about the same state in a single request. Either backend can screen metrics with a pass or fail. The built-in evaluators' explanations then cover what went wrong. After a question is validated on the team's own traffic, a decision model can be sufficient for that decision or score. The same approach extends to specific harmfulness or stereotyping categories in a policy. Strands Decider is the open-source, 2-billion-parameter model from the Strands Agents team.

Use decision models with Amazon Bedrock: how it works

A small, specialized decision model returns a probability distribution that a program can threshold, aggregate, and act on. A general-purpose LLM evaluator produces a score with a rationale. A noul question is yes/no and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies and returns the most probable option, the full distribution, and a confidence. A score question asks which of 2 to 10 ordered levels applies and returns a probability-weighted score, the distribution over levels, and a confidence.

"Is every fact in this response supported by the tool results?" is a yes/no question. "How helpful was this response, on a four-level rubric?" is a score. "Is this response grounded?" and "Did the agent complete the user's goal?" are fixed-choice checks. JevGroundedness and DeciderGroundedness are trace-level noul evaluators that label a turn Grounded or Ungrounded and report P(grounded). JevHelpfulness and DeciderHelpfulness are trace-level 4-level score evaluators that return the most probable level and the expected level scaled from 0 to 1. JevGoalCompletion is the third Jev evaluator, and Strands Decider 2B backs the same three. The Lambda turns evaluator input into a decision-model request and the probabilities into the evaluation result.

Use decision models with Amazon Bedrock: why it matters

Built-in evaluators in AgentCore use a general-purpose LLM to assess the response against a rubric and explain the judgment. The explanation carries the result when the work is connecting evidence, identifying missing information, or working through a response in depth. A decision model fits when the team wants a well-calibrated confidence score beside the label, and when a rubric must be applied the same way on every turn. The sample runs deterministic code checks, built-in LLM-as-a-judge evaluators, and Jev or Strands Decider 2B evaluators together on the HR Assistant.

On-demand evaluation uses EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance. An online configuration scores live sessions at 100 percent sampling. Decision-model evaluators are opt-in and run beside that flow with no changes to it. Where ground truth exists, a team can confirm whether each evaluator catches the right failures. The decision model supplies a probability the release gate can threshold. The built-in evaluator supplies the explanation for the turns the screen selects.

Use decision models with Amazon Bedrock: who is affected

The example is an HR Assistant built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. Five tools return fixed mock data: paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. Fixed data makes it straightforward to verify behavior from the LLM-as-a-judge and from the decision models. The sample is the framework for that comparison on a team's own cases, with TypeSafe's hosted model or a Strands Decider server the team runs.

The sample defaults to US West (Oregon), in a Region where AgentCore Evaluations is available. Amazon CloudWatch Transaction Search must be enabled, with access to a foundation model in Amazon Bedrock. Local tools are Python 3.12 or later, the AWS CLI, and Docker. Jev requires a TypeSafe API key, with access details at typesafe.ai. Strands Decider requires its server locally for testing, or on Amazon EC2 inside a VPC so the Lambda can reach it. The AgentCore samples GitHub repository holds the code, and ../utils/deploy.py deploys the HR Assistant to AgentCore runtime.

Use decision models with Amazon Bedrock: what to watch

Flags add the decision models on top of those steps. --with-jev and --jev-secret-arn, passed an ARN, deploy jev-evaluator and register JevGroundedness, JevHelpfulness, and JevGoalCompletion. --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. --decider-ec2 auto-provisions the Amazon EC2 instance, security groups, IAM role, and Lambda VPC configuration without manual console steps. hr-response-length and hr-fact-checker remain. Validate the question on the team's own traffic before a groundedness probability or a four-level helpfulness score gates a release.