Amazon has extended Amazon Bedrock AgentCore Evaluations, the service that scores AI agents throughout their lifecycle, with support for a new type of evaluator called a decision model. Instead of asking a large language model to write out a judgment and an explanation, a decision model returns a calibrated probability over a small, fixed set of answers. AWS argues this is a better fit for the narrow, repeatable questions that come up constantly in agent testing, such as whether a response stayed on policy or whether a tool call is safe to approve.

What is new

AgentCore Evaluations already ships with 15 built-in evaluators and lets developers write custom ones in two ways: as an LLM-as-a-judge evaluator, which applies a rubric using a model hosted in Amazon Bedrock, or as a code-based evaluator, which runs arbitrary logic inside an AWS Lambda function. The new element described in this post is a way to wire decision models into that code-based evaluator path. AWS demonstrates this using two decision-model backends: Jev, a hosted model from a company called TypeSafe, and Strands Decider 2B, an open-source, 2-billion-parameter model from the Strands Agents team that developers run themselves, for example on an Amazon EC2 instance.

A decision model, as described in the post, is trained to pick from a predefined set of options rather than generate free text. It supports three question shapes: a yes/no question that returns a probability between 0 and 1 (called 'noul'), a choice question with up to 255 possible answers that returns the most likely option plus the full probability distribution, and a score question with two to ten ordered levels that returns a probability-weighted score. Many evaluation rubrics map naturally onto one of these, such as 'is every fact in this response supported by the tool results?' (a yes/no check) or 'how helpful was this response on a four-level scale?' (a score).

How it works

To illustrate the setup end to end, AWS built a sample HR Assistant agent using the Strands Agents framework, deployed on AgentCore runtime. It answers employee questions using five tools returning fixed mock data: PTO balances, PTO requests, HR policies, benefits and pay stubs. A script called evaluate.py deploys two deterministic Lambda-based evaluators (one checking response length, one regex-checking facts against the mock data), registers three built-in evaluators (Correctness, Helpfulness, ResponseRelevance), and runs on-demand evaluation across five labelled scenarios. It also sets up continuous online evaluation of live sessions at 100 percent sampling.

Decision-model evaluators are added as an opt-in layer using command-line flags: --with-jev (plus a secret ARN for the TypeSafe API key) or --with-decider, with an additional flag to auto-provision the EC2 instance needed to host Strands Decider. Each backend then contributes three evaluators to the pipeline, covering groundedness (a trace-level yes/no check), helpfulness (a trace-level four-level score) and goal completion (a session-level three-way choice). A Lambda function receives the agent's OpenTelemetry trace data, reformats it into a request for the decision model, and converts the returned probability distribution back into the label and numeric value AgentCore expects.

  • Decision models answer three question types: yes/no (noul), multi-choice (up to 255 options), and ordered-level scores (2–10 levels)
  • Two backends shown: Jev (hosted, from TypeSafe) and Strands Decider 2B (open-source, self-hosted)
  • Decision models reportedly run in hundreds of milliseconds versus roughly 2–10 seconds for LLM-as-a-judge evaluators
  • Sample pipeline evaluates an HR Assistant agent using deterministic Lambda checks, built-in evaluators, and decision-model evaluators side by side
  • Full code sample is published in the AgentCore samples GitHub repository

Why it matters

For teams running agents in production, evaluation cost and speed matter as much as accuracy. A general-purpose LLM judge can explain why a response failed, which is useful for debugging, but calling a large model for every turn of every session is slower and more expensive than necessary for simple pass/fail checks. AWS frames decision models as a screening layer: run a fast, calibrated check on everything, and only send the turns that fail or sit near the decision threshold to a full LLM evaluator for deeper review. The post references a preprint describing this as a distinction between 'reading' (narrow recognition tasks decision models are suited for) and 'deriving' (the connecting of evidence and reasoning that LLM judges do better). The suggested cascade pattern keeps the built-in evaluators as the foundation for broad quality assessment, while decision models add narrow, validated checks that produce a number teams can threshold and act on automatically.

Limits and open questions

AWS is explicit that decision models are not a replacement for LLM-as-a-judge evaluators; they are an addition for narrow, well-defined checks, and they come with fixed prompt templates rather than open-ended customisation. The post also stresses that rewriting a rubric as a decision-model question effectively creates a new measurement, which can turn out stricter or looser than the original intent. It lays out a five-step validation process before anyone should gate a release on a decision-model evaluator: build a labelled sample of real traffic, get majority-vote labels from three independent human raters, run both evaluator types on the same inputs, check agreement rate and calibration (whether higher confidence scores really do track higher accuracy) broken out by easy and hard cases, and investigate specific disagreements, which often trace back to ambiguous wording in the question itself rather than a model error. AWS also recommends re-running this validation whenever a question is reworded or a model version changes.

How to try it

The sample requires an AWS account in a region where AgentCore Evaluations is available (the sample defaults to US West, Oregon), Amazon CloudWatch Transaction Search enabled, access to a foundation model in Amazon Bedrock, Python 3.12 or later, the AWS CLI, and Docker. Using Jev requires a TypeSafe API key; using Strands Decider requires running its server, either locally or on an EC2 instance reachable from the Lambda function. The GitHub repository provides a deploy.py script to stand up the HR Assistant agent, an evaluate.py script to deploy evaluators and run the evaluation modes with optional decision-model flags, and a cleanup.py script to remove the agent, evaluators, Lambda functions, EC2 instance, and associated AWS resources afterward to avoid ongoing charges.

Sources