Cut generative AI costs: Code for rules, models for judgment
AWS detailed architectural patterns showing how deterministic code and fast paths cut generative AI costs by replacing unneeded foundation model calls.
By Dillip Chowdary • Nov 18, 2026 • Source: AWS Machine Learning Blog

At a glance
- Evaluation framework
- Three-question workflow evaluation
- Primary platforms
- Amazon Bedrock and AWS Lambda
- Workflow orchestrator
- AWS Step Functions with JSONata
- Edge validation
- Amazon API Gateway REST API
- Open source model
- Strands Decider under Apache 2.0
- Model accuracy metric
- 95 percent accuracy at 0.9 confidence
In this post
Amazon Web Services published architectural guidance detailing how engineering teams can cut generative artificial intelligence operating expenses by replacing unneeded model calls with deterministic code. According to AWS Machine Learning Blog's report, routing routine tasks like input validation and intent classification through large language models inflates production budgets while stacking invocation latency. Skipping foundation models whenever deterministic rules or lightweight decision logic can handle the step removes unnecessary expenses and system variance.
This technical analysis covers the evaluation framework for auditing generative workflow steps and mapping them directly to native cloud services. Engineering leads, solutions architects, and software developers can apply this three-tier pattern to cut operational costs across production pipelines without degrading task accuracy. The strategy balances deterministic rules in serverless compute, constrained classification components, and right-sized Amazon Bedrock foundation models.
01Production spend and latency stacking
Many generative software implementations introduce excessive model calls during early prototyping phases. A foundation model successfully executes an initial prototype step, which leads engineers to adopt model invocations by default for basic downstream tasks such as validating inputs, reformatting payload schemas, and routing requests. While these invocations appear inexpensive during low-volume testing, they rapidly compound into major architectural expenses once traffic scales to enterprise production.
Unneeded model calls impose three major engineering penalties across scaled production environments. Financial expenditure grows directly with total request volume even for low-value workflow steps. In addition, chaining sequential model calls compounds latency because each remote inference call adds hundreds of milliseconds, while variable model outputs degrade test reliability and slow down compliance audits.
02Three-question workflow evaluation framework
The decision methodology relies on a sequential evaluation funnel that categorizes each workflow action into code, questions, or foundation models. Architects evaluate tasks against three questions in fixed order and halt at the first matching implementation mechanism. Open-ended generative tasks proceed to Amazon Bedrock models only when deterministic code and constrained classifiers cannot resolve the requirement.
- Rule verification Check whether a deterministic rule can verify success through a format, schema, keyword, or range.
- Rule authoring Determine whether a model can write the rule once as a regular expression or schema for human review and commit.
- Fixed list classification Identify whether the step selects a single outcome from a predetermined list of options or routes.
- Model execution Route open-ended tasks that fail all prior checks to a right-sized Amazon Bedrock model.
Applying this evaluation framework to a customer support assistant separates basic formatting from open-ended dialogue. Formatting an order number resolves immediately through Amazon API Gateway request validation or AWS Lambda code. Similarly, routing customer intent can be split, allowing an AWS Step Functions Choice state to handle keywords like refunds while ambiguous inquiries escalate to constrained classification.
03Deterministic code and model trade-offs
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Selecting between deterministic code and generative foundation models requires weighing explicit operational trade-offs across cost, speed, and governance. Developers should reserve language models for ambiguous, generative assignments such as summarizing documents, extracting meaning from prose, and drafting responses. Standard operations with clear criteria, such as checking numeric ranges or validating email structures, belong in standard application code.
| Property | Deterministic code | Model call |
|---|---|---|
| Cost per invocation | Typically orders of magnitude lower | Per-token, non-trivial at scale |
| Latency | Single-digit milliseconds or less | Hundreds of milliseconds |
| Correctness on a well-specified rule | Exact for the rule as written | High, but with a non-zero error rate |
| Auditability | The rule sits in source control | Harder to explain to an auditor |
| Behavior over time | Stable until you change it | Changes when you move to a new model version, or when a version is retired |
Regulated software environments favor deterministic rules because explicit source-controlled logic guarantees auditable consistency. Foundation models introduce hallucinations, baseline error rates, and behavioral drift whenever cloud providers update or retire underlying model versions. Software architects can apply a straightforward operational test: any task whose acceptance criteria can be defined as a deterministic rule should be implemented directly as code.
04Constrained decision calls and classification
Intermediate workflow stages like classification and routing rarely require free-form text generation. Instead of invoking large generative models, systems should use constrained components that select an answer from a predefined enum list and emit a confidence signal. Confident decisions continue through automated processing, while uncertain predictions escalate to downstream foundation models or human reviewers based on labeled calibration data.
Teams can construct constrained classification logic on AWS using four distinct technical mechanisms. These options include small Amazon Bedrock models enforcing structured JSON outputs with an unsure enum, Bedrock embedding similarity matches, custom classifiers trained on Amazon SageMaker AI, or open source decision models. For structured JSON outputs, Amazon Bedrock requires schemas to set additionalProperties to false.
Note: Strands Decider is an experimental open source decision model distributed under the Apache License 2.0, reporting roughly 95 percent accuracy at confidence levels of 0.9 or higher on short classification benchmarks. Teams must self-host the model on Amazon SageMaker AI endpoints and validate confidence thresholds against internal labeled data before production deployment.
05Deterministic fast paths and committed rules
The deterministic fast-path architecture processes common requests through lightweight rules before invoking any artificial intelligence components. For instance, customer intent routing often contains obvious keywords like refund or password reset that native logic resolves in single-digit milliseconds. If deterministic checks resolve 95 percent of incoming transactions, only 5 percent of total traffic ever reaches a billable foundation model.
Architects can compose this fast-path pattern using native cloud services. Amazon API Gateway request validation drops malformed inputs at the edge, while AWS Step Functions Choice states parse incoming payloads with JSONata expressions to divert known intents. AWS Lambda functions execute complex programmatic rules, ensuring that downstream Amazon Bedrock models process only ambiguous, non-standard workflow residuals.
Engineering teams can also utilize generative models during development to author deterministic rules a single time. A foundation model can draft complex regular expressions or JSON schemas that engineers review, test, and commit directly to version control. At runtime, these committed rules execute inside AWS Lambda functions at code execution speeds and negligible invocation costs.
Frequently asked questions
When should a generative AI workflow step use deterministic code instead of a model?
Deterministic code should handle any task that has verifiable rules, such as checking schemas, formats, ranges, or keywords. If acceptance criteria can be written as a rule, the step should run as code to reduce cost and latency.
How does a deterministic fast path reduce generative AI expenses?
A fast path checks predictable inputs with rules before calling a model, such as matching keywords in an AWS Step Functions Choice state. If rules resolve the majority of requests, only the remaining unresolved requests reach foundation models.
What is Strands Decider and how is it deployed on AWS?
Strands Decider is an experimental open source decision model licensed under Apache 2.0 that provides calibrated confidence scores for classification. It can be hosted on an Amazon SageMaker AI endpoint and evaluated against labeled domain data.
How do structured outputs prevent open-ended text generation in classification steps?
Structured outputs constrain model responses to a predefined enum list, such as specific queue names and an unsure option, while requiring additionalProperties set to false in the JSON schema.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement