Datadog Brings Production Experimentation to AI Agent Observability
Datadog added Online Experiments to Agent Observability on November 17, 2026, finding a 44.3% cost reduction when comparing GPT-6 Luna to GPT-5.6 Luna.
In short
Engineering teams building AI agents · 44.3% estimated cost reduction · November 17, 2026 · Split traffic with feature flags and inspect traces
By Dillip Chowdary • Nov 17, 2026 • Source: InfoQ

At a glance
- Who it is for
- Engineering teams building AI agents
- The number
- 44.3% estimated cost reduction
- Available
- November 17, 2026
- What to do
- Split traffic with feature flags and inspect traces
- Tested models
- GPT-5.6 Luna and GPT-6 Luna
- Quality score
- 3.84 versus 3.86 out of 5
In this post
Datadog expanded its Agent Observability platform on November 17, 2026, by launching Online Experiments to compare autonomous agent versions against live production traffic, delivering an observed 44.3% estimated cost reduction in dual-model testing across active workloads, according to InfoQ's report.
This reporting is for software engineering and platform teams operating AI agents that face unpredictable real-world prompts. Engineers can immediately use the platform to split live traffic with feature flags, assess response quality, and inspect granular execution traces.
01Online experiments for agent observability
Datadog is extending its Agent Observability platform with an Online Experiments capability designed to evaluate AI agent variants against real production traffic. The capability pairs agent evaluation scores directly with operational telemetry, including latency, token consumption, execution cost, and error rates. This setup helps software teams verify whether changes to underlying models, prompts, retrieval strategies, or tools actually improve overall agent performance.
The release targets an operational challenge where agent modifications that perform well in controlled offline tests fail when exposed to unpredictable user requests. Controlled offline evaluations remain useful for testing agents against predefined datasets to catch regressions before initial deployment. However, predefined test datasets only capture known scenarios, leaving production agents vulnerable to novel workflows, unexpected user inputs, and shifting third-party tool responses.
Online experimentation addresses the blind spots of synthetic evaluations by assessing agent behavior during live user interactions. Datadog connects controlled experimentation, production telemetry, and execution traces into a single view. This combination gives software engineers verifiable evidence regarding both the final output quality and the step-by-step system behavior behind every response.
02Production traffic distribution and feature flags
The platform allows teams to compare an existing control agent against a modified variant by randomly distributing incoming production traffic between them. This randomized allocation balances differences across varied user requests to ensure fair comparisons under identical real-world conditions. Operating against actual workloads exposes edge cases that engineering teams can capture and convert into permanent additions to their offline regression suites.
Datadog integrates its Agent Observability evaluators directly with its Experiments capability to display response quality beside standard infrastructure telemetry. Teams track response quality, factual accuracy, and output completeness while simultaneously monitoring execution duration, token counts, estimated financial cost, and workflow error rates. Comparing these numbers in a shared dashboard reveals the practical tradeoffs that accompany configuration changes.
Traffic allocation is managed through built-in Feature Flags, allowing engineers to release modifications gradually to a small fraction of users before expanding access. This staged rollout prevents broad service disruptions when an update introduces unintended side effects. System modifications rarely improve all performance vectors simultaneously, meaning teams must balance operational cost against execution speed and answer accuracy.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Operational trade-offs in agent execution
System improvements in autonomous architectures frequently require compromises between computational overhead and response quality. For example, switching to a cheaper model variant can reduce spending while requiring more retries to complete a task. Similarly, an updated retrieval strategy might supply higher quality reference evidence to the model while significantly increasing overall execution duration for end users.
To clarify why variants perform differently, Datadog links every experiment outcome directly to individual agent execution traces. Engineers can open an individual trace to examine exact tool invocations, supplied arguments, and returned tool payloads. This granular trace data explains why an agent version generated a specific response or failed during execution.
04Luna model comparison and evaluation results
An experiment detailed by Datadog tested GPT-5.6 Luna against GPT-6 Luna by splitting production traffic evenly between the two models. An LLM-based evaluator scored investigation quality on a scale from one to five, measuring accuracy, completeness, and evidence grounding. The newer GPT-6 Luna achieved a 44.3% reduction in estimated cost while maintaining comparable quality scores of 3.86 versus 3.84 for GPT-5.6 Luna.
| Model variant | Quality score | Cost impact | Operational findings | | GPT-5.6 Luna | 3.84 / 5 | Baseline cost | Baseline duration and error rates | | GPT-6 Luna | 3.86 / 5 | 44.3% reduction | Elevated duration, errors, and parameter retries |
Note: The operational metrics revealed potential concerns despite the lower cost. The newer model showed estimated increases in execution duration and workflow errors, although confidence intervals were too broad to establish definitive regressions. Inspecting the traces showed that the newer model supplied invalid tool parameters more frequently before recovering after retries.
Datadog used the trace findings to improve tool handling schemas before proceeding with the planned model migration. The invalid tool invocations captured during the live experiment also provided realistic test cases for the team's offline regression suite. This prevented a premature rollout that would have increased latency across production environments.
05Continuous feedback loops and competing platforms
Datadog operates alongside competing platforms in the expanding agent monitoring market. Frameworks such as LangSmith and Langfuse offer execution tracing and evaluation tools, while rollout systems like LaunchDarkly manage feature gating. Datadog differentiates its offering by combining agent evaluation, feature flag rollouts, and infrastructure telemetry directly within its established monitoring platform.
The broader technical strategy establishes a continuous feedback loop between production interactions and offline development testing. Unexpected agent failures and unhandled edge cases discovered during online experiments become new test items in offline evaluation datasets. Subsequent online experiments then verify whether prompt, tool, or code adjustments successfully resolve those problems under real workloads.
Frequently asked questions
What is Datadog Online Experiments for Agent Observability?
It is an extension to Datadog's Agent Observability platform that lets engineering teams evaluate AI agent versions against live production traffic. The tool pairs agent evaluation scores with operational metrics such as latency, token consumption, execution cost, and workflow error rates.
How did GPT-6 Luna compare to GPT-5.6 Luna in Datadog's experiment?
GPT-6 Luna reduced estimated cost by 44.3% while maintaining a quality score of 3.86 compared to 3.84 for GPT-5.6 Luna on a five-point scale. However, operational telemetry showed higher execution duration, more workflow errors, and frequent invalid tool parameter calls.
How does Datadog manage traffic distribution between agent variants?
Datadog uses Feature Flags to randomly distribute production traffic between control agents and modified variants. This allows engineering teams to expose changes to a small percentage of users before initiating a wider rollout.
How does Datadog differ from tools like LangSmith, Langfuse, and LaunchDarkly?
LangSmith and Langfuse specialize in tracing and evaluation, while LaunchDarkly handles feature rollouts. Datadog integrates evaluation scoring, feature flag rollouts, execution traces, and operational telemetry within its unified observability platform.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement