Trace every agent run with span IDs, tool logs, cost meters, and replay bundles before incidents hit production. Use the checklist. Full breakdown.
Why Agent Runs Need Their Own Observability Layer
An agent run is not a single request. It is a branching sequence of model calls, tool invocations, retries, and intermediate decisions, and any one of them can be where things went wrong. If your only record is the final output, you are debugging blind: you can see that the agent booked the wrong flight or looped forever, but not which step made the bad choice or what input it saw at that moment.
The goal is to make every run reconstructable after the fact. That means capturing the shape of the run as a trace, recording what each tool actually did, tracking what the run cost, and keeping enough state to re-run it. Do this before incidents hit production, because you cannot add instrumentation to a failure that already happened.
Traces and Span IDs
Give every run a unique trace ID and every step within it a span ID. Each span should record its parent, so a nested tool call made during a model turn links back to the turn that triggered it. When you can walk the tree, "the agent hung" becomes "span 14, the database tool, waited 30 seconds and timed out," which is a fixable statement instead of a vague one.
Span timing also exposes the boring failures that dominate real systems: a slow tool, a retry storm, a model call that quietly fell back to a smaller context. Without spans these blur into one number; with them you can point at the exact segment.
Tool Logs, Cost Meters, and Replay Bundles
Structured tool logs are where most root causes actually live, because the model usually behaves reasonably given what it was told. Log both sides of every tool call so you can see whether the agent asked the wrong question or the tool returned bad data. A cost meter attached to each span keeps token and dollar spend visible per run, so a pricing regression or an accidental loop shows up as a spike you can trace to its source rather than an end-of-month surprise.
A replay bundle is the artifact that ties it together: enough captured state to run the same sequence again deterministically. Capture these for each span:
- Full input and output payloads, including system and tool messages
- Tool call arguments and the raw responses they returned
- Model, parameters, and any seed or configuration used
- Timestamps, latency, and token or cost figures
With a bundle you can reproduce a bad run on your own machine, change one thing, and confirm the fix actually addresses the failure instead of guessing.
Using the Checklist Before Production
Treat these as gates, not aspirations. Before an agent ships, confirm that every run emits a trace ID, that spans are nested with parents, that both sides of each tool call are logged, that cost is metered per run, and that a replay bundle is written and can actually be replayed. If any item is missing, you will discover it during an incident, which is the worst possible time.
The payoff is that debugging shifts from re-reading conversations to reading evidence. When something breaks, you open the trace, find the span, read the tool log, check the cost, and replay the bundle — a repeatable process instead of a scramble.