How to Monitor Agents in Production: Implementing Observability and Tracing
When an agent goes wrong in production, it usually doesn't crash. It just silently wastes money. This post covers what traces and spans a production agent should emit, which fields to log, and the four metric categories to track so you catch runaway behavior before costs spike or the system fails.
By
Tenten AI 研究團隊
應用 AI
Published
February 8, 2026
Read time
5 分鐘
Agent observability uses structured traces, spans, and metrics to capture your agent's reasoning, tool calls, and token usage from request arrival through response. This lets you see what happened and set up alerts. Traditional monitoring asks if something failed. Your agent makes its own decisions, so you need to ask why it decided to act that way.
A customer's research agent had its bill spike 40x at 3 AM. The culprit was a malformed PDF that trapped the agent in a retrieve-fail-retrieve loop, making over 900 tool calls without anyone knowing. The system didn't crash. No errors were thrown. The logs just said "task completed." That night's cost could have funded an engineer for a month to build monitoring infrastructure.
The real problem was our logging. We recorded the input and final output, nothing else. Everything in between was completely invisible.
What Should One Agent Execution Record in Traces and Spans
Start with definitions. A trace is one complete agent execution, from user request to final result. A span is one step within that trace. Carve out each agent decision and action into its own span, then link them all with the same trace ID.
In practice, a production agent should log at least these layers of spans with these key fields:
| Span Type | What to Log | Why It Matters |
|---|---|---|
| LLM reasoning | prompt, response, model, input/output tokens, latency | Catch cost overruns and prompt drift |
| Tool calls | tool name, arguments, return value, success or failure, duration | Catch loops and external dependency failures |
| Retrieval (RAG) | query, retrieved document IDs and scores | Catch retrieval misses causing hallucinations |
| Decision node | which path the agent selected and why | Rebuild the reasoning chain afterward |
| Guardrails | which rule triggered, what was blocked | Audit trail and compliance evidence |
Four fields are essential: trace_id, parent_span_id, token 數, and 工具呼叫次數. Without the parent ID, you can't trace this step to the earlier step that triggered it. Without tool call count, you can't catch sessions like the one that cost us that night.
Alert on Runaway Signals, Not Error Rates
Most teams only alert on HTTP 500. But the most dangerous agent behavior doesn't error. It just quietly goes wrong, does too much, or runs too long. That means rethinking how you alert.
We attach four alert categories to every agent we ship to production. First is cost: alert if a trace's token count or tool call count exceeds 2x the P95. This rule would have stopped the 40x bill by the 20th call. Second is loops and runaway: if the same tool is called more than a threshold (say 10 times) in a single trace, you have logic spinning. Third is quality: low retrieval scores or rising guardrail rejection rates signal the agent is guessing. Fourth is latency tail: watch P95, not average, because long-tail requests usually mean the agent is stuck in a loop.
These metrics don't have universal thresholds. Run your agent in canary for a week or so, collect the real distribution, then set thresholds at the edge of that distribution. Don't guess at numbers.
Make Observability Part of Launch, Not a Post-Mortem Patch
Many teams only add traces after something breaks. That's backward. Observability isn't just a debugging tool. It's a requirement for shipping to production. An agent with no traces is like putting an employee in front of customers and not being able to see what they're doing.
You don't have to build this yourself. OpenTelemetry already defines semantic conventions for GenAI spans, and tools like LangSmith, Langfuse, and Arize Phoenix can consume the standard format directly. What matters is embedding trace_id in every call on day one.
At Tenten, we ship the monitoring dashboard and alert rules alongside every agent. For us, an agent you can't see in action isn't truly in production.
One stuck workflow
is enough to begin
Tell us what the team does today, where it breaks down, and what a better working day should look like.