Tracing Agent Runs with Spans

By Michael Samuel Naeem · July 1, 2026 · 4 min read

When an agent fails in production, "the model gave a wrong answer" is not a useful bug report. You need to know: which turn failed, what tools were called, what they returned, how many tokens were burned, and whether a guardrail blocked something silently. Agent observability through distributed tracing — one trace per run, spans per LLM call and tool execution — turns agent debugging from archaeology into engineering. I won't ship an agent without tracing anymore; the first production incident without it always costs more than the instrumentation effort.

Trace structure

One user request = one trace. Everything else is a span:

Trace: agent_run_abc123 (user: "refund order 4521")
├── span: guardrail_input (2ms, pass)
├── span: llm_call_1 (planning) (1.2s, 840 in / 120 out tokens)
├── span: tool:lookup_order (340ms, success)
├── span: llm_call_2 (reasoning) (980ms, 1200 in / 85 out tokens)
├── span: guardrail_action (5ms, requires_approval)
├── span: approval_wait (12min, approved)
├── span: tool:create_refund (520ms, success)
├── span: llm_call_3 (response) (750ms, 900 in / 95 out tokens)
└── span: guardrail_output (3ms, pass)

Parent span is the agent run. Child spans are the work units. This maps directly to how you think about agent execution.

Instrumentation with OpenTelemetry

from opentelemetry import trace

tracer = trace.get_tracer("agent.orchestrator")

async def run_agent(user_message: str, session_id: str):
    with tracer.start_as_current_span("agent_run") as span:
        span.set_attribute("agent.session_id", session_id)
        span.set_attribute("agent.user_message_length", len(user_message))

        with tracer.start_as_current_span("llm_call") as llm_span:
            llm_span.set_attribute("gen_ai.system", "openai")
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            response = await llm.complete(messages)
            llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.input)
            llm_span.set_attribute("gen_ai.usage.output_tokens", response.usage.output)
            llm_span.set_attribute("gen_ai.response.finish_reasons", [response.finish_reason])

        with tracer.start_as_current_span("tool.execute") as tool_span:
            tool_span.set_attribute("agent.tool.name", "lookup_order")
            tool_span.set_attribute("agent.tool.args", json.dumps(args))
            result = await execute_tool("lookup_order", args)
            tool_span.set_attribute("agent.tool.success", result.success)

Use the OpenTelemetry GenAI semantic conventions — gen_ai.request.model, gen_ai.usage.input_tokens, etc. — so traces are portable across backends.

What to capture vs what to avoid

Capture in spans:

• Latency per LLM call and tool
• Token counts and model ID
• Tool name and argument schema (not full PII payloads)
• Success/failure and error type
• Cost estimate per span
• Retrieval scores and chunk IDs (not chunk text)

Do NOT capture in spans:

• Full prompt text with user PII
• Complete tool responses with sensitive data
• API keys or auth tokens

For debugging prompts, use a separate secure debug mode with explicit opt-in, not production tracing.

Dashboards that matter

Build these views from day one:

Run explorer — search traces by session ID, user ID, outcome (success/fail/timeout). Click into span waterfall.

Cost attribution — sum gen_ai.usage.* tokens by agent type, tenant, time period. Pair with budget tracking.

Tool reliability — error rate and p95 latency per tool. A degrading tool shows up here before users complain.

Step distribution — histogram of span count per trace. A bimodal distribution (5 steps vs 40 steps) usually means two distinct failure modes.

Guardrail blocks — count and reason codes. Spikes indicate attack or prompt drift.

Connecting traces to eval

Export failed traces to your eval dataset:

async def on_run_complete(trace_id: str, outcome: str):
    if outcome == "failure" or trace.step_count > STEP_THRESHOLD:
        scenario = await trace_to_scenario(trace_id)
        await eval_dataset.add(scenario, source="production", trace_id=trace_id)

Production failures become eval scenarios automatically. This closed loop is the highest-ROI observability investment for agents.

Sampling strategy

Tracing every span of every run gets expensive in storage. Sample intelligently:

• 100% capture: failures, runs exceeding cost threshold, first N runs per day per agent type
• 10% sample: successful happy-path runs
• 0%: health check / warmup runs

Always capture full traces for failures. The cost of storing a failed trace is negligible compared to the cost of not being able to debug it.

Span naming conventions

Consistent names enable aggregation:

agent.run                    # root span per user request
agent.llm.chat              # LLM API call
agent.tool.{tool_name}      # tool execution
agent.retrieval             # RAG lookup
agent.guardrail.input       # input validation
agent.guardrail.output      # output validation

Attributes on every span:

span.set_attribute("agent.session_id", session_id)
span.set_attribute("agent.tenant_id", tenant_id)
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.usage.input_tokens", input_tokens)

Cost attribution from traces

Sum token attributes per trace → per-tenant daily cost:

SELECT tenant_id, SUM(input_tokens + output_tokens) AS tokens
FROM trace_spans
WHERE span_name = 'agent.llm.chat'
  AND timestamp > NOW() - INTERVAL '1 day'
GROUP BY tenant_id
ORDER BY tokens DESC;

Traces without token attributes are useless for FinOps — instrument at LLM client wrapper, not ad hoc.

Pair with agent evaluation trajectory analysis when exporting failed traces to eval datasets.

Common production mistakes

Teams get observability tracing spans wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Agent systems using observability tracing spans loop infinitely when tool errors are swallowed, subagent budgets have no hard cap, and human-in-the-loop gates are bypassed under latency pressure.

Resources

• OpenTelemetry GenAI semantic conventions
• LangSmith tracing documentation
• Langfuse observability platform
• OpenLLMetry — OTel instrumentation for LLMs
• LLM observability with OpenTelemetry