Distributed Tracing Across Services

By Michael Samuel Naeem · June 13, 2025 · 6 min read

A user reports checkout took 12 seconds. Your API gateway logs show 200ms. The order service logs show 800ms. The payment service logs show nothing — it was called but has no record because the request ID format differs. Three services, three log formats, no way to connect them into one timeline.

Distributed tracing links every operation in a request into a single trace — a waterfall showing exactly where time was spent and which service caused the delay. OpenTelemetry is the standard that makes this work across languages and vendors.

Traces, spans, and context

Trace (trace_id: abc123)
├── Span: API Gateway (200ms)
│   ├── Span: Auth check (15ms)
│   └── Span: Order Service call (170ms)
│       ├── Span: Validate order (5ms)
│       ├── Span: DB query (20ms)
│       └── Span: Payment Service call (140ms)
│           ├── Span: Fraud check (30ms)
│           └── Span: Charge card (100ms)  ← the bottleneck

Each span records: operation name, start/end time, status, attributes (HTTP method, status code, DB statement), and events (log messages within the span).

Instrumenting with OpenTelemetry

Automatic instrumentation for Python:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317")))
trace.set_tracer_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()

This automatically creates spans for incoming HTTP requests and outgoing HTTP calls, propagating trace context via headers.

Manual spans for business logic:

tracer = trace.get_tracer("order-service")

async def process_order(order_id: str):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)

        with tracer.start_as_current_span("validate_inventory"):
            inventory = await check_inventory(order_id)

        with tracer.start_as_current_span("charge_payment") as payment_span:
            payment_span.set_attribute("payment.amount", order.total)
            result = await payment_service.charge(order)

        span.set_attribute("order.status", "confirmed")
        return result

Context propagation

W3C Trace Context headers pass trace identity between services:

GET /payments/charge HTTP/1.1
traceparent: 00-abc123def456-789abc-01
tracestate: vendor=value

OpenTelemetry SDKs inject and extract automatically for supported HTTP clients and servers. For message queues, propagate context in message headers:

from opentelemetry.propagate import inject, extract

# Producer: inject context into message headers
carrier = {}
inject(carrier)
kafka_producer.send("orders", value=payload, headers=list(carrier.items()))

# Consumer: extract context and create linked span
context = extract(dict(msg.headers()))
with tracer.start_as_current_span("process_order_event", context=context):
    process(json.loads(msg.value()))

Without context propagation in async messaging, traces break at queue boundaries.

Sampling strategies

Tracing every request at 10,000 RPS generates unsustainable data volume.

Head-based sampling — decide at trace start:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

provider = TracerProvider(sampler=TraceIdRatioBased(0.1))  # 10% of traces

Tail-based sampling — collect all spans, export selectively (requires OpenTelemetry Collector):

# otel-collector-config.yaml
processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Tail-based sampling keeps all error and slow traces while sampling 5% of normal traffic — the best of both worlds but requires collector infrastructure.

Debugging with traces

Finding the 12-second checkout in Jaeger or Grafana Tempo:

1. Search by trace ID (from response header or access log).
2. Or search by service + min duration > 5s.
3. Open the trace waterfall.
4. Identify the widest span — that is where time was spent.
5. Click into that span's attributes for details.

Common findings:

• Missing database index: DB query span takes 8 seconds.
• N+1 queries: 50 sequential DB spans instead of one batch.
• Slow external API: payment service span waits 10 seconds.
• Missing timeout: span shows 30-second gap with no child spans.

Adding trace ID to logs

Correlate traces with logs for deeper debugging:

import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, '032x') if ctx.trace_id else 'none'
        return True

logging.getLogger().addFilter(TraceIdFilter())
# Log format: "%(asctime)s [%(trace_id)s] %(message)s"

Search logs by trace ID to see application logs alongside the trace waterfall.

Treat production rollout as a measured change: ship with observability, validate rollback, and review metrics 24 hours after deploy — patterns that look obvious in docs fail when skipped under release pressure.

Resources

• OpenTelemetry tracing documentation
• W3C Trace Context specification
• Jaeger distributed tracing
• Grafana Tempo (trace storage)
• OpenTelemetry Python auto-instrumentation

Production notes for LLM stacks

When microservices-distributed-tracing sits on an inference or RAG path, treat user prompts and retrieved chunks as untrusted input. Log correlation IDs and policy decisions—not raw prompts—in production telemetry. Gate risky operations behind explicit authorization at the gateway, not inside ad-hoc tool handlers.

Roll out changes with shadow mode first: record what would have happened under the new rule without blocking traffic. Compare deny rates, latency impact, and false positives for at least one business week before enforcing. Pair enforcement with a runbook entry: symptom, dashboard, rollback (feature flag or config), and owner.

Load-test with production-shaped concurrency. LLM workloads burst differently from CRUD APIs—tail latency and token throttling dominate. If distributed tracing across services protects an invariant (security, billing, data residency), prove the invariant with an automated test that fails CI when someone removes the check.

What teams get wrong

Teams copy a reference architecture without matching their compliance tier, then discover in audit that logs, backups, or support exports reintroduced the data they thought they had eliminated. Another pattern: shipping the demo integration without idempotency, then fighting duplicate side effects when clients retry on model timeouts.

Document the tradeoff you chose—strictness vs recall, cost vs quality, sync vs async—and the metric that tells you if the choice still holds six months later.

Production notes for LLM stacks

When microservices-distributed-tracing sits on an inference or RAG path, treat user prompts and retrieved chunks as untrusted input. Log correlation IDs and policy decisions—not raw prompts—in production telemetry. Gate risky operations behind explicit authorization at the gateway, not inside ad-hoc tool handlers.

Roll out changes with shadow mode first: record what would have happened under the new rule without blocking traffic. Compare deny rates, latency impact, and false positives for at least one business week before enforcing. Pair enforcement with a runbook entry: symptom, dashboard, rollback (feature flag or config), and owner.

Load-test with production-shaped concurrency. LLM workloads burst differently from CRUD APIs—tail latency and token throttling dominate. If distributed tracing across services protects an invariant (security, billing, data residency), prove the invariant with an automated test that fails CI when someone removes the check.

What teams get wrong

Teams copy a reference architecture without matching their compliance tier, then discover in audit that logs, backups, or support exports reintroduced the data they thought they had eliminated. Another pattern: shipping the demo integration without idempotency, then fighting duplicate side effects when clients retry on model timeouts.

Document the tradeoff you chose—strictness vs recall, cost vs quality, sync vs async—and the metric that tells you if the choice still holds six months later.

For microservices-distributed-tracing, treat observability and security controls as part of the user experience: silent failures erode trust faster than explicit error messages. Instrument deny paths, measure tail latency, and review dashboards with on-call weekly.