Continuous Profiling in Production

By Michael Samuel Naeem · September 30, 2025 · 5 min read

A deploy increased P99 latency from 200 ms to 800 ms. You SSH in, run perf record, capture 30 seconds of data, and find nothing—the spike happened four hours ago during peak traffic. Continuous profiling solves this by sampling stack traces every 10–100 milliseconds around the clock, storing them as time-series data. When latency spikes, you open a flame graph from that exact time window and see which function grew.

How continuous profiling works

Application process
  ↓ (every 10ms)
Sampler collects stack trace → Label with pod, version, region
  ↓
Compress and ship to profile store (Pyroscope, Parca, Datadog)
  ↓
Query by time range + labels → Render flame graph

eBPF profilers attach at the kernel level—no code changes, no agent per language. They read stack frames from /proc or DWARF debug info.

Pyroscope setup

# docker-compose.yml
services:
  pyroscope:
    image: grafana/pyroscope:latest
    ports:
      - "4040:4040"
    volumes:
      - pyroscope-data:/var/lib/pyroscope

  app:
    image: my-app:latest
    environment:
      PYROSCOPE_SERVER_ADDRESS: http://pyroscope:4040
      PYROSCOPE_APPLICATION_NAME: my-app

Go integration:

import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   os.Getenv("PYROSCOPE_SERVER_ADDRESS"),
    Tags: map[string]string{
        "hostname": os.Getenv("HOSTNAME"),
        "version":  os.Getenv("APP_VERSION"),
    },
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileInuseSpace,
    },
})

Python integration:

import pyroscope

pyroscope.configure(
    application_name="my-app",
    server_address="http://pyroscope:4040",
    tags={"region": "us-east-1", "version": "1.2.3"},
)

Reading flame graphs

┌──────────────────────────────────────────────────────────┐
│                      main()                               │  ← root
├──────────────────────┬───────────────────────────────────┤
│   handle_request()   │         startup()                 │
├──────────┬───────────┤                                   │
│ db_query │ json_parse│                                   │
│  (45%)   │   (30%)   │                                   │
└──────────┴───────────┴───────────────────────────────────┘

• Width = proportion of samples (CPU time or memory).
• Y-axis = call stack depth (bottom = entry point).
• Look for wide plateaus — functions consuming disproportionate time.
• Compare profiles — diff flame graphs before/after a deploy.

Parca with Kubernetes

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: parca-agent
spec:
  template:
    spec:
      containers:
        - name: parca-agent
          image: ghcr.io/parca-dev/parca-agent:v0.30.0
          args:
            - --remote-store-address=parca.parca.svc:7070
            - --node=$(NODE_NAME)
          securityContext:
            privileged: true  # required for eBPF

Parca runs as a DaemonSet—one agent per node profiles all pods on that node.

Turning profiles into fixes

Scenario: CPU spike after deploy.

1. Open Pyroscope, filter version=1.2.3, time range = spike window.
2. Flame graph shows json.Marshal at 35% (was 5% in version=1.2.2).
3. Git diff shows new code serializing a 50 MB struct on every request.
4. Fix: serialize only required fields. CPU returns to baseline.

Scenario: Memory leak over 48 hours.

1. Query inuse_space profile type, compare hour 1 vs hour 48.
2. cache.Store grows from 2% to 40% of heap.
3. Cache has no TTL—entries accumulate.
4. Fix: add LRU eviction with 1-hour TTL.

Overhead management

Start with eBPF profiling on 100% of instances at 100 Hz. Reduce sampling rate to 20 Hz if overhead exceeds 5%.

Integrating with traces

Grafana pairs profiles with traces via exemplars:

Trace span (slow request) → "View profile at this timestamp" → Flame graph

This connects "this request was slow" to "this function was hot during that request."

Pyroscope and Parca setup

Deploy continuous profiling alongside existing observability stack:

# docker-compose addition
pyroscope:
  image: grafana/pyroscope:latest
  ports: ["4040:4040"]
  volumes: ["pyroscope-data:/data"]

# Application config (Go example)
import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "api-server",
    ServerAddress:   "http://pyroscope:4040",
    ProfileTypes:    []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileInuseObjects,
    },
})

Zero instrumentation code for supported languages — agent samples automatically. Query by service, pod, and time range in Grafana.

Reading flame graphs

Flame graph interpretation for production debugging:

Width  = time spent in function (wider = more CPU)
Height = call stack depth (bottom = entry point, top = leaf)
Color  = package/namespace (usually random, ignore color)

Look for:

• Wide plateau at top — hot leaf function; optimization target
• Wide bar mid-stack — caller spending time in one callee
• Unexpected library frames — JSON serialization, regex, logging in hot path
• Growing over time — memory leak (inuse_space growing across profiles)

Compare profiles before and after deploy to catch performance regressions.

Profile types and when to use each

Run CPU profiling continuously. Trigger inuse_space profiling when heap metrics trend upward.

Failure modes

• Profiling only on demand — miss intermittent spikes; profile continuously
• 100% sampling on all pods — >5% overhead; reduce to 20Hz if needed
• Profiles not linked to traces — can't connect slow request to hot function
• Flame graph misread — optimize leaf function when caller is the real issue
• No baseline profile — can't detect regression after deploy

Production checklist

• Continuous profiling deployed (Pyroscope, Parca, or Datadog Profiler)
• CPU profiling at 100Hz on all production instances
• Profiles linked to trace exemplars in Grafana
• Baseline profile captured before each major deploy
• Alert on inuse_space growth trend (>10% over 24 hours)
• Flame graph review in post-incident process for latency incidents

Profile production at 1–5% sample rate continuously — episodic profiling during incidents captures the wrong code path because traffic patterns differ under stress.

Resources

• Grafana Pyroscope documentation — setup and query guide
• Parca GitHub — open-source continuous profiling
• Brendan Gregg's flame graph guide — how to read flame graphs
• eBPF profiling fundamentals — kernel-level sampling
• async-profiler for JVM — low-overhead Java profiling