Taming Metric Cardinality

By Michael Samuel Naeem · October 9, 2025 · 6 min read

Prometheus memory usage hit 28 GB overnight. Nobody deployed new instrumentation. Investigation found http_requests_total grew from 50,000 to 4.2 million series because someone added user_id as a label. Each user now has their own time series for every endpoint, status code, and method combination. Metric cardinality—the number of unique label combinations per metric name—is the silent killer of observability stacks. One bad label turns a cheap histogram into a storage catastrophe.

How cardinality works

http_requests_total{method="GET", path="/api/users", status="200"}     → series 1
http_requests_total{method="POST", path="/api/users", status="201"}    → series 2
http_requests_total{method="GET", path="/api/users", status="200", user_id="42"}  → series 3
http_requests_total{method="GET", path="/api/users", status="200", user_id="99"}  → series 4

Each unique label value combination is a separate time series stored in memory and on disk.

Cardinality formula:

series count = |label_1 values| × |label_2 values| × ... × |label_n values|

5 methods × 50 paths × 10 status codes = 2,500 series. Add user_id with 100,000 users: 250 million series.

Labels to never use

Safe label design

# BAD — unbounded path label
REQUESTS.labels(method="GET", path=request.path, status=200).inc()
# /api/users/42, /api/users/99, /api/users/101 → new series each

# GOOD — normalized path
def normalize_path(path: str) -> str:
    return re.sub(r"/\d+", "/:id", path)

REQUESTS.labels(method="GET", path=normalize_path(request.path), status=200).inc()
# /api/users/:id for all user IDs → one series

Standard labels that work:

service, environment, region, method, route (normalized), status_class (2xx, 4xx, 5xx)

Detecting cardinality problems

# Top 20 metrics by series count
topk(20, count by (__name__) ({__name__=~".+"}))

# Series count for a specific metric
count({__name__="http_requests_total"})

# Label values contributing most to cardinality
topk(10, count by (path) (http_requests_total))

Set alerts:

- alert: HighCardinalityMetric
  expr: count by (__name__) ({__name__=~".+"}) > 100000
  for: 15m
  annotations:
    summary: "Metric {{ $labels.__name__ }} has {{ $value }} series"

Fixing existing explosions

1. Relabel at scrape time — drop bad labels before storage:

metric_relabel_configs:
  - source_labels: [user_id]
    regex: .+
    action: labeldrop

2. Recording rules — aggregate away high-cardinality labels:

groups:
  - name: aggregations
    rules:
      - record: http_requests:rate5m
        expr: sum by (method, route, status_class) (rate(http_requests_total[5m]))

Query the aggregated metric in dashboards. Keep the raw metric with short retention (1 day) for debugging.

3. Fix the instrumentation — remove the label at the source. This is the correct long-term fix.

Cardinality budgets

Assign per-team budgets:

New metrics require review if they add labels beyond the standard set.

High-cardinality alternatives

When you need per-user or per-request detail, use the right tool:

Metrics aggregate. Logs and traces disaggregate. Using metrics for per-entity data is an architecture mistake.

Grafana Mimir limits

limits:
  max_label_names_per_series: 30
  max_label_value_length: 2048
  cardinality_limit: 100000  # per metric name per tenant
  max_global_series_per_user: 5000000

Configure ingestion limits to reject high-cardinality metrics at the source rather than crashing the store.

Detecting cardinality explosions early

Set alerts before the TSDB falls over:

# Top metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

# Rate of new series (cardinality growth)
sum(rate(prometheus_tsdb_head_series_created_total[5m]))

Alert when:

• Any single metric exceeds 100K series
• Total series growth > 10% day-over-day
• Ingestion rejected samples spike (Mimir cortex_discarded_samples_total)

Run weekly cardinality reports — assign owners to metrics in the top 20.

Label naming conventions

Standardize labels across services to enable cross-team dashboards:

Use OpenTelemetry semantic conventions as the baseline — mixing http.method and method doubles cardinality for the same concept.

Cost impact

Managed observability bills by ingested samples or series. One bad deploy adding user_id label to a counter can 100× your bill overnight:

Before: http_requests_total{route="/api/orders"} → 50 series
After:  http_requests_total{route="/api/orders", user_id="..."} → 2M series

Add cardinality review to service onboarding checklist. Platform team provides approved instrumentation library with safe defaults.

Pair with SLIs, SLOs, and error budgets — high-cardinality metrics don't improve SLO measurement; they threaten the metrics store that SLOs depend on.

Production checklist

• Weekly cardinality report with metric owners assigned
• user_id and request_id banned from metric labels
• Ingestion limits configured in Mimir/Prometheus
• OpenTelemetry semantic conventions enforced in SDK
• Traces used for per-request detail, not high-cardinality metrics

Common production mistakes

Teams get metrics cardinality wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Never export per-user or per-prompt labels to Prometheus — push billing aggregates to the warehouse instead.

LLM-specific cardinality traps

Model routers tempt teams to label histograms with prompt_version, tenant_id, and tool_name on every span metric. Keep metrics bounded: model, route, status_class, region. Store prompt versions in logs/traces with sampling, not in time series that multiply with every deploy.

Recording rules and aggregation

Use Prometheus recording rules or Mimir aggregations to pre-compute high-traffic queries without high-cardinality labels. Example: sum rate by service, route, status — never by user_id. Review recording rule output cardinality in staging before promoting; a bad rule can explode series count faster than raw scrapes.

Otel Collector cardinality limiter

processors:
  cardinality/drop:
    metric_names:
      match_type: regexp
      regexp: ".*"
    label_limits:
      label_name: user_id
      max_label_values: 0  # drop label

Reject at ingest before TSDB crash—pair with CI lint on application metric definitions.

Ownership model

Each metric __name__ in top 50 series count has named owner in catalog. Weekly Slack bot posts cardinality leaderboard—shame works better than surprise 28GB RAM Prometheus.

Downsampling vs dropping labels

Cortex/Mimir downsampling reduces long-term retention cost but does not fix cardinality explosion—still need label hygiene at source.

PromQL subquery cardinality explosions

group by (user_id) (...) in ad-hoc queries does not create series—but recording rules accidentally grouping by high-cardinality label do. Review recording rule PRs for forbidden labels same as instrumentation PRs.

Federation and remote write duplicates

Dual remote write to two vendors accidentally with different label rewrite rules duplicates effective cardinality cost—audit remote_write configs quarterly.

Resources

• Prometheus metric and label naming — official best practices
• Grafana Mimir cardinality management — production guidance
• Prometheus relabeling — drop labels at scrape time
• Robust Perception cardinality blog — cardinality benchmarks
• OpenTelemetry attribute naming — semantic conventions for labels