Getting Resource Limits Right

By Michael Samuel Naeem · March 13, 2026 · 5 min read

Java service OOMKilled every Tuesday. Limits were 512Mi; heap was -Xmx768m. The container runtime did not care about JVM flags—it enforced cgroup memory. Requests tell the scheduler where to place pods; limits cap runtime usage. Getting both wrong wastes money or kills pods.

Kubernetes resource management separates scheduling (requests) from enforcement (limits). Understand QoS classes and what happens at the boundary.

Requests vs limits

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: "2"
    memory: 1Gi

• CPU request — guaranteed share; scheduler uses sum of requests per node
• CPU limit — CFS quota throttles usage beyond limit (Linux)
• Memory request — scheduling minimum
• Memory limit — hard cap; exceed → OOMKill

CPU is compressible (throttled). Memory is not (killed).

QoS classes

Production APIs should aim for Guaranteed on memory at minimum:

resources:
  requests:
    memory: 1Gi
    cpu: 500m
  limits:
    memory: 1Gi
    cpu: 500m

LimitRange defaults

Prevent BestEffort pods in namespace:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: checkout
spec:
  limits:
    - default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 200m
        memory: 256Mi
      max:
        cpu: "4"
        memory: 8Gi
      min:
        cpu: 50m
        memory: 64Mi
      type: Container

New containers without resources inherit defaults.

ResourceQuota caps

apiVersion: v1
kind: ResourceQuota
metadata:
  name: checkout-quota
  namespace: checkout
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"

Stops one team from consuming cluster.

Sizing workflow

1. Deploy with VPA Off or metrics-server historical usage
2. Set requests to p95 usage over 7 days
3. Set memory limit to request + headroom (20–30%) or equal for Guaranteed
4. Set CPU limit ≥ request if burst needed; equal for no burst
5. Load test and watch container_cpu_cfs_throttled_seconds_total

For JVM:

container memory limit ≥ heap + metaspace + threads + native + 25% buffer

Use -XX:MaxRAMPercentage with container-aware JVM.

Debugging throttling and OOM

kubectl describe pod checkout-api-xyz | grep -A5 "Last State"
kubectl get events -n checkout --field-selector reason=OOMKilling

Prometheus:

rate(container_cpu_cfs_throttled_seconds_total{pod="checkout-api-xyz"}[5m])

High throttle → raise CPU limit or reduce work per thread.

Vertical Pod Autoscaler hints

VPA recommendation object:

kubectl get vpa checkout-api-vpa -o yaml

Apply recommendations iteratively; do not copy blindly for Java without heap review.

Common mistakes

Right-size per environment—staging requests can be half production.

Node pressure and eviction behavior

When nodes fill, the kubelet evicts BestEffort pods first, then Burstable pods exceeding requests. Guaranteed pods survive until the node truly runs out of memory at the OS level—a misconfigured Guaranteed pod with a memory leak can take a node down. Monitor node-level memory.available and node_pressure conditions alongside per-pod metrics.

Cluster Autoscaler and Karpenter use pod requests, not limits, when deciding whether new nodes are needed. Under-requested Deployments cause scheduling hotspots; over-requested ones block bin-packing and waste capacity. Review quarterly: compare VPA recommendations, actual usage, and spend per namespace.

CPU limits and latency-sensitive services

CPU throttling manifests as tail latency spikes without high CPU metrics—threads wait for CFS quota refresh. For latency-critical APIs, either set limit equal to request (no throttling ceiling) or omit CPU limit entirely while keeping requests for scheduling (controversial but common on GKE guidance for predictable latency). Document the choice in your platform runbook; mixed clusters need consistency per tier.

What to measure after rollout

Track error rates, tail latency, and resource utilization for two weeks after changes land—most regressions appear under real traffic mixes, not in staging smoke tests. Keep a rollback path documented: feature flags, Helm revision, or Git revert with known good digest. Review on-call pages tied to the topic quarterly; delete alerts that never fire and add thresholds that would have caught your last incident.

Run a short blameless postmortem if production surprised you, even for minor issues. The goal is updating this runbook section with one concrete lesson per quarter so the next engineer inherits context, not just configuration snippets.

Documentation your team should maintain

Maintain a one-page runbook link from your main service README: prerequisites, owner rotation, last drill date, and known sharp edges. Link to vendor docs in the Resources section below but capture org-specific decisions (CIDR ranges, cluster names, approval gates) in internal docs that stay current. New hires should deploy a safe canary within a week using only that runbook—if they cannot, the doc is incomplete.

Pre-production checklist

Before promoting to production, walk through this list with someone who was not the primary author—fresh eyes catch assumptions.

• Staging parity: The staging environment exercises the same code paths as production, including failure modes you expect to handle (timeouts, retries, partial outages).
• Observability: Dashboards and alerts exist for the metrics and log patterns discussed above; on-call knows where to look first.
• Rollback: You can revert to the previous known-good state in one documented step without improvising.
• Access control: Only the principals that need access have it; audit logs are enabled where the topic touches secrets or infrastructure APIs.
• Load test: You have evidence—not intuition—about behavior at expected peak plus headroom.

If any item is "we will do that later," treat it as a release blocker for tier-1 services.

Resources

• Managing resources for containers — official semantics
• QoS classes documentation — eviction behavior
• LimitRange API — namespace defaults
• VPA recommendations FAQ — automated sizing input