Blue-Green vs Canary Deployments

By Michael Samuel Naeem · December 30, 2025 · 5 min read

A bad deployment used to mean a bad hour. We picked blue-green for our payment API because rollback had to be a load balancer flip, not a gradual retreat. For the marketing site, we picked canary because nobody wanted to pay for two full CDN origins to A/B test a hero image change. Both patterns solve "ship without downtime." They optimize for different failure budgets.

Blue-green: two stacks, one switch

Blue (current) and green (new) are complete parallel environments — separate deployments, sometimes separate databases or read replicas. Traffic sits on blue until you validate green, then you switch 100%.

                    ┌─────────┐
  Users ──────────► │   LB    │
                    └────┬────┘
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
        ┌──────────┐         ┌──────────┐
        │ Blue v1  │         │ Green v2 │  (idle or soak traffic)
        │ (active) │         │ (standby)│
        └──────────┘         └──────────┘

Kubernetes implementation without a service mesh: two Deployments, one Service, swap selector labels:

# Active service points to version label
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
    slot: blue   # flip to green after validation

Or use two Services (api-blue, api-green) and update the Ingress/backend target. AWS ALB weighted target groups achieve the same at the load balancer layer.

Rollback: flip the selector back. Sub-minute if health checks pass on the old stack. Keep blue running for at least one release cycle — deleting it immediately is how teams discover green had a slow memory leak.

Cost: 2× compute during overlap. For GPU or large JVM heaps, that hurts. Schedule green at reduced replicas during soak, scale to match blue before the switch.

Canary: progressive traffic shift

Canary keeps one baseline deployment and adds a canary replica set. The load balancer or mesh routes increasing traffic to the new version:

With Istio:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  hosts: [api.internal]
  http:
    - route:
        - destination:
            host: api
            subset: stable
          weight: 95
        - destination:
            host: api
            subset: canary
          weight: 5

Argo Rollouts and Flagger automate weight changes and metric analysis — manual VirtualService edits don't survive on-call at 3 AM.

Decision matrix

Pick blue-green when versions can't coexist (breaking API contract without versioning), when you need instant rollback for compliance, or when traffic is small.

Pick canary when you want metric-gated promotion, when infra cost matters, or when you already run a service mesh / ingress controller with weight support.

Database migrations break both patterns

Neither blue-green nor canary fixes schema changes. The expand-contract pattern is mandatory:

1. Expand: deploy migration that adds new column/table (backward compatible)
2. Deploy app that writes to both old and new
3. Contract: remove old column after backfill

Blue-green with a shared database requires both app versions to work against the same schema during the switch window. Plan migrations before picking a deployment strategy.

Common mistakes

Health checks that lie. /health returns 200 while the app can't reach Postgres. Canary traffic hits real errors. Add dependency checks or synthetic canaries.

Sticky sessions. Users bounce between versions mid-session if load balancer affinity isn't aligned with canary weights. Enable session affinity or accept mixed-version sessions.

Monitoring the wrong signals. CPU on canary pods looks fine while checkout conversion drops 2%. Wire business metrics into promotion gates.

Forgetting background workers. You canaried the API but deployed the new worker at 100%. Queue consumers process incompatible message formats. Deploy workers with the same progressive strategy or version your message schema.

Decision matrix

Use blue-green for schema-compatible releases; canary for behavior changes needing metric comparison.

Common production mistakes

Teams get blue green vs canary wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of blue green vs canary fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Debugging and triage workflow

When blue green vs canary misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Argo Rollouts canary strategy
• Kubernetes Deployment rolling update docs
• Istio traffic management
• Martin Fowler on blue-green deployment
• Expand-contract migration pattern