Deadlines and Cancellation in gRPC

By Michael Samuel Naeem · June 19, 2025 · 5 min read

A payment service once held database locks for 30 seconds because an upstream gRPC call timed out at 2 seconds but never told the downstream service to stop. The client got DEADLINE_EXCEEDED, retried, and hit the same locked rows. Three retries later, we had a minor incident over missing deadlines — not missing code, missing propagated deadlines. gRPC gives you the mechanism. You still have to wire it through every call.

How deadlines propagate

Client                    Service A                  Service B
  |                          |                          |
  |-- GetOrder (deadline: T) |                          |
  |                          |-- GetInventory (deadline: T - elapsed)
  |                          |                          |-- Query DB
  |                          |                          |
  |  (T expires)             |  (ctx cancelled)           |  (ctx cancelled)
  |<-- DEADLINE_EXCEEDED     |                          |

The client sets a deadline. Service A receives it in the context, subtracts elapsed time, and forwards the remaining budget to Service B. When time runs out, all three layers cancel.

Setting deadlines on the client

Go:

ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()

resp, err := client.GetOrder(ctx, &pb.GetOrderRequest{Id: orderId})
if err != nil {
    if status.Code(err) == codes.DeadlineExceeded {
        // timed out — don't retry blindly
    }
}

Java:

Order response = stub
    .withDeadlineAfter(500, TimeUnit.MILLISECONDS)
    .getOrder(GetOrderRequest.newBuilder().setId(orderId).build());

Kotlin:

val response = stub
    .withDeadlineAfter(500, TimeUnit.MILLISECONDS)
    .getOrder(getOrderRequest { id = orderId })

Always set a deadline. An unset deadline means "wait forever," and forever becomes your p99.

Server-side: respect the context

func (s *Server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.Order, error) {
    if deadline, ok := ctx.Deadline(); ok {
        remaining := time.Until(deadline)
        if remaining < 50*time.Millisecond {
            return nil, status.Error(codes.DeadlineExceeded, "insufficient time remaining")
        }
    }

    select {
    case <-ctx.Done():
        return nil, status.FromContextError(ctx.Err()).Err()
    default:
    }

    order, err := s.repo.FindByID(ctx, req.Id)
    // ...
}

Check ctx.Done() in long loops:

for _, item := range items {
    select {
    case <-ctx.Done():
        return nil, status.FromContextError(ctx.Err()).Err()
    default:
    }
    process(item)
}

Propagating to downstream calls

The most common mistake: creating a fresh context for downstream calls.

// Wrong — loses parent deadline
resp, err := inventoryClient.GetStock(context.Background(), req)

// Right — propagates remaining deadline
resp, err := inventoryClient.GetStock(ctx, req)

Every gRPC call in a chain must use the incoming context. In a service mesh (Istio, Linkerd), the deadline is also forwarded in headers automatically if you use the gRPC context.

Timeout budgets

Don't give every downstream call the full parent budget:

func (s *Server) Checkout(ctx context.Context, req *pb.CheckoutRequest) (*pb.CheckoutResponse, error) {
    // Reserve 100ms for local work; give inventory 60% of remaining, payment 30%
    deadline, _ := ctx.Deadline()
    total := time.Until(deadline)

    invCtx, invCancel := context.WithDeadline(ctx, time.Now().Add(total * 60 / 100))
    defer invCancel()
    stock, err := s.inventory.GetStock(invCtx, req)

    payCtx, payCancel := context.WithDeadline(ctx, time.Now().Add(total * 30 / 100))
    defer payCancel()
    payment, err := s.payment.Charge(payCtx, req)
    // ...
}

If inventory eats 60% and fails, you still have budget left to return a meaningful error instead of a generic timeout.

Cancellation vs deadline exceeded

Handle them differently. DEADLINE_EXCEEDED on a read → maybe safe to retry. CANCELLED → the caller gave up; don't retry unless idempotent and you're sure the original didn't succeed.

Idempotency and retries

Retries + missing idempotency + missed cancellation = duplicate charges. If you retry after DEADLINE_EXCEEDED:

1. Check whether the original call actually succeeded (idempotency key lookup)
2. Use shorter deadlines on retries, not the same budget
3. Propagate an idempotency key in metadata so the server can deduplicate

md := metadata.Pairs("idempotency-key", key)
ctx = metadata.NewOutgoingContext(ctx, md)

Observability

Log remaining deadline on entry:

if d, ok := ctx.Deadline(); ok {
    log.Info("rpc started", "remaining", time.Until(d))
}

In traces (OpenTelemetry), gRPC records deadline in span attributes. Alert when services consistently consume >80% of the parent budget — that's a latency problem waiting to happen.

Propagate deadlines

remaining = context.time_remaining()
stub.Call(request, timeout=remaining)

Client sets 5s deadline — every downstream hop gets shrinking budget. Server respects cancellation — stop work when context cancelled.

Common production mistakes

Teams get deadlines cancellation wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of deadlines cancellation fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Debugging and triage workflow

When deadlines cancellation misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• gRPC Deadlines documentation — official guide to deadline propagation
• gRPC Go context package — client deadline APIs
• Google SRE — Handling overload — timeout budgets and cascading failure
• gRPC Status Codes — DEADLINE_EXCEEDED vs CANCELLED semantics