The Saga Pattern for Distributed Transactions

Engineering
Share on LinkedIn Share on X Share on Reddit Share on HN Share on Bluesky

Once a business operation touches more than one service, the database transaction you relied on for correctness disappears. You can't wrap "reserve inventory, charge the card, create the shipment" in a single BEGIN/COMMIT when those three things live in three services with three databases. The saga pattern is the standard answer: model the operation as a sequence of local transactions, and for each step define a compensating action that semantically undoes it, so that if step three fails you can walk back steps two and one. You give up atomic all-or-nothing in exchange for a system that either finishes or cleans up after itself.

I've built these for order flows and payment pipelines, and the pattern is genuinely good — but it's also where I've seen teams underestimate the work, because "compensation" is a design problem, not a library you install. Here's how sagas actually work and the decisions that determine whether yours stays maintainable.

Why the ACID transaction is gone

In a monolith with one database, consistency is nearly free — the database gives you atomicity, and a failed step rolls back everything automatically. Split the operation across services and that guarantee evaporates. The textbook fix, two-phase commit (2PC), coordinates a distributed atomic commit, but it's a poor fit for microservices: participants hold locks while awaiting the coordinator's decision, throughput craters under contention, and a coordinator crash between "prepare" and "commit" can leave participants blocked indefinitely. On top of that, most modern datastores and third-party APIs don't even support enlisting in a distributed transaction.

So the industry largely abandoned 2PC for service-to-service work and embraced eventual consistency: accept that the system passes through inconsistent intermediate states and guarantee it converges to a consistent end state. The saga is the concrete mechanism for that guarantee.

Compensations are the hard part

The core primitive is the compensating transaction. For every forward step Tn, you write a Cn that undoes its effect. Crucially, compensation is semantic, not literal — you rarely "roll back," you take a new action that corrects the world.

Consider an order saga:

Forward step Compensating action
Reserve inventory Release inventory reservation
Charge payment Refund payment
Create shipment Cancel shipment

Notice that "refund payment" is not the inverse of "charge payment" — money moved, was recorded, maybe triggered a receipt email. The compensation is a new business fact, not a rollback. This is where sagas get subtle: some actions are hard or impossible to compensate cleanly (you can't un-send a physical package that already left). The senior move is to order your steps so the hardest-to-undo action comes last, minimizing the compensation surface. Reserve-then-charge-then-ship is deliberate: shipping is nearly irreversible, so it goes last, and by then everything before it succeeded.

Every step and every compensation must also be idempotent, because in a distributed system messages get redelivered and steps get retried. If "release inventory" runs twice it must not release twice. This is the same non-negotiable I keep hammering in idempotency for distributed systems — sagas simply don't work without it.

Choreography: events, no conductor

There are two ways to coordinate the steps. In a choreography saga, there's no central controller. Each service listens for events and emits its own, and the workflow emerges from the chain of reactions:

OrderService   --OrderCreated-->        InventoryService
InventoryService --InventoryReserved-->  PaymentService
PaymentService --PaymentCharged-->       ShippingService
PaymentService --PaymentFailed-->        InventoryService (compensate)

Each service reliably publishes its events — almost always via the event-driven outbox pattern so the local transaction and the event publication are atomic. Choreography is beautifully decoupled: no service knows the whole flow, and adding a participant can be as simple as subscribing to an event.

The downside shows up as the saga grows. With six or seven services reacting to each other, no single place describes the workflow. Debugging "why didn't the shipment get created?" means tracing events across many services, and cyclic event dependencies can sneak in. Choreography is my default for short flows (2–4 steps); past that, the emergent complexity turns into a liability.

Orchestration: a central brain

In an orchestration saga, a dedicated orchestrator owns the workflow. It calls each service (or sends commands), tracks the saga's state in a durable store, and decides what to do next — including triggering compensations in reverse order on failure:

class OrderSaga:
    def run(self, order):
        try:
            self.inventory.reserve(order.id, order.items)      # T1
            self.payment.charge(order.id, order.total)         # T2
            self.shipping.create(order.id, order.address)      # T3
            self.state.mark_completed(order.id)
        except StepFailed as e:
            self.compensate(order, failed_at=e.step)           # run C(n-1)..C1

    def compensate(self, order, failed_at):
        if failed_at > 2: self.payment.refund(order.id)        # C2
        if failed_at > 1: self.inventory.release(order.id)     # C1
        self.state.mark_compensated(order.id)

The orchestrator's state must be persisted at every transition so it can resume after a crash mid-saga — the orchestrator itself has to be crash-recoverable, which is the main new burden this style introduces. In return you get a single, readable definition of the workflow, easy visibility into where any saga is stuck, and a natural home for timeouts and retries. For complex, long-running, or auditable flows, orchestration wins decisively, which is why tools like Temporal, AWS Step Functions, and Camunda exist to be that durable orchestrator.

Choosing, and living with, a saga

My rule of thumb:

And two hard truths regardless of style. First, the intermediate states are visible. During the saga, inventory is reserved but not yet paid; a user might see an order that's "processing" for seconds. Your UI and your other reads must tolerate these in-between states — that's the price of dropping ACID. Second, compensation can itself fail. What happens when the refund API is down during compensation? You need retries, alerting, and ultimately a human-review dead-letter path for sagas that can neither complete nor compensate. Pretending compensation always succeeds is the most common way I see saga implementations quietly leave money and inventory in wrong states.

Used with eyes open, the saga pattern is the right tool for cross-service consistency — it trades the impossible dream of distributed atomicity for a pragmatic, recoverable, eventually-consistent workflow. The pattern is straightforward; the discipline it demands around idempotency, compensation design, and failure-of-failure handling is what separates a robust implementation from a demo.

Operational notes for saga pattern distributed transactions

Support tooling should list saga instances by state with links to step logs. When compensation fails, operators need a single screen showing forward steps completed, compensations attempted, and external API error bodies — not five Kibana tabs. Load-test sagas with injected failures monthly: kill payment service mid-flight, verify inventory releases and no double refunds. Document semantic compensation rules in product language so engineers and PMs agree what undo means for each step.

Notes on saga pattern distributed transactions

Support tooling should list saga instances by state with links to step logs. When compensation fails, operators need a single screen showing forward steps completed, compensations attempted, and external API error bodies — not five Kibana tabs. Load-test sagas with injected failures monthly: kill payment service mid-flight, verify inventory releases and no double refunds. Document semantic compensation rules in product language so engineers and PMs agree what undo means for each step.

Resources

Saga orchestrators should expose a query API for support: current step, elapsed time, last error. Without visibility, 'order stuck processing' tickets become archaeology across five services.

Frequently asked questions

What is the saga pattern?

A sequence of local transactions with compensating actions if a later step fails.

Choreography vs orchestration?

Choreography uses events; orchestration uses a central coordinator with durable state.

Why not two-phase commit?

2PC blocks resources and most modern services do not support distributed transactions.

Hiring a senior Android / Flutter engineer?

I architect and ship production mobile software — Kotlin, Jetpack Compose, Flutter — for robotics, EV infrastructure, fintech, and real-time systems. Open to remote roles in Europe and the US.

Get in touch →