System Design: Payment System

By Michael Samuel Naeem · November 5, 2025 · 5 min read

Payment systems have zero tolerance for "mostly correct." A double charge generates angry customers and chargebacks. A lost payment means free products. A security breach with card data means regulatory fines and destroyed trust. The architecture must guarantee that every dollar is accounted for, every operation is idempotent, and raw card data never touches your servers.

Architecture overview

Client → Payment API → Payment Service → Payment Processor (Stripe/Adyen)
              ↓              ↓                    ↓
         Idempotency     Ledger Service      Card Networks (Visa/MC)
           Store              ↓
                        Reconciliation Jobs

Your payment service is a state machine wrapper around a payment processor. The ledger records every financial movement. Reconciliation jobs detect discrepancies between your records and the processor's settlement reports.

Payment state machine

Every payment follows a strict lifecycle:

created → authorized → captured → settled
                ↓           ↓
             voided     refunded (partial/full)
                ↓
             failed

class PaymentState(Enum):
    CREATED = "created"
    AUTHORIZED = "authorized"
    CAPTURED = "captured"
    VOIDED = "voided"
    FAILED = "failed"
    REFUNDED = "refunded"

async def authorize(payment_id: str, amount: int, payment_method: str, idempotency_key: str):
    existing = await idempotency_store.get(idempotency_key)
    if existing:
        return existing

    payment = await payment_store.get(payment_id)
    if payment.state != PaymentState.CREATED:
        raise InvalidStateTransition()

    result = await processor.authorize(
        amount=amount,
        payment_method=payment_method,
        idempotency_key=idempotency_key
    )

    if result.success:
        payment.state = PaymentState.AUTHORIZED
        payment.processor_auth_id = result.auth_id
    else:
        payment.state = PaymentState.FAILED
        payment.failure_reason = result.error

    await payment_store.save(payment)
    await idempotency_store.set(idempotency_key, payment)
    await ledger.record_authorization(payment)
    return payment

Valid state transitions are enforced in code — never allow captured → authorized or refunded → captured.

Idempotency implementation

async def process_payment(request: PaymentRequest) -> PaymentResult:
    cached = await redis.get(f"idempotency:{request.idempotency_key}")
    if cached:
        return PaymentResult.parse(cached)

    lock = await redis.set(
        f"idempotency:lock:{request.idempotency_key}",
        "1", nx=True, ex=86400
    )
    if not lock:
        # Another request is processing — wait and retry
        await asyncio.sleep(0.5)
        return await process_payment(request)

    try:
        result = await execute_payment(request)
        await redis.setex(
            f"idempotency:{request.idempotency_key}",
            86400, result.serialize()
        )
        return result
    finally:
        await redis.delete(f"idempotency:lock:{request.idempotency_key}")

The lock prevents concurrent duplicate processing. The cached result handles retries after completion.

Double-entry ledger

Every financial movement is recorded in a double-entry ledger:

CREATE TABLE ledger_entries (
    id UUID PRIMARY KEY,
    transaction_id UUID,
    account TEXT,        -- 'merchant:123', 'platform:fees', 'processor:settlement'
    debit_amount BIGINT, -- in cents
    credit_amount BIGINT,
    currency TEXT,
    created_at TIMESTAMP,
    metadata JSONB
);

-- Constraint: sum of debits = sum of credits per transaction

async def record_capture(payment: Payment):
    fee = calculate_fee(payment.amount)
    merchant_amount = payment.amount - fee

    entries = [
        LedgerEntry(account=f"processor:settlement", debit=payment.amount),
        LedgerEntry(account=f"merchant:{payment.merchant_id}", credit=merchant_amount),
        LedgerEntry(account="platform:fees", credit=fee),
    ]
    await ledger.append(entries, transaction_id=payment.id)

The ledger is append-only — never update or delete entries. Corrections are new entries (refunds, adjustments).

Refund handling

async def refund(payment_id: str, amount: int, reason: str, idempotency_key: str):
    payment = await payment_store.get(payment_id)

    if payment.state != PaymentState.CAPTURED:
        raise CannotRefund()

    total_refunded = await ledger.sum_refunds(payment_id)
    if total_refunded + amount > payment.amount:
        raise RefundExceedsCapture()

    result = await processor.refund(
        charge_id=payment.processor_charge_id,
        amount=amount,
        idempotency_key=idempotency_key
    )

    if result.success:
        if total_refunded + amount == payment.amount:
            payment.state = PaymentState.REFUNDED
        await ledger.record_refund(payment, amount)
        await payment_store.save(payment)

Partial refunds are common (one item returned from a multi-item order). Track cumulative refunded amount against the captured amount.

PCI compliance architecture

Never let card data touch your servers:

Client → Stripe.js (hosted fields) → Stripe API → Card Networks
                ↓
         Your server receives token (tok_abc123)
                ↓
         Payment Service uses token for charge

Your server handles tokens, not card numbers. This qualifies for SAQ A (simplest PCI self-assessment). If you must handle card data (custom checkout), use a PCI-compliant vault service and tokenize immediately — card data exists in your system for milliseconds.

Reconciliation

Daily jobs compare your ledger against processor settlement reports:

async def reconcile(date: date):
    our_records = await ledger.get_settled_transactions(date)
    processor_report = await processor.get_settlement_report(date)

    for txn in our_records:
        processor_txn = processor_report.find(txn.processor_id)
        if not processor_txn:
            await alert("Missing in processor report", txn)
        elif processor_txn.amount != txn.amount:
            await alert("Amount mismatch", txn, processor_txn)

    for processor_txn in processor_report.unmatched:
        await alert("Missing in our ledger", processor_txn)

Discrepancies trigger alerts for manual investigation. Common causes: timing differences (authorized today, settled tomorrow), processor fees calculated differently, or failed webhook delivery.

Treat production rollout as a measured change: ship with observability, validate rollback, and review metrics 24 hours after deploy — patterns that look obvious in docs fail when skipped under release pressure.

Common production mistakes

Teams get payment system wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

System design for payment system breaks at scale when hot keys, thundering herds, and cache stampedes are discovered during launch week instead of load test week.

Debugging and triage workflow

When payment system misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Stripe payment lifecycle documentation
• PCI DSS requirements overview
• Double-entry bookkeeping for engineers
• Adyen payment flow architecture
• Payment card industry data security standard (PCI DSS v4.0)