Agent reliability via runbook as code

By Michael Samuel Naeem · March 22, 2026 · 7 min read

Agent reliability via runbook as code means you ship agent runbook as code with human override paths — with a named owner, a measurable signal, and a rollback a tired on-call can run. I reach for this when cost or error budgets are burning too fast; that is also when shortcuts like treating agent runbook as code as a pure library problem start paging people.

This write-up is specific to agent-runbook-as-code in a agent context, using Redis, Temporal, OpenTelemetry for the mechanics while keeping ownership human.

Decision guide for Agent reliability via runbook as code

Agent loops amplify mistakes: one bad tool call can fan out across systems. For agent runbook as code, that means making failure visible early.

With Redis, Temporal, OpenTelemetry, the mechanics are straightforward; the hard part is invariants. The anti-pattern I still see is treating agent runbook as code as a pure library problem.

Acceptance check: an on-call engineer can explain system state for agent runbook as code from one dashboard and one runbook page.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

When to refuse this approach

I treat Agent reliability via runbook as code as an operations problem first. The goal is to ship agent runbook as code with human override paths, not to collect frameworks.

Keep side effects at the edges and make every write idempotent. Agent reliability via runbook as code without retry semantics is a future incident write-up.

Acceptance check: an on-call engineer can explain system state for agent runbook as code from one dashboard and one runbook page.

Concretely, being able to ship agent runbook as code with human override paths forces explicit choices: source of truth, timeout budgets, and which errors users see versus operators.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

// Agent reliability via runbook as code
export async function handle_agent_runbook_as_code(input: unknown): Promise<Result> {
  const parsed = schema.safeParse(input);
  if (!parsed.success) throw new ValidationError(parsed.error);
  const span = tracer.startSpan("agent-runbook-as-code");
  try {
    if (await repo.seen(parsed.data.idempotencyKey)) return { ok: true, deduped: true };
    const out = await repo.execute(parsed.data);
    await repo.mark(parsed.data.idempotencyKey);
    return out;
  } finally {
    span.end();
  }
}

Minimal production setup

Teams usually discover Agent reliability via runbook as code after a quiet failure — wrong data, slow pages, or a bill spike. Design for cost or error budgets are burning too fast.

Put a metric on the user-visible effect of agent runbook as code before you optimize internals. If cost or error budgets are burning too fast, you need that graph on day one.

Acceptance check: an on-call engineer can explain system state for agent runbook as code from one dashboard and one runbook page.

My never-again list for agent runbook as code: treating agent runbook as code as a pure library problem; shipping without a kill switch; and alerting only on infrastructure CPU.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

Cost, complexity, and ownership

Agent loops amplify mistakes: one bad tool call can fan out across systems. For agent runbook as code, that means making failure visible early.

Keep side effects at the edges and make every write idempotent. Agent reliability via runbook as code without retry semantics is a future incident write-up.

Ship behind a flag, canary by cohort, and write the rollback in the PR description. Agent reliability via runbook as code that needs a hero is not done.

Review prompts I use: what happens twice, what happens never, what happens partially? If Agent reliability via runbook as code cannot answer, it is not production-ready.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

Migration without dual-running forever

Agent loops amplify mistakes: one bad tool call can fan out across systems. For agent runbook as code, that means making failure visible early.

Keep side effects at the edges and make every write idempotent. Agent reliability via runbook as code without retry semantics is a future incident write-up.

Document what 'success' and 'undo' mean in product language. Future reviewers will not share your context on agent runbook as code.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

Related reading:

• webhooks reliable delivery
• idempotency distributed systems
• event driven outbox pattern

Definition of done

Teams usually discover Agent reliability via runbook as code after a quiet failure — wrong data, slow pages, or a bill spike. Design for cost or error budgets are burning too fast.

With Redis, Temporal, OpenTelemetry, the mechanics are straightforward; the hard part is invariants. The anti-pattern I still see is treating agent runbook as code as a pure library problem.

Ship behind a flag, canary by cohort, and write the rollback in the PR description. Agent reliability via runbook as code that needs a hero is not done.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

Practical defaults for Agent reliability via runbook as code

Teams usually discover Agent reliability via runbook as code after a quiet failure — wrong data, slow pages, or a bill spike. Design for cost or error budgets are burning too fast.

With Redis, Temporal, OpenTelemetry, the mechanics are straightforward; the hard part is invariants. The anti-pattern I still see is treating agent runbook as code as a pure library problem.

Document what 'success' and 'undo' mean in product language. Future reviewers will not share your context on agent runbook as code.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

In review, require a short failure note covering retry, partial deploy, and treating agent runbook as code as a pure library problem. Missing that note blocks merge.

Review questions before merging agent runbook as code work

I treat Agent reliability via runbook as code as an operations problem first. The goal is to ship agent runbook as code with human override paths, not to collect frameworks.

With Redis, Temporal, OpenTelemetry, the mechanics are straightforward; the hard part is invariants. The anti-pattern I still see is treating agent runbook as code as a pure library problem.

Acceptance check: an on-call engineer can explain system state for agent runbook as code from one dashboard and one runbook page.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

In review, require a short failure note covering retry, partial deploy, and treating agent runbook as code as a pure library problem. Missing that note blocks merge.

Field notes after thirty days of agent runbook as code

Teams usually discover Agent reliability via runbook as code after a quiet failure — wrong data, slow pages, or a bill spike. Design for cost or error budgets are burning too fast.

With Redis, Temporal, OpenTelemetry, the mechanics are straightforward; the hard part is invariants. The anti-pattern I still see is treating agent runbook as code as a pure library problem.

Acceptance check: an on-call engineer can explain system state for agent runbook as code from one dashboard and one runbook page.

Slug-specific note (agent-runbook-as-code): prioritize code behavior under load and verify with a fixture named agent-runbook-as-code-smoke.

After a month, delete unused flags and dual paths. agent-runbook-as-code accumulates temporary bridges faster than teams expect.

Resources

• Internal runbook seed: agent-runbook-as-code
• https://12factor.net/
• https://martinfowler.com/