Automating Secret Rotation

By Michael Samuel Naeem · July 2, 2025 · 6 min read

The database password rotated quarterly through a ticket queue. Someone pasted the new value into the wrong environment variable, and production connection pools rejected auth for twelve minutes while on-call grep'd deployment manifests. That incident convinced leadership that rotation should be boring infrastructure—not a calendar reminder with human copy-paste in the loop.

Automated rotation treats credentials as short-lived resources: generate, deploy, verify, revoke, repeat. The hard part is not random string generation. It is keeping traffic healthy while connection pools, CI pipelines, partner webhooks, and cron jobs still hold yesterday's key.

Why manual rotation fails at scale

Manual rotation optimizes for the happy path documented in a wiki page. Production has long-lived pods that never restarted after deploy, staging databases that share credential names with production, and third-party SaaS dashboards where only one engineer knows how to update the webhook secret. Each manual cycle adds drift between what the secret manager stores and what actually authenticates.

Compliance auditors ask for evidence: rotation interval, who triggered each event, and proof old credentials were revoked. Spreadsheet tracking does not survive acquisitions or team churn. Automated pipelines emit structured audit events with secret ARN and version IDs—never the secret value itself.

Rotation pipeline overview

Scheduler / Event ──▶ Secret Manager (version N+1)
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         App reload   CI sync    Partner notify
              │            │            │
              └────────────┼────────────┘
                           ▼
              Synthetic verify ──▶ Revoke version N

Publish secret.rotated events on a message bus so subscribers reload without polling. Database rotation Lambdas, Kubernetes operators, and custom sidecars should all subscribe to the same contract.

Dual-credential window for databases

The safest database pattern keeps two valid users during overlap:

1. Create app_v3 with identical grants to app_v2
2. Write app_v3 as the primary in Vault or Secrets Manager
3. Rolling restart applications or send SIGHUP to reload connection strings
4. Monitor pg_stat_activity (or equivalent) for connections still using app_v2
5. Drop app_v2 only after overlap TTL and zero connections

SELECT usename, count(*) AS sessions
FROM pg_stat_activity
WHERE usename IN ('app_v2', 'app_v3')
GROUP BY usename;

RDS and Cloud SQL managed rotation often create a shadow user, swap the secret pointer, then delete the old user. Understand your provider's steps before trusting the default Lambda—custom extensions and read replicas sometimes need extra grants.

Connection poolers like PgBouncer cache credentials at pool creation. After rotation, set pool_mode transactions appropriately and recycle pools or use DISCARD ALL on borrowed connections. A green health check on one pod does not prove every pooler shard picked up the new password.

API key rotation with partner lead time

Public API keys cannot flip instantly. Issue sk_live_NEW, register it in your auth middleware, deploy code that accepts both keys during overlap, then mark the old key rotating with expires_at:

def authenticate(raw_key: str) -> Principal:
    digest = hmac_sha256(raw_key)
    record = db.lookup_key_hash(digest)
    if record and record.status in ("active", "rotating"):
        if record.status == "rotating" and record.expires_at < utcnow():
            raise AuthError("key_expired")
        return record.principal
    raise AuthError("invalid_key")

Email integrators thirty days ahead for scheduled rotation. Emergency rotation compresses that window—maintain a contact list and status page template. Log which key ID authenticated each request during overlap so you know when old traffic dropped to zero.

TLS and mTLS certificates

Public TLS is largely solved by ACME clients. Monitor expiry independently—automation fails when DNS validation breaks or rate limits hit during incident-driven reissues. For internal mTLS, rotate client certificates before fifty percent of lifetime consumed; short-lived certs reduce the value of stolen material.

Store private keys in HSM or cloud KMS where policy allows. Rotation scripts that write PEM files to /tmp recreate the leakage path you are trying to eliminate.

AWS Secrets Manager rotation hooks

AWS implements rotation as a four-step Lambda contract:

# Conceptual steps inside rotation Lambda
def handler(event, context):
    step = event["Step"]
    if step == "createSecret":
        generate_new_password()
    elif step == "setSecret":
        apply_to_database(new_password)
    elif step == "testSecret":
        verify_connection(new_password)
    elif step == "finishSecret":
        mark_current_version()

testSecret must run a real query—not SELECT 1 against a read replica that still accepts the old password on a lagging node. finishSecret moves the AWSCURRENT label; premature finish during a failed canary locks you out.

Kubernetes secret mounting

Kubernetes Secrets mounted as volumes update files on disk when the Secret object changes—but applications must watch inotify or poll. Environment variable injection from Secrets does not update without pod restart. Prefer volume mounts plus reload hooks over env vars for rotatable credentials.

External Secrets Operator syncs cloud secret versions into cluster Secrets on interval. Tune sync frequency against API rate limits and blast radius requirements.

Verification gates before revocation

Never revoke the old credential because the scheduler finished. Require:

• Synthetic transaction success against production-like path
• Error rate flat versus baseline for fifteen minutes minimum
• Zero connections or requests authenticated with old version
• Explicit rollback path if canary fails—keep old version AWSPENDING until promoted

Rollback means keeping version N active and deleting N+1 from the manager, not redeploying application code. Runbooks should name the exact CLI commands.

Emergency rotation playbook

When Gitleaks fires or a laptop is stolen:

1. Identify affected secret scope—one repo token vs organization root
2. Generate and deploy new credential through automation, not manual paste
3. Revoke old credential at provider immediately after deploy starts
4. Scan audit logs for use of old credential after known compromise time
5. Notify partners if their integrations break

Panic manual rotation skips step four and leaves attackers with a longer window.

Observability and audit evidence

Log rotation events with: trigger (scheduled vs emergency), actor service account, secret identifier, old and new version IDs, duration, and verification result. Ship logs to immutable storage—CloudTrail, Vault audit devices, or SIEM with tamper detection.

Dashboards should show time-since-last-successful-rotation per secret class. Alert when rotation jobs fail twice consecutively or overlap windows exceed policy max.

Organizational habits that stick

Rotation automation succeeds when application teams own reload behavior. Platform provides the scheduler and secret store; product teams implement SIGHUP handlers and pool recycling. Quarterly game days rotate a non-production secret end-to-end with engineers who did not write the original automation.

Treat overlap TTL as a documented contract in runbooks, not tribal knowledge. The next on-call should not guess whether app_v2 was dropped early during a previous incident.

Resources

• AWS Secrets Manager rotation
• HashiCorp Vault database secrets
• Google Cloud Secret Manager rotation
• NIST SP 800-57 key management
• OWASP Secrets Management Cheat Sheet

Vault dynamic database credentials

Vault database secrets engine issues short-lived users per lease — rotation becomes issuance, not password editing. Tune lease TTL against connection pool recycle; pools outliving lease hold dead passwords until recycle.

The database password rotated quarterly—manually, via ticket, with a typo that locked out production for twelve minutes. Automated rotation turns credentials into short-lived resources: generate new, deploy, verify, revoke old, repeat on schedule. The hard part is not generating random strings but keeping zero-downtime while connection pools, CI pipelines, and partner webhooks still hold yesterday's key.

Rotation architecture

Scheduler → Secret Manager → New version N+1
                ↓
    Deploy/canary → Health checks → Revoke version N

Emit events (secret.rotated) for subscribers to reload.