Secrets Management with Vault

By Michael Samuel Naeem · June 28, 2025 · 5 min read

Forty microservices each mount a Kubernetes Secret copied from last year's onboarding doc. Rotation means editing YAML in forty repos and hoping nobody cached the old DB password in a crash dump. HashiCorp Vault stores secrets centrally, audits every read, and can mint database credentials that expire in an hour. The operational cost is real—unseal, HA, backups—but static secrets in git is technical debt with compound interest.

Core concepts

• Secrets engine: KV (static), database (dynamic), PKI, AWS IAM
• Policy: path-based ACL (secret/data/prod/api read)
• Auth method: Kubernetes, AppRole, OIDC, AWS IAM
• Lease: TTL after which secret revokes automatically

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Static secrets in KV v2

vault kv put secret/prod/payment stripe_key=$STRIPE_KEY
vault kv get -field=stripe_key secret/prod/payment

Versioned; rollback supported. Still static—rotate via CI pushing new version and signaling apps to reload.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Dynamic PostgreSQL credentials

Configure database secrets engine with connection URL and creation SQL:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON orders TO "{{name}}";

Application:

 creds = vault.read("database/creds/orders-readonly")
 conn = psycopg2.connect(user=creds["username"], password=creds["password"], ...)

Renew lease before TTL; Vault revokes role on expiry.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Kubernetes auth

# Pod annotation for Vault Agent Injector
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "api-service"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/prod/api"

Sidecar renders file or env vars; app unchanged. Map ServiceAccount → Vault role with minimal policy.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Policy least privilege

path "secret/data/prod/api" {
  capabilities = ["read"]
}
path "database/creds/orders-readonly" {
  capabilities = ["read"]
}

Separate policies per service. Deny by default.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Unseal and HA

Production uses auto-unseal via cloud KMS. Shamir shards for break-glass only. Monitor seal status; sealed Vault denies all reads—page immediately.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Enable audit device to immutable storage. Every secret read ties to identity—compliance and incident forensics.

Unseal monitoring pages immediately—sealed Vault denies all reads. Auto-unseal via KMS for production; Shamir shards break-glass only.

Policy least privilege per service account. Deny by default. Audit device to immutable storage for every secret read during incidents.

Kubernetes auth maps service account JWT to role—no root token in cluster. Vault Agent Injector sidecar pattern keeps apps unchanged while credentials rotate.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Document the decision, owner, and rollback path in your team wiki the same week you ship. Future you will not remember which environment variable toggled the behavior unless it is written next to the runbook entry and linked from the alert. That habit costs ten minutes per change and saves hours when pagination or auth misbehaves under a single large tenant.

Run the change through your standard PR checklist: tests, observability, and a two-minute rollback drill in staging. Small operational habits accumulate into systems that survive on-call nights without heroics.

Share a short write-up in your engineering channel after rollout: what shipped, what metric you watch, and who owns follow-up. That closes the loop for teammates who were not in the PR and surfaces gaps in docs before the next person repeats the same investigation.

Prefer boring, repeatable process over one heroic migration weekend.

Resources

• HashiCorp Vault documentation
• Vault Kubernetes auth method
• Vault database secrets engine
• Vault Agent Injector
• OWASP Secrets Management Cheat Sheet