Secrets Management with Vault

By Michael Samuel Naeem · June 28, 2025 · 6 min read

SecurityVaultDevOpsSecrets

Forty microservices each mount a Kubernetes Secret copied from last year's onboarding doc. Rotation means editing YAML in forty repos and hoping nobody cached the old DB password in a crash dump. HashiCorp Vault stores secrets centrally, audits every read, and can mint database credentials that expire in an hour. The operational cost is real—unseal, HA, backups—but static secrets in git is technical debt with compound interest.

Core concepts

Secrets engine: KV (static), database (dynamic), PKI, AWS IAM
Policy: path-based ACL (secret/data/prod/api read)
Auth method: Kubernetes, AppRole, OIDC, AWS IAM
Lease: TTL after which secret revokes automatically

Static secrets in KV v2

vault kv put secret/prod/payment stripe_key=$STRIPE_KEY
vault kv get -field=stripe_key secret/prod/payment

Versioned; rollback supported. Still static—rotate via CI pushing new version and signaling apps to reload.

Dynamic PostgreSQL credentials

Configure database secrets engine with connection URL and creation SQL:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON orders TO "{{name}}";

Application:

 creds = vault.read("database/creds/orders-readonly")
 conn = psycopg2.connect(user=creds["username"], password=creds["password"], ...)

Renew lease before TTL; Vault revokes role on expiry.

Kubernetes auth

# Pod annotation for Vault Agent Injector
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "api-service"
vault.hashicorp.com/agent-inject-secret-config: "secret/data/prod/api"

Sidecar renders file or env vars; app unchanged. Map ServiceAccount → Vault role with minimal policy.

Policy least privilege

path "secret/data/prod/api" {
  capabilities = ["read"]
}
path "database/creds/orders-readonly" {
  capabilities = ["read"]
}

Separate policies per service. Deny by default.

Unseal and HA

Production uses auto-unseal via cloud KMS. Shamir shards for break-glass only. Monitor seal status; sealed Vault denies all reads—page immediately.

Enable audit device to immutable storage. Every secret read ties to identity—compliance and incident forensics.

Unseal monitoring pages immediately—sealed Vault denies all reads. Auto-unseal via KMS for production; Shamir shards break-glass only.

Policy least privilege per service account. Deny by default. Audit device to immutable storage for every secret read during incidents.

Kubernetes auth maps service account JWT to role—no root token in cluster. Vault Agent Injector sidecar pattern keeps apps unchanged while credentials rotate.

Dynamic database credentials

Vault database secrets engine issues two-hour PostgreSQL users per application role. Application connects with short-lived creds; DBA revokes lease on compromise. Connection pool must handle credential rotation mid-process—refresh on auth error with backoff.

Namespace isolation per environment

prod/, staging/, dev/ Vault namespaces or path prefixes prevent staging CI from reading prod secrets. Policy as code in Terraform reviews path grants in PR.

Break-glass procedures

Emergency root token sealed in physical safe with quarterly drill. Automated alerts on root token usage. Normal operations use OIDC auth to Vault with group mappings from IdP.

Secrets Management Vault: operational depth

Centralized secrets management fails when applications still read Kubernetes secrets as plaintext env vars at rest. Teams that skip instrumentation ship blind—baseline p75 latency and error rate on affected routes one week before change and compare seven days after.

Integration boundaries deserve contract tests with golden fixtures sampled from production traffic anonymized. Synthetic empty payloads pass CI while production fails on nullable fields you never modeled.

Security review asks three questions: what untrusted input enters, what secrets could leak in logs, and what happens when upstream is slow or malicious. Answers belong in the PR, not a post-launch wiki page.

Rollout prefers feature flags or canary deploys when behavior touches authentication, payments, or PII. Rollback command documented in runbook header—not discovered during incident via git archaeology.

On-call dashboards slice metrics by region and device class. Global averages hide mobile regressions until App Store reviews mention slowness—field data honesty beats demo Lighthouse scores.

Resources

Extended guidance (1) for Secrets Management Vault

Operators owning secrets management vault should run a pre-mortem before launch: dependency unavailable, duplicate events, certificate expiry, regional failover. Each scenario needs detectable metrics, a runbook step, and a tested rollback. Game days beat postmortems for building muscle memory.

Contract tests at boundaries use anonymized production samples—nullable fields and unicode edge cases break synthetic fixtures. Security review documents untrusted inputs and log redaction rules in the PR description so auditors and on-call engineers inherit context without archaeology.