Automating Certificates with ACME

By Michael Samuel Naeem · June 16, 2025 · 5 min read

Someone pasted a calendar reminder: "Renew SSL cert – March." In 2025 that process is a outage waiting to happen. ACME (Automatic Certificate Management Environment) lets agents prove domain control and download certificates without human ticket queues. Let's Encrypt issues free 90-day certs; automation renews at day 60. When renewal fails, monitoring should page—not the first customer seeing NET::ERR_CERT_DATE_INVALID.

ACME flow overview

1. Agent generates key pair and CSR
2. Order created with ACME CA (Let's Encrypt)
3. Complete challenge (HTTP-01 or DNS-01)
4. Download certificate chain
5. Install on load balancer or ingress
6. Repeat before expiry

# Caddy obtains and renews automatically
example.com {
    reverse_proxy app:8080
}

Caddy embeds ACME client—zero config for simple cases.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

cert-manager on Kubernetes

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
spec:
  secretName: api-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - api.example.com

ClusterIssuer with DNS-01 via Route53 or Cloudflare API. Ingress annotation cert-manager.io/cluster-issuer: letsencrypt-prod triggers issuance.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

HTTP-01 behind load balancers

Ensure /.well-known/acme-challenge/ routes to the solver pod, not cached CDN. Temporarily bypass cache for challenge path or use DNS-01 if CDN cannot origin-pull challenges.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Wildcards require DNS-01

*.example.com + example.com

Create _acme-challenge.example.com TXT via provider API. Scope IAM to TXT record on challenge subdomain only.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Monitoring expiry

Prometheus ssl_exporter or cloud load balancer metrics. Alert:

days_until_expiry < 14

Include cert name and SAN list in alert body. Run weekly synthetic TLS handshake from external vantage.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Revocation and key compromise

ACME supports revocation if private key leaks. Document runbook: revoke cert, reissue with new key, invalidate old secret in Kubernetes. CAA DNS records restrict which CAs may issue for your domain—add before automation.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Use https://acme-staging-v02.api.letsencrypt.org/directory until challenges succeed reliably, then switch issuer to production to avoid rate limit bans during typos.

Staging ACME endpoint first—production rate limits punish typos during setup. Monitor days_until_expiry with alert under 14 days.

CAA DNS records restrict which CAs may issue for your domain before automation goes live. Revocation runbook for key compromise: revoke, reissue, invalidate old secret in all load balancers.

HTTP-01 requires port 80 reachable; CDN must forward acme-challenge path or use DNS-01 for wildcard certs.

Validate this in staging with production-like data volume before declaring done. Capture metrics baseline the week before change and compare for seven days after—subtle regressions hide in aggregates until a large tenant hits the path. Update the on-call runbook with the failure signature and rollback command so responders need not rediscover steps during an incident.

Document the decision, owner, and rollback path in your team wiki the same week you ship. Future you will not remember which environment variable toggled the behavior unless it is written next to the runbook entry and linked from the alert. That habit costs ten minutes per change and saves hours when pagination or auth misbehaves under a single large tenant.

Run the change through your standard PR checklist: tests, observability, and a two-minute rollback drill in staging. Small operational habits accumulate into systems that survive on-call nights without heroics.

Share a short write-up in your engineering channel after rollout: what shipped, what metric you watch, and who owns follow-up. That closes the loop for teammates who were not in the PR and surfaces gaps in docs before the next person repeats the same investigation.

Resources

• RFC 8555: ACME Protocol
• Let's Encrypt documentation
• cert-manager documentation
• Caddy automatic HTTPS
• Mozilla SSL Configuration Generator