Self-Service Infrastructure
Platform team's Jira queue had 200 open "need RDS instance" tickets. Median turnaround: nine days. Developers spun up Docker Postgres locally and called it staging. Self-service infrastructure doesn't eliminate platform engineers — it eliminates ticket-driven provisioning so platform work shifts to paved roads, policy, and reliability.
The self-service stack
Developer portal (Backstage / custom UI)
│
▼
Orchestration (Terraform Cloud, Crossplane, Humanitec)
│
▼
Approved modules / compositions
│
▼
Policy gates (OPA, Sentinel, Checkov)
│
▼
Cloud API (AWS, GCP, Azure)
Every layer is auditable. Console access is read-only for debugging, not creation.
Terraform module self-service
Expose curated modules via private registry:
# Developer PR to infra-requests repo
module "checkout_db" {
source = "app.terraform.io/acme/postgres/aws"
version = "~> 2.1"
environment = "staging"
team = "payments"
instance_class = "db.t4g.medium" # constrained by module variable validation
}
Module enforces:
variable "instance_class" {
type = string
validation {
condition = contains(["db.t4g.micro", "db.t4g.small", "db.t4g.medium"], var.instance_class)
error_message = "Staging limited to t4g micro/small/medium. Request exception for larger."
}
}
Terraform Cloud workspace runs plan on PR; apply on merge. No platform engineer clicks Apply.
Crossplane for Kubernetes-native self-service
apiVersion acme.io/v1alpha1
kind: PostgresInstance
metadata:
name: checkout-staging
namespace: team-payments
spec:
size: small
environment: staging
Composite Resource Definition (XRD) maps to managed resources (RDS, security groups, secrets) via Composition. Developers apply YAML in their namespace; Crossplane reconciles.
RBAC limits who can create PostgresInstance in which namespaces.
Portal-driven flows
Backstage software template for infra:
- Developer selects "PostgreSQL database"
- Form: environment, team, size tier
- Action opens PR in gitops-infra repo OR triggers Terraform Cloud run
- PR auto-assigned to platform bot for policy check
- Merge → provision → credentials in Vault → linked in catalog
Slack notification when ready. No ticket number.
Guardrails that matter
| Risk | Guardrail |
|---|---|
| Cost explosion | Instance size enums, quota per team tag |
| Security | No public SG rules in module defaults |
| Sprawl | Naming convention enforced, TTL on ephemeral envs |
| Orphans | Mandatory team tag; weekly untagged resource report |
| Prod mistakes | Separate AWS account; prod requires approval workflow |
Policy-as-code blocks bad plans before resources exist — cheaper than cleanup.
Approval workflows
Fully automated for dev/staging. Production adds:
PR opened → terraform plan comment → policy pass
→ platform approval (CODEOWNERS) → apply
Or risk-based: standard module + staging tested → auto-apply prod; custom sizing → human review.
Audit log: who requested, what module version, plan output, approver.
Operating the IDP
Product manage the portal. If UI is confusing, developers revert to Slack asks.
SLO the platform. Provisioning success rate, time-to-ready, failed applies. Your customers are internal dev teams.
Feedback loop. Monthly office hours; track "why did you bypass self-service?" themes.
Deprecation. Old module versions get EOL dates; scorecard flags services on deprecated modules.
We cut infra ticket volume 70% in two quarters after self-service Postgres, S3 buckets, and K8s namespaces shipped — platform headcount didn't change; project work replaced toil.
Cost visibility in self-service
Show estimated monthly cost in the provisioning form before submit — pulled from Infracost or internal pricing API. Engineers choose smaller instance when they see "$240/mo" next to the dropdown.
Quota enforcement per team tag prevents one enthusiastic intern from provisioning ten GPU instances via API loop bug.
Operational notes
Provide dry-run mode in self-service portal — show Terraform plan summary without apply. Engineers learn cost and blast radius before committing. Dry-run logs feed audit trail for compliance.
Run game days where developers provision and tear down resources via self-service only — no platform Slack rescue. Game day gaps become portal UX fixes within two sprints.
Publish monthly self-service usage report to engineering leadership — provisioning volume, failure rate, top resource types — justifies platform headcount and identifies templates needing polish.
Align self-service resource TTL defaults with finance policy — ephemeral environments auto-destroy after seven days unless extended via ticket, preventing silent cost accumulation.
Self-service portals fail adoption when error messages expose raw Terraform stderr — translate failures into actionable fixes with links to docs and example PRs teams can copy.
Self-service guardrails
Developers get:
- Terraform module catalog (VPC, RDS, S3)
- Policy-as-code limits (max instance size, allowed regions)
- Auto-generated IAM roles scoped to service
Platform team reviews module PRs, not every infra PR from 40 teams.
Common production mistakes
Teams get self service infra wrong in predictable ways:
- Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
- Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
- Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
- Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.
Production implementations of self service infra fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.
Debugging and triage workflow
When self service infra misbehaves in production, work top-down instead of guessing:
- Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
- Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
- Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
- Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
- Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
- Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.
Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.
Resources
- Backstage documentation
- Crossplane concepts guide
- Terraform Cloud run triggers
- Humanitec platform orchestration
- CNCF IDP whitepaper
Frequently asked questions
What is self-service infrastructure?
Developers provision approved infrastructure — databases, queues, namespaces, S3 buckets — through a portal or GitOps workflow without filing tickets and waiting for platform engineers. Guardrails enforce security, cost, and naming standards automatically.
How do you prevent self-service from becoming shadow IT?
All paths go through audited automation with policy checks — Terraform Cloud, Crossplane, or Backstage actions backed by approved modules. Nothing is created via console click. IAM scopes limit what each team can provision; tags enforce cost attribution.
When should self-service require human approval?
Production resources with high blast radius — public ingress, cross-account access, GPU instances, PHI storage — benefit from automated plan review plus platform approval. Dev/staging environments can be fully automated with tighter quotas.
Hiring a senior Android / Flutter engineer?
I architect and ship production mobile software — Kotlin, Jetpack Compose, Flutter — for robotics, EV infrastructure, fintech, and real-time systems. Open to remote roles in Europe and the US.
Get in touch →