Service Scorecards and Maturity

By Michael Samuel Naeem · February 23, 2026 · 6 min read

Leadership asked for a "service health dashboard." Platform built a spreadsheet. Teams gamed it by marking docs "complete" with a one-line README. Six months later nobody opened the spreadsheet. Scorecards work when metrics are automatically collected, tier-appropriate, and connected to help — not when they're manual checkbox audits updated quarterly.

Scorecard vs maturity model

Scorecard: point-in-time grades per service (api-checkout: B, legacy-batch: D).

Maturity model: defined levels (1–4) per capability area with criteria for advancement.

Service: api-checkout (Tier 1 — customer-facing)
├── Reliability:    Level 3  (SLO defined, error budget policy)
├── Security:       Level 4  (SAST, dependency scan, secrets scan pass)
├── Observability:  Level 3  (tracing, dashboards, runbook linked)
├── CI/CD:          Level 4  (automated deploy, canary, rollback tested)
└── Documentation:  Level 2  (API docs yes, on-call runbook stale)

Levels map to concrete checks — not subjective "feels mature."

Tier-based requirements

Not every service needs the same bar:

Scorecard weights Tier 1 failures heavier. Failing "has README" on a Tier 3 cron matters less than missing SLO on checkout.

Automate data collection

Manual scorecards die. Wire sources:

# Scorecard check definitions (Backstage Scorecards plugin style)
checks:
  - id: has_slo
    name: SLO Defined
    type: json-rules
    successCondition: metadata.annotations['acme.com/slo-target'] != null

  - id: ci_passing
    name: CI Green
    type: github
    filter: 'state == "success" && branch == "main"'

  - id: vuln_critical_zero
    name: No Critical CVEs
    type: sonarqube
    successCondition: metrics.critical_vulnerabilities == 0

Backstage Scorecards, Cortex, or custom jobs querying Prometheus, PagerDuty, and GitHub APIs. Same data feeds service catalog and executive dashboards.

DORA metrics on scorecards

Four keys from Accelerate research:

• Deployment frequency — how often deploys hit prod
• Lead time for changes — commit to prod duration
• Change failure rate — deploys causing incidents
• MTTR — recovery time

Collect from CI/CD and incident tooling. Compare teams to themselves over time, not against Netflix on day one.

Low deployment frequency + low failure rate sometimes means fear, not quality — scorecards should flag "no deploys in 30 days" as amber for Tier 1 services.

Presenting results constructively

Team dashboards, not wall of shame. Show "3 services below Tier 1 observability threshold" with links to platform office hours.

Improvement campaigns. Quarter theme: "100% Tier 1 services have runbooks." Platform ships runbook template; scorecard tracks adoption.

Executive rollup. Percentage of Tier 1 services at Level 3+ per capability — trends over quarters.

Avoid tying scorecard grades to individual performance reviews. Gaming follows instantly.

Maturity advancement path

Define what Level N+1 requires and who helps:

Observability Level 2 → 3:
  Requires: distributed tracing on all HTTP handlers
  Platform provides: OpenTelemetry SDK bump in golden path template
  Team action: merge platform PR or adopt manually
  Deadline: none — tracked on scorecard

Platform owns making advancement easy. Product teams own prioritization against feature work — visibility creates negotiation data, not mandates.

Common failure modes

Too many checks. Start with 8–12 high-signal checks. Expand when adoption stabilizes.

Stale automated data. Broken GitHub integration shows all services failing CI — credibility gone.

Checks without owners. Each check maps to a team that fixes failures (platform for CI template, security for CVE scan).

Ignoring context. Legacy mainframe adapter won't have canary deploy — exclude or separate track.

Avoiding metric gaming

When scorecards tie to performance reviews, teams optimize metrics not systems — 100% doc coverage with empty stubs. Prefer automated technical checks over self-reported fields. Human review only for subjective maturity levels with calibration sessions across teams.

Operational notes

Integrate scorecard API with incident retros — repeated incidents on low-scoring services prioritize platform investment. Correlation beats guessing which teams need golden path upgrades.

Review scorecard weights annually — over-weighting documentation maturity starves reliability investment when teams optimize README length instead of SLO coverage.

Weight scorecard metrics by user pain — 100% test coverage matters less than deploy frequency if releases take two weeks.

Scorecard metrics that matter

DORA metrics per team, not vanity "100% doc coverage" — executives understand deployment frequency; they don't understand linter scores.

Weighting checks by service tier

Tier 1 checkout might weight: SLO 30%, security 25%, deploy automation 20%, observability 15%, docs 10%. Tier 3 batch flips docs weight near zero. Publish weight formula.

Scorecard-driven platform roadmap

When 60% of Tier 1 services fail distributed tracing enabled, platform ships OTel bump in golden path template rather than nagging 40 teams.

Historical trending

Store weekly score snapshots. Graph pct_tier1_level3_observability over quarters for exec reviews.

Integration with incident data

Join PagerDuty incident count per service with scorecard grade. Low observability plus high incidents prioritizes runbook work with incident commander endorsement.

Executive rollup without blame

Present quarterly: percent Tier-1 services at Level 3+ per capability domain, trend arrow, platform initiatives tied to lowest domain. Never rank individual engineers — rank capability gaps. "Observability lagging security" justifies OTel investment better than "Team X is D grade."

Automating doc freshness checks

Link runbook URL in catalog; weekly crawler asserts HTTP 200 and modified date within 90 days. Stale runbook fails scorecard check automatically — no self-reported checkbox.

Normalizing scores across org size

Raw DORA numbers unfairly compare 3-person team to 40-person product line. Normalize deployment frequency per engineer or per service. Scorecard shows service-level maturity, team rollup separate — exec sees portfolio trend, team sees actionable service list.

Custom checks without plugin hell

Not everything fits Backstage Scorecards plugin — custom cron job queries GitHub API for CODEOWNERS file existence, posts JSON to catalog annotation. Keep checks in Git as YAML either way; avoid spreadsheet side channel reappearing because one check was "too hard" to automate.

Linking scorecards to OKRs

Platform OKR "80% Tier-1 services with SLO" maps directly to scorecard check has_slo — executives see one number, engineers see per-service gap list. Misalignment when OKR uses manual survey while scorecard automated — pick one source of truth.

Scorecard API for CI gates

Tier-1 services below observability Level 3 block prod deploy in CI — pipeline queries catalog scorecard API, fails with link to platform office hours. Hard gate controversial but moved needle when soft nudge ignored for two quarters; start with warn then enforce after remediation sprint.

Closing notes

Scorecard checks should include runbook link HTTP 200 and on-call rotation configured in PagerDuty — operational readiness beyond green CI.

Additional guidance

Service tier metadata drives scorecard weights — checkout Tier-1 failing SLO check ranks above internal cron missing README. Automate tier assignment in catalog from domain tag or manual architect approval; stale tier defaults misallocate platform remediation effort toward low-impact services displaying red grades executives misinterpret as customer-facing risk.

Export scorecard pass rate to executive quarterly deck as trend line not snapshot — flat pass rate during headcount doubling indicates platform kept pace; declining pass rate triggers headcount or scope discussion with engineering leadership backed by data not anecdotal platform team frustration.

Treat scorecard regressions after template deprecation as platform bug — not team negligence — when eighty percent of services fail new check simultaneously.

Resources

• Backstage Scorecards plugin
• DORA metrics guide
• Google SRE — service level objectives
• Cortex service catalog platform
• Accelerate book (Forsgren, Humble, Kim)