Fleet Device Provisioning

By Michael Samuel Naeem · July 22, 2025 · 5 min read

Provisioning 10 devices is a spreadsheet and a prayer. Provisioning 10,000 is a factory line, a provisioning service, and a failure handling strategy for the 2% that don't connect on first boot. I've seen fleets where every device shipped with the same MQTT password — fine for a demo, catastrophic at scale. Fleet provisioning is the infrastructure that gives each device a unique identity before it leaves the factory.

Provisioning stages

Factory floor          First boot              Operational
─────────────         ──────────              ────────────
Generate key pair  →  Connect with claim  →  Unique cert + config
Flash firmware        cert to provisioning    Full cloud access
Print serial label    service                 Claim cert revoked
QA test station       Receive operational
                      certificate + policy

Three distinct phases, three distinct credential sets.

Factory programming

The factory station runs automated tests:

def factory_provision(device_serial: str, device_port: str) -> ProvisioningResult:
    device = connect_uart(device_port)

    # Verify firmware version
    fw_version = device.get_firmware_version()
    assert fw_version >= MIN_FIRMWARE, f"Outdated firmware: {fw_version}"

    # Verify claim certificate is present
    claim_fp = device.get_certificate_fingerprint("claim")
    assert claim_fp in KNOWN_CLAIM_CERTS, "Unknown claim certificate"

    # Trigger provisioning
    result = device.run_provisioning(
        endpoint=PROVISIONING_ENDPOINT,
        template="production-v2",
    )
    assert result.success, f"Provisioning failed: {result.error}"

    # Verify operational certificate
    op_fp = device.get_certificate_fingerprint("operational")
    registry.register(device_serial, op_fp, claim_fp)

    # Functional test
    device.publish_test_message()
    assert device.wait_for_command(timeout=30), "Command delivery failed"

    return ProvisioningResult(serial=device_serial, cert_fingerprint=op_fp)

Every device gets a record: serial, certificate fingerprint, provisioning timestamp, test results.

Claim certificate design

The claim certificate is deliberately limited:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["iot:Connect", "iot:Publish", "iot:Receive"],
    "Resource": "*",
    "Condition": {
      "StringEquals": { "iot:Connection.Thing.ProvisioningTemplate": "production-v2" }
    }
  }]
}

It can only connect to the provisioning endpoint and call RegisterThing. Nothing else.

All devices in a batch share the same claim certificate — that's acceptable because:

• It's revoked after first use (per device)
• It can only register, not operate
• Compromise of one claim cert only affects unprovisioned devices

Bulk registration API

For devices provisioned offline (no cloud during factory):

@app.post("/api/v1/devices/bulk-register")
def bulk_register(devices: list[BulkDevice]):
    results = []
    for device in devices:
        try:
            cert = ca.sign_csr(device.csr, subject=f"CN={device.serial}")
            thing = iot.create_thing(thing_name=device.serial, thing_type=device.type)
            iot.attach_certificate(thing.thing_name, cert.certificate_arn)
            iot.attach_policy(f"policy-{device.type}", cert.certificate_arn)
            results.append({"serial": device.serial, "status": "registered"})
        except Exception as e:
            results.append({"serial": device.serial, "status": "failed", "error": str(e)})
    return results

The factory uploads a CSV of serial numbers and CSRs; the service returns signed certificates for flashing.

First-boot provisioning flow

On first power-on in the field:

def first_boot_provision(config):
    if has_operational_cert():
        return connect_operational()

    claim_cert, claim_key = load_claim_credentials()
    client = mqtt_connect(config.provisioning_endpoint, claim_cert, claim_key)

    register_request = {
        "certificateSigningRequest": generate_csr(),
        "parameters": {
            "SerialNumber": get_device_serial(),
            "DeviceType": config.device_type,
        }
    }

    response = client.publish_and_wait(
        "$aws/provisioning-templates/{}/provision/json".format(config.template),
        register_request,
        timeout=60,
    )

    save_operational_cert(response["certificatePem"], response["privateKey"])
    revoke_claim_cert(claim_cert)
    connect_operational()

Retry with exponential backoff if the provisioning endpoint is unreachable. Store a "provisioning attempted" flag to distinguish first boot from reboot.

Handling failures

Safe mode: device blinks a fault code, creates a Wi-Fi AP for technician diagnostics, and refuses to operate until provisioned.

Inventory and traceability

Maintain a device registry from factory to retirement:

serial → claim_cert_fp → operational_cert_fp → thing_arn →
         factory_date → firmware_version → customer → deployed_location → status

This enables targeted certificate rotation, firmware updates, and recall if a batch has a hardware defect.

Zero-touch provisioning flow

Device boots → claims certificate via factory cert → cloud assigns identity → receives config blob

Track provisioning state: factory, claimed, active, retired. Never reuse device IDs — retired devices stay in denylist.

Common production mistakes

Teams get device provisioning fleet wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

IoT deployments of device provisioning fleet fail in the field when firmware assumes stable Wi-Fi, OTA rollback is untested, and device certificates expire without automated renewal.

Debugging and triage workflow

When device provisioning fleet misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• AWS IoT Fleet Provisioning — just-in-time and just-in-time provisioning templates
• Azure IoT Hub Device Provisioning Service — zero-touch provisioning for Azure
• FIDO Device Onboard (FDO) — emerging standard for automatic IoT onboarding
• ESP-IDF Provisioning Guide — Espressif's Wi-Fi and BLE provisioning for ESP32