MQTT Bridging and Clustering

By Michael Samuel Naeem · July 21, 2025 · 6 min read

The factory MQTT broker hit 12,000 connected PLCs before CPU pegged at 100%. Adding a second Mosquitto instance didn't help — clients were still pinned to one node with no shared state. We needed bridging to get telemetry into AWS IoT Core without exposing the plant network, and clustering so no single VM became the bottleneck. MQTT scaling is two different problems with overlapping vocabulary; picking the wrong pattern gives you duplicate messages, lost sessions, or a bridge loop that took down staging twice.

Single broker limits

One broker node handles surprising load — tens of thousands of lightweight clients publishing every 30 seconds. Limits appear in:

• Connection count — file descriptors, memory per session
• Publish throughput — single-threaded routing in simpler brokers
• Persistent sessions — RAM/disk for offline QoS 1/2 queues
• Topic fan-out — 1 publish × 5,000 subscribers = 5,000 deliveries

Monitor messages.publish.received.rate, heap usage, and socket counts before scaling.

MQTT bridging

A bridge is an MQTT client connection between brokers. Broker A forwards matching topics to Broker B (and optionally reverse).

Mosquitto bridge config (site → cloud):

connection bridge-to-cloud
address iot.example.com:8883
bridge_cafile /etc/mosquitto/certs/ca.crt
bridge_insecure false
remote_username bridge-user
remote_password ${BRIDGE_PASSWORD}

topic factory/+/telemetry out 1
topic factory/+/commands in 1
topic $SYS/# out 0

cleansession false
try_private false
notifications_local true

Key directives:

• topic pattern direction qos — out publishes from local to remote; in subscribes remote to local
• cleansession false — bridge reconnects without losing inflight state
• Avoid # bridges — loop risk and bandwidth explosion; be explicit

Bridge topologies

  [Plant Mosquitto] ──bridge──► [Cloud EMQX] ◄── [Mobile apps]
         ▲                              │
         │                              ▼
    [PLCs / sensors]              [Analytics / rules]

Hub-and-spoke: edge brokers bridge upward; cloud never bridges back to every edge (use commands down one path).

Peer bridge: two sites exchange topics — watch for loops. Use separate inbound/outbound topic prefixes (site-a/, site-b/).

Protocol bridge: MQTT ↔ Kafka ↔ AMQP via connector services when brokers can't bridge natively.

MQTT clustering

Clustered brokers (EMQX, HiveMQ, VerneMQ) share routing tables and often session state via backend (MDB, Raft, or proprietary gossip).

                    ┌─────────┐
         Clients ──►│  LB     │
                    └────┬────┘
              ┌──────────┼──────────┐
              ▼          ▼          ▼
          [Node 1]   [Node 2]   [Node 3]
              └──────────┼──────────┘
                    shared state
                   (DB / Raft)

Clients connect through a load balancer (TCP passthrough on 8883, not HTTP LB). Sticky sessions help for persistent connections but shouldn't be mandatory if session state is truly shared.

EMQX cluster excerpt — nodes discover via static seed or DNS:

# emqx.conf
cluster {
  discovery_strategy = static
  static.seeds = ["[email protected]", "[email protected]"]
}

Adding nodes: join cluster, verify emqx ctl cluster status, then add to LB pool. Remove nodes gracefully — emqx ctl cluster leave — or clients get abrupt disconnects.

Shared subscriptions.

When many backend services subscribe to the same topic, every instance receives every message. Shared subscriptions load-balance:

$share/analytics-group/factory/+/telemetry

Only one member of analytics-group gets each message. Supported in EMQX, HiveMQ 4+, MQTT 5 shared subscription spec. Mosquitto requires multiple bridge consumers or an external queue — plan accordingly.

Bridge plus cluster pattern.

Our production layout:

1. Edge: Mosquitto per line, bridges telemetry to regional EMQX
2. Regional: 3-node EMQX cluster behind NLB
3. Global: EMQX rule engine republishes aggregates to a central cluster via bridge (filtered topics only)

Rules:

• Edge buffers during WAN outage (queue_qos0_messages carefully — memory risk)
• Cloud commands use retained + LWT only on command topics, not telemetry flood
• Idempotent consumers keyed by (device_id, timestamp, sequence)

Failure modes.

Choosing components.

Managed cloud MQTT removes cluster ops but limits bridge flexibility — evaluate egress costs on bridged volume.

Capacity planning should include bridge bandwidth separately from client fan-in. A site publishing 5 KB/s per device × 2,000 devices is 10 MB/s sustained — compress payloads (CBOR vs JSON) before bridging if WAN links are tight. Run game days: kill one cluster node during peak, disconnect a bridge for 30 minutes, and verify edge buffers drain without OOM. Alert on bridge connection state (notifications_local in Mosquitto) so ops knows when cloud is blind to a plant. Document ownership: who rotates bridge credentials, who approves new topic patterns crossing the DMZ, and which team consumes $share consumer groups. Without that RACI, bridges become permanent tunnels with stale ACLs. Keep a runbook entry for bridge-silent incidents — verify DNS, TLS cert expiry, and upstream connection limits before restarting edge brokers blindly.

Resources

• MQTT Version 5.0 specification — shared subscriptions
• Eclipse Mosquitto bridge configuration
• EMQX clustering documentation
• HiveMQ bridge documentation
• AWS IoT Core MQTT broker limits

Production notes for LLM stacks

When mqtt-bridging-clustering sits on an inference or RAG path, treat user prompts and retrieved chunks as untrusted input. Log correlation IDs and policy decisions—not raw prompts—in production telemetry. Gate risky operations behind explicit authorization at the gateway, not inside ad-hoc tool handlers.

Roll out changes with shadow mode first: record what would have happened under the new rule without blocking traffic. Compare deny rates, latency impact, and false positives for at least one business week before enforcing. Pair enforcement with a runbook entry: symptom, dashboard, rollback (feature flag or config), and owner.

Load-test with production-shaped concurrency. LLM workloads burst differently from CRUD APIs—tail latency and token throttling dominate. If mqtt bridging and clustering protects an invariant (security, billing, data residency), prove the invariant with an automated test that fails CI when someone removes the check.

What teams get wrong

Teams copy a reference architecture without matching their compliance tier, then discover in audit that logs, backups, or support exports reintroduced the data they thought they had eliminated. Another pattern: shipping the demo integration without idempotency, then fighting duplicate side effects when clients retry on model timeouts.

Document the tradeoff you chose—strictness vs recall, cost vs quality, sync vs async—and the metric that tells you if the choice still holds six months later.