# Message Broker Interfaces: Design Assumptions and Verification Evidence - Canonical: https://pearlgateway.net/docs/message-broker-interfaces-design-assumptions-and-verification-evidence/ - Updated: 2026-10-09T08:47:00+00:00 ## Introduction Message broker interfaces are the connective tissue between programmable logic controllers (PLCs) and warehouse control systems (WCS). They carry task commands, status acknowledgements, heartbeat signals, and alarm notifications across a boundary where two very different engineering cultures meet: the deterministic world of controls and the event-driven world of software orchestration. Despite their apparent simplicity — one side publishes, the other subscribes — message broker interfaces fail more often from mismatched design assumptions than from hardware faults or network loss. This article is an independent technical review of those assumptions. It explains what a broker interface is designed to assume, what evidence proves or disproves those assumptions, and how to interpret that evidence without falling into common diagnostic traps. The guidance is aimed at warehouse operators, maintenance engineers, and controls teams who must keep interface failures short, safe, and well-documented. Site procedures, lockout requirements, OEM documentation, and competent engineering judgment always take priority over any general guidance presented here. ## The Broker as a Boundary, Not a Wire A common mental model treats the message broker as a transparent pipe: the WCS publishes a task, the PLC receives it, and the world moves on. In practice, the broker is an active participant with its own persistence, retry logic, session management, and ordering rules. It decouples the two systems in time and in failure space. That decoupling is valuable, but it also means that neither side can assume the other is in the same state at the same instant. Consider a typical palletizer handshake. The WCS publishes a “move to lane 3” command. The broker stores it and forwards it to the PLC. The PLC executes it and publishes a “completed” event. Between those two publications, the broker may have restarted, the PLC may have lost its subscription, or a duplicate copy of the command may have been delivered. Each of these conditions is survivable, but only if the interface design anticipated it and the verification evidence can prove it. Treating the broker as a boundary object — something both systems share but neither owns — is the first step toward productive troubleshooting. It is not a wire. It is a service with its own operational state, and that state must be monitored independently. ## Design Assumptions Carried into Integration Most interface failures trace back to five design assumptions. These are seldom written down in a single document, but they are embedded in code, configuration, and testing habits. Naming them explicitly gives a team a checklist for verification. ### Delivery Semantics The most common assumption is that every published message is delivered exactly once. Most brokers, however, offer at-least-once delivery by default: a message may be redelivered after an acknowledgement is lost or after a subscriber reconnects. If the WCS command is not idempotent — meaning the PLC can process the same command twice without harm — a single redelivery can cause a double cycle, a second pallet drop, or a conflicting zone move. The design assumption must state explicitly which level of delivery the interface requires and what the PLC will do with an unexpected duplicate. ### Ordering Guarantees Ordering is often assumed to be global: messages arrive in the same order they were published. Brokers typically guarantee ordering only within a single topic, partition, or queue, not across the entire message space. If a WCS publishes “allocate destination” and then “assign conveyor route” as two separate messages, the PLC may receive the route before the destination. This is not a broker malfunction; it is a design assumption that was never verified. The interface design must either consolidate state into a single message or define ordering rules at the topic level. ### Idempotency Expectations Idempotency is the property that processing the same message twice produces the same result as processing it once. Many interfaces assume the PLC is idempotent because commands are written as absolute instructions, such as “set gate to position A.” But side effects — like incrementing a counter, appending to a log, or triggering an interlock sequence — are often non-idempotent. The verification process must test duplicate delivery explicitly, not assume it away. ### Schema and Naming Stability The interface assumes that topic names, payload fields, data types, and units remain stable over time. In practice, a WCS upgrade may add a field, a PLC firmware update may change an integer encoding, or a broker reconfiguration may rename a topic. Messages that fail to parse are often silently dropped by the subscriber, and the only evidence is a missing action at the physical layer. The design assumption should include a versioning strategy and a mechanism for discovering schema mismatches before they become operational outages. ### Timing and Heartbeat Budgets Every interface has an implicit timing budget: how long the WCS will wait before assuming a command failed, how long the PLC will wait before assuming the broker is unavailable, and how often heartbeats are exchanged. These budgets are rarely aligned. A WCS may time out after two seconds while the broker requires three seconds to promote a new leader. The result is an interface that performs well in steady state and fails catastrophically during broker failover. Design assumptions must include the full range of broker lifecycle events, not just normal operation. ## Component Interactions in the Message Path A complete message path includes the WCS application, the broker client library, the broker server, the network, the PLC-side broker interface, and the PLC logic itself. Each component can delay, drop, or duplicate a message. When verifying an interface, evidence must be collected from at least three points in this path: the point of publication, the point of delivery, and the point of execution. Two-point evidence — for example, broker logs and PLC logs — is usually sufficient to localize a fault, but only if the timestamps are synchronized and the correlation identifiers match. Correlation identifiers are the most underused diagnostic asset in warehouse integration. A command ID that is generated by the WCS, embedded in the payload, echoed in the PLC acknowledgement, and logged by the broker gives an investigator a single thread through the entire path. If no correlation ID exists in the payload, the interface design should be corrected before further troubleshooting is attempted. ## Symptoms That Indicate Assumption Failure Certain operational symptoms point directly to design assumptions rather than hardware faults. The following list is representative, not exhaustive: - Repeated execution of the same command after a broker restart, visible as double pallet movements or duplicate conveyor activation. - Commands arriving out of order, causing the PLC to execute a newer task before an older one and then overwrite the result. - Missing closeout messages, where the WCS believes a task is still in progress even though the PLC has physically completed it. - Silent topic mismatches, where messages are published but never subscribed to because of a case difference, a wildcard mismatch, or a trailing slash. - Intermittent timeouts that occur only during broker leader election or certificate rotation. - Heartbeat failures that appear simultaneously on multiple PLCs, suggesting a broker-level issue rather than a field device problem. - Messages that are delivered after the WCS has already timed out, generating late acknowledgements that conflict with the WCS recovery state. Each symptom has multiple possible causes. The purpose of evidence collection is to narrow the cause set to one verifiable explanation. ## Collecting Verification Evidence Verification evidence should be collected deliberately, not reconstructed after a fault. A minimal evidence pack for a message broker interface includes the following items: - Broker logs covering publish, delivery, acknowledgement, and reconnection events for the affected topic. - WCS application logs showing command generation, timeout handling, and recovery actions. - PLC-side interface logs showing message receipt, parsing results, and execution acknowledgements. - Network capture at the broker and PLC interfaces for at least one failure window, if available. - Configuration snapshots of topic names, quality-of-service settings, session persistence, and heartbeat intervals. - A timestamp synchronization check across the WCS server, the broker host, and the PLC time source. Evidence should be timestamped to a common time base. A delta of more than a few hundred milliseconds between system clocks can make a healthy interface appear faulty or a faulty interface appear healthy. The simplest way to confirm time alignment is to publish a timestamped beacon message and compare the reception times recorded on each side. ## Verification Evidence Decision Table The table below maps common design assumptions to the evidence required to verify them, the criterion for acceptance, and the failure signature that indicates a broken assumption. Use it as a starting point for interface test plans and post-incident reviews. Design Assumption | Evidence Required | Acceptable Verification | Failure Signature Command delivery is exactly once | Correlation IDs tracked across WCS, broker, and PLC logs | Each command ID appears once in publish logs and once in PLC execution logs over a test window | Duplicate command IDs appear in PLC execution logs after a broker retry or subscriber reconnect Message ordering is preserved | Per-topic sequence numbers and timestamps on both sides | Sequence numbers are strictly monotonic on each topic or partition during a stress test | PLC receives a higher sequence number before a lower one, causing stale state override Broker acknowledgement implies PLC action | Broker publish acknowledgements compared with PLC execution acknowledgements | PLC execution ACK follows broker publish ACK within the agreed timing budget | Broker ACK present with no PLC execution ACK; message disappears within the broker session Schema and topic names are stable | Schema registry or versioned topic audit at both endpoints | All payloads parse successfully after broker, WCS, and PLC restarts | Silent payload drops, parse exceptions, or messages routed to an unmonitored topic Reconnect restores full subscription state | Connection and session recovery logs in the broker and PLC client | Subscriptions re-establish within broker session timeout after a forced PLC restart | Messages published during PLC downtime are never delivered, and the PLC starts without the latest command End-to-end latency is bounded | Timestamps from WCS publish, broker delivery, and PLC execution | P95 latency remains within the system budget during normal inbound and outbound load | Consumer lag grows while broker CPU and network utilization remain low, indicating a downstream bottleneck The table is a diagnostic decision aid, not a substitute for a site-specific test procedure. When using it, record which assumption is being tested, what evidence was captured, and who reviewed the result. ## Interpretation Errors That Derail Troubleshooting Evidence is only as useful as the interpretation applied to it. Several recurring interpretation errors undermine interface investigations. The first error is treating a broker publish acknowledgement as proof of PLC delivery. A broker may acknowledge a message once it is persisted, not once it is consumed. The gap between persistence and consumption is where messages are lost due to subscription failures or session expiry. Checking the broker console will not reveal this gap; only the PLC execution log can confirm consumption. The second error is confusing consumer lag with broker failure. Consumer lag — the difference between the newest published message and the newest consumed message — can grow because the PLC is slow, because the PLC client is blocked, or because a downstream process in the PLC routine is stuck. A broker with low CPU and memory usage is probably not the bottleneck. The evidence should be sought at the subscriber, not the broker. The third error is misreading order preservation. A broker may maintain order within a partition but not across partitions. If the WCS publishes to multiple partitions, the PLC can legitimately receive messages in a different order. Investigators should confirm the partitioning scheme before blaming the broker for reordering. The fourth error is treating a single successful test as sufficient proof. Message broker interfaces are stochastic: they fail under load, during failover, or after certificate rotation, but they may pass hundreds of simple ping-style tests. Verification evidence should include failure-injection scenarios, restart scenarios, and sustained load, not a single happy-path run. ## Maintenance Implications and Interface Governance Once an interface is verified, it must be governed. Maintenance activities that affect the broker, the WCS, or the PLC can silently invalidate the verified assumptions. Certificate rotation is a frequent source of interface outages. When a broker certificate expires, client connections are dropped. If the WCS does not automatically reconnect with the new trust chain, messages queue on the broker side, and the PLC sees a silent gap. The maintenance plan should include a certificate renewal drill that verifies reconnection and message replay behavior, not just a certificate validity check. Session persistence settings also require governance. A PLC that uses a clean session will not receive messages published while it is offline. That may be the desired behavior, but it must be an explicit decision, not a default. If the operation requires guaranteed delivery of commands across a PLC restart, the session must be persistent and the topic must be configured for retention. The maintenance documentation should record which sessions are persistent and what behavior is expected during a PLC outage. Topic and schema governance is a third maintenance concern. Any change to a topic name, payload field, or enum value must go through the same verification process as the original integration. Informal changes, such as a WCS developer adding a field without updating the PLC parser, are a leading cause of silent interface degradation. A versioned topic convention — for example, a topic name that includes a major version number — allows both sides to evolve without breaking the interface. Finally, the broker itself has a lifecycle. Upgrades to broker software, operating system patches, and network firewall changes all have the potential to alter delivery semantics. The controls team should treat these changes as interface changes, not infrastructure maintenance, and apply the same evidence-based verification before rollout. ## Escalation Boundaries Interface troubleshooting often blurs the line between controls responsibility and IT responsibility. Clear escalation boundaries prevent both duplicated effort and dangerous gaps. As a general division, the controls team owns behavior on the PLC side of the broker boundary, including subscription logic, payload parsing, and execution acknowledgement. The WCS or software team owns the publishing application, correlation ID generation, and timeout handling. The broker platform team, if one exists, owns the broker process, its configuration, and its infrastructure. The network team owns the physical and logical connectivity between the hosts. Escalate to the broker platform team when evidence shows message loss, duplication, or reordering that the broker itself causes. Escalate to the network team when packet capture shows loss, retransmission, or asymmetric routing. Escalate to a higher engineering authority when the failure involves safety-related interlocks, zone control, or any condition that could put personnel at risk. In those cases, stop testing, preserve evidence, and follow the site’s incident and safety procedures. Never attempt to bypass a safety device or interlock to keep a message stream alive. The escalation boundary should also define who has authority to declare the interface healthy. A developer who wrote the publisher, a technician who observed the PLC behavior, and an operator who saw the physical movement may all have partial ## Related Pearl Gateway Guides - [Warehouse Control Systems: Operating Principles and System Boundaries](/docs/warehouse-control-systems-operating-principles-and-system-boundaries/) - [WMS-to-WCS Interfaces: Operating Principles and System Boundaries](/docs/wms-to-wcs-interfaces-operating-principles-and-system-boundaries/) - [Industrial I/O Mapping: Operating Principles and System Boundaries](/docs/industrial-i-o-mapping-operating-principles-and-system-boundaries/)