Direct answer #
Event-driven warehouse control depends on precise message contracts, strict idempotency, and deterministic recovery patterns. This reference defines the editorial position of Pearl Gateway Systems on these topics. We specify that every event must carry a unique identifier, a correlation identifier, and a sequence number to support ordering and reconciliation. Idempotency is achieved through a consumer-side deduplication key derived from the event identifier, not from business data alone. Retry and dead-letter behavior must be explicit in the contract, with exponential backoff and a bounded retry count. Reconciliation requires a periodic snapshot or state-compare mechanism to detect silent loss. These patterns apply to the integration layer between warehouse control systems, equipment control modules, and higher-level orchestration platforms. The guidance is educational and does not replace site-specific engineering review, safety validation, or the requirements of any cited standard.
Key takeaways #
- Message contracts are the foundation: Define event types, payload schemas, and header semantics before implementation. Ambiguity in the contract propagates to every downstream consumer.
- Idempotency is a consumer responsibility: The producer must provide a unique event identifier, but the consumer must enforce deduplication. Relying on “at-most-once” delivery is not sufficient for warehouse control.
- Ordering is local, not global: Total ordering across all topics is impractical. Use sequence numbers within a partition or key scope to preserve order where it matters, such as for a single crane or conveyor segment.
- Retry and dead-letter policies must be bounded: Infinite retry loops can mask systemic failures. Define a maximum retry count, a backoff schedule, and a dead-letter topic for manual or automated analysis.
- Reconciliation is mandatory: Event streams can silently lose messages. Periodic state comparison between the event log and the actual equipment state is the only reliable way to detect gaps.
- Trace context is essential for diagnosis: Propagate a trace identifier across all hops to correlate events with PLC alarms, equipment control module actions, and recovery attempts.
- Recovery is a designed behavior: Sequence recovery logic must be specified in the contract, including how to handle duplicate, missing, or out-of-order events. This is not an implementation detail.
Scope and intended audience #
This article addresses the integration layer between a warehouse control system (WCS), equipment control modules (ECMs), and programmable logic controllers (PLCs). The audience includes integration engineers, controls engineers, software architects, and technical leads who design or maintain event-driven interfaces in automated warehouses. The content is specifically relevant to systems involving mini-load storage cranes, conveyor segments, shuttle systems, and automated storage and retrieval systems. The editorial recommendations here complement the operational guidance found in the Pearl Gateway article on Equipment Control Modules: Operating Principles and System Boundaries, which defines the boundary between the ECM and the higher-level control system.
The focus is on the message-level behavior: what is sent, how it is identified, how duplicates are handled, how failures are retried, and how the system returns to a known state after an interruption. We do not cover physical layer protocols, PLC programming specifics, or safety-rated communication. Safety functions must follow the applicable machinery safety standards and site-specific risk assessments, which are outside the scope of this editorial guidance.
Event-driven architecture overview for warehouse control #
An event-driven warehouse control architecture uses asynchronous messages to communicate state changes, commands, and acknowledgements between system components. The primary components are the WCS, the ECMs, and the PLCs. The WCS typically handles order management, inventory allocation, and task sequencing. The ECM manages the execution of a single equipment unit, such as a mini-load crane or a conveyor segment. The PLC executes the real-time logic that directly controls motors, actuators, and sensors.
In this architecture, the ECM acts as the intermediary. It receives commands from the WCS, translates them into PLC-level instructions, and reports status back to the WCS. The communication between the WCS and the ECM is typically event-driven, using a message broker such as MQTT or a similar publish-subscribe system. The MQTT Version 5.0 specification [S1] provides a standardized protocol for this type of communication, including features for message expiry, topic aliases, and user properties that can be used to carry trace context.
The key architectural decision is the boundary between the ECM and the WCS. The Pearl Gateway article on Equipment Control Modules: Operating Principles and System Boundaries recommends that the ECM should be responsible for the execution of a single task, while the WCS is responsible for the coordination of multiple tasks. This boundary has direct implications for the message contract. The WCS sends a task command to the ECM, and the ECM responds with a series of status events. The contract must define the lifecycle of a task, from creation to completion or cancellation.
The event-driven approach provides several benefits. It decouples the WCS from the ECM, allowing each to be updated independently. It provides a natural audit trail, as every state change is recorded as an event. It also enables horizontal scaling, as multiple consumers can process events in parallel. However, these benefits come at the cost of increased complexity in handling message loss, duplication, and ordering. The remainder of this article addresses these challenges.
Message contract definition and structure #
A message contract is a formal specification of the messages exchanged between components. It defines the event types, the payload schema, the header fields, and the semantics of each field. The contract is the single source of truth for both producers and consumers. Without a precise contract, integration issues arise from ambiguous field meanings, inconsistent units, and undocumented optionality.
The Pearl Gateway editorial recommendation is to define the contract in three layers: the transport layer, the envelope layer, and the payload layer. The transport layer defines the protocol, such as MQTT 5.0 [S1], and the topic structure. The envelope layer defines the standard headers that every message must carry. The payload layer defines the domain-specific data for each event type.
The envelope layer is the most critical for event-driven reliability. It must include the following fields:
| Field | Type | Description | Example |
|---|---|---|---|
| event_id | UUID (string) | Globally unique identifier for this event instance. Used for deduplication. | 9f8c2b4a-7e3d-4f1a-9c5b-2d6e8f0a1b3c |
| correlation_id | UUID (string) | Identifier that links related events in a conversation or task lifecycle. | 3a1b2c3d-4e5f-6a7b-8c9d-0e1f2a3b4c5d |
| trace_id | string (hex) | Distributed trace identifier for end-to-end correlation. Format per W3C Trace Context [S3]. | 4bf92f3577b34da6a3ce929d0e0e4736 |
| sequence_number | uint64 | Monotonically increasing number within the scope of the correlation_id. Used for ordering. | 1042 |
| event_type | string | Machine-readable type name, e.g., task.started, task.completed. |
task.completed |
| timestamp | ISO 8601 (UTC) | Time when the event was created by the producer. | 2026-08-14T14:32:05.123Z |
| schema_version | string | Version of the payload schema, e.g., 1.2.0. |
1.2.0 |
| content_type | string | Media type of the payload, e.g., application/json. |
application/json |
The event_id is the primary key for idempotency. The correlation_id groups events that belong to the same logical operation, such as a single crane movement cycle. The sequence_number provides a partial ordering within the correlation scope. The trace_id follows the W3C Trace Context specification [S3] and allows the integration team to correlate events across the WCS, the ECM, and the PLC diagnostics.
The payload layer defines the domain-specific data. For a task command, the payload might include the task type, the source location, the destination location, and the load identifier. For a status event, the payload might include the current state, the error code, and the position. The payload schema must be versioned, and the schema_version field in the envelope indicates which version is being used.
Topic structure and naming conventions #
The topic structure determines how messages are routed and filtered. A well-designed topic hierarchy simplifies the consumer logic and reduces the risk of unintended message consumption. The Pearl Gateway editorial recommendation is to use a hierarchical topic structure that mirrors the physical and logical organization of the warehouse.
A recommended topic structure is:
warehouse/{site_id}/equipment/{equipment_id}/{category}/{event_type}
For example:
warehouse/wh001/equipment/crane_07/command/task.assign
warehouse/wh001/equipment/crane_07/status/task.started
warehouse/wh001/equipment/crane_07/status/task.completed
warehouse/wh001/equipment/crane_07/error/task.aborted
The category segment distinguishes between commands, status events, errors, and acknowledgements. This separation allows consumers to subscribe only to the categories they need. For example, a monitoring dashboard might subscribe to status/# and error/#, while the WCS subscribes to status/# and ack/#.
The topic structure also has implications for MQTT 5.0 features. The MQTT 5.0 specification [S1] supports topic aliases, which can reduce the overhead of long topic names. However, topic aliases are session-scoped and must be handled carefully in a warehouse environment where sessions may be interrupted. The editorial recommendation is to use full topic names for critical command messages and to consider topic aliases only for high-frequency status messages where the connection is stable.
The topic structure should be documented in the message contract. The contract should specify the allowed characters, the maximum length, and the naming convention for each segment. This documentation is essential for onboarding new developers and for troubleshooting integration issues.
Event types and task lifecycle modeling #
The message contract must define the complete set of event types and the valid transitions between them. For a task-based system, the lifecycle typically includes the following states: PENDING, ASSIGNED, STARTED, COMPLETED, FAILED, CANCELLED, and ABORTED. The event types correspond to the transitions between these states.
The following table defines a representative task lifecycle for a mini-load crane:
| Event Type | From State | To State | Description | Payload Example |
|---|---|---|---|---|
task.created |
N/A | PENDING | A new task has been created by the WCS. | {"task_id":"T-1001","source":"A-01","destination":"B-03"} |
task.assigned |
PENDING | ASSIGNED | The task has been assigned to a specific ECM. | {"task_id":"T-1001","ecm_id":"crane_07"} |
task.started |
ASSIGNED | STARTED | The ECM has begun executing the task. | {"task_id":"T-1001","position_m":12.5} |
task.completed |
STARTED | COMPLETED | The task finished successfully. | {"task_id":"T-1001","end_position_m":45.0,"duration_s":18.2} |
task.failed |
STARTED | FAILED | The task failed due to an error. | {"task_id":"T-1001","error_code":"E-203","message":"Target position not reached"} |
task.cancelled |
ASSIGNED | CANCELLED | The task was cancelled before execution started. | {"task_id":"T-1001","reason":"Order changed"} |
task.aborted |
STARTED | ABORTED | The task was interrupted by a safety stop or manual intervention. | {"task_id":"T-1001","reason":"E-stop pressed"} |
The lifecycle model must be strictly enforced by the ECM. The ECM should reject any event that does not follow the valid transition path. For example, a task.completed event for a task that is in the ASSIGNED state is invalid and should be treated as a contract violation. The Pearl Gateway article on Sequence Recovery Logic: Common Failure Modes and Diagnostic Evidence provides additional context on how sequence violations manifest in practice.
The contract must also define the semantics of each state. For example, STARTED means that the ECM has committed to executing the task and has begun physical movement. COMPLETED means that the task has reached its final state and no further action is expected. These definitions must be unambiguous to avoid misinterpretation by the WCS.
Idempotency patterns for duplicate event handling #
Idempotency is the property that processing the same event multiple times produces the same result as processing it once. In a distributed system, message duplication is inevitable. The producer may retry a publish operation after a timeout, or the broker may redeliver a message after a consumer failure. The consumer must be able to detect and ignore duplicates.
The Pearl Gateway editorial recommendation is to implement idempotency at the consumer level using the event_id as the deduplication key. The consumer maintains a store of recently processed event_id values. When a new event arrives, the consumer checks the store. If the event_id is present, the event is a duplicate and is ignored. If it is not present, the event is processed and the event_id is added to the store.
The deduplication store must have a retention policy. The retention period should be at least as long as the maximum possible delay between a duplicate delivery. For example, if the broker is configured to redeliver messages for up to 24 hours, the deduplication store must retain entries for at least 24 hours. A longer retention period is safer but requires more storage. A typical warehouse integration might use a retention period of 7 days as an illustrative assumption.
The deduplication key must be the event_id, not a business key. Consider a task completion event. The business key might be the task_id. However, if a task is completed, then a correction is issued, and the task is completed again, the business key would be the same, but the events are different. The event_id distinguishes these two events. The correlation_id and sequence_number can be used to detect out-of-order delivery, but they should not be used as the deduplication key.
The idempotency pattern must be applied to all event types, including commands. If the WCS sends a task.assign command and the command is duplicated, the ECM must not assign the task twice. The ECM should use the event_id of the command to detect the duplicate and respond with an acknowledgement that the command was already processed.
Ordering guarantees and sequence number semantics #
Message ordering is a complex topic in distributed systems. Total ordering across all messages is impractical and unnecessary for warehouse control. The editorial recommendation is to provide partial ordering guarantees within a specific scope. The scope is defined by the correlation_id. For example, all events for a single task execution share the same correlation_id and must be processed in sequence number order.
The sequence_number is a monotonically increasing integer that is unique within the scope of the correlation_id. The producer assigns the sequence_number at the time of event creation. The consumer must track the last processed sequence_number for each correlation_id. If an event arrives with a sequence_number that is not the expected next value, the consumer must handle the gap.
There are three possible gap scenarios:
- Missing event: The consumer receives sequence 1 and then sequence 3, but never receives sequence 2. The consumer should wait for a configurable timeout before declaring sequence 2 as missing. The timeout is an illustrative assumption, typically set to 5 seconds in a warehouse environment.
- Out-of-order event: The consumer receives sequence 3 before sequence 2. The consumer should buffer sequence 3 and wait for sequence 2. If sequence 2 does not arrive within the timeout, the consumer should trigger a reconciliation request.
- Duplicate event: The consumer receives sequence 2 twice. The deduplication mechanism handles this case, and the duplicate is ignored.
The ordering guarantee is only as strong as the broker’s delivery mechanism. MQTT 5.0 [S1] provides ordered delivery within a single topic and QoS level for a given connection. However, if the consumer reconnects, the ordering guarantee may be reset. The consumer must be prepared to handle a reset by re-synchronizing its state with the producer.
The sequence number also enables the consumer to detect gaps that indicate a lost message. If the consumer detects a gap, it should not attempt to process subsequent events for that correlation_id until the gap is resolved. Processing events out of order can lead to incorrect state transitions. For example, processing a task.completed event before the task.started event would be invalid.
Retry strategies and exponential backoff #
Retry is the mechanism by which a consumer attempts to process a failed event again. Retry is necessary for transient failures, such as a temporary database outage or a brief network interruption. However, retry is not appropriate for permanent failures, such as a schema violation or an invalid state transition. The message contract must define the retry policy for each event type.
The Pearl Gateway editorial recommendation is to use exponential backoff with a bounded retry count. Exponential backoff means that the delay between retries increases exponentially. A common formula for the delay is:
delay_n = initial_delay * base^(n-1)
Where:
delay_nis the delay before the n-th retry, in seconds.initial_delayis the delay before the first retry, in seconds.baseis the exponential base, typically 2.nis the retry attempt number, starting at 1.
For example, with an initial_delay of 1 second and a base of 2, the delays would be 1, 2, 4, 8, 16, and 32 seconds. These values are illustrative assumptions and must be tuned based on the specific system’s latency requirements and failure characteristics.
The retry count must be bounded. An infinite retry loop can mask a systemic failure and cause the event queue to back up. The editorial recommendation is a maximum of 5 retries as an illustrative assumption. After the maximum retry count is reached, the event is moved to a dead-letter topic.
The retry policy must be part of the message contract. The contract should specify the initial delay, the base, the maximum retry count, and the dead-letter topic. The consumer should log each retry attempt with the retry count and the error reason. This log is essential for diagnosing persistent failures.
It is important to distinguish between retry at the consumer level and retry at the broker level. MQTT 5.0 [S1] provides QoS levels that control message delivery. QoS 1 guarantees at-least-once delivery, which means the broker will redeliver the message if the acknowledgement is lost. QoS 2 guarantees exactly-once delivery but has higher overhead. The editorial recommendation for warehouse control is to use QoS 1 for most messages and to rely on consumer-side idempotency to handle duplicates. QoS 2 may be used for critical commands, but the overhead must be evaluated.
Dead-letter topics and error handling #
A dead-letter topic is a destination for messages that cannot be processed successfully after the maximum retry count. The dead-letter topic provides a place for these messages to be stored for analysis, manual intervention, or automated reprocessing after a fix is deployed.
The message contract must define the dead-letter topic structure. The editorial recommendation is to use a separate topic hierarchy for dead letters:
warehouse/{site_id}/deadletter/{original_category}/{event_type}
For example:
warehouse/wh001/deadletter/status/task.completed
When a message is moved to the dead-letter topic, the envelope must be preserved, and additional metadata must be added. The additional metadata should include the original topic, the retry count, the last error message, and the timestamp of the final failure. This metadata is essential for diagnosing the root cause.
The dead-letter topic must be monitored. A growing dead-letter queue is an indicator of a systemic problem. The Pearl Gateway article on Condition Monitoring: Common Failure Modes and Diagnostic Evidence discusses how to monitor such indicators. The monitoring system should alert the integration team when the dead-letter queue depth exceeds a threshold. The threshold is an illustrative assumption, typically set to 10 messages per hour for a single equipment unit.
There are two approaches to handling dead-letter messages: automated reprocessing and manual intervention. Automated reprocessing is appropriate when the root cause is a transient condition that has been resolved. For example, if the dead-letter message was caused by a temporary database outage, the message can be reprocessed after the database is back online. Manual intervention is appropriate when the root cause is a permanent condition, such as a schema violation or a bug in the consumer logic. In this case, the message must be inspected and the consumer logic must be fixed before reprocessing.
The contract should specify the reprocessing procedure. The procedure should include a mechanism to replay dead-letter messages to the original topic or to a special reprocessing topic. The reprocessing must preserve the original event_id to maintain idempotency.
Reconciliation and state comparison #
Reconciliation is the process of comparing the state derived from the event stream with the actual state of the equipment. This process is necessary because event streams can silently lose messages. A message may be lost due to a broker failure, a network partition, or a consumer crash. Reconciliation detects these losses and triggers corrective action.
The editorial recommendation is to implement a periodic reconciliation process. The process works as follows:
- The WCS maintains a desired state for each piece of equipment. The desired state is derived from the commands that have been sent.
- The ECM maintains an actual state for each piece of equipment. The actual state is derived from the PLC signals.
- At a defined interval, the WCS requests a state snapshot from the ECM. The interval is an illustrative assumption, typically set to 60 seconds for a warehouse control system.
- The ECM responds with a complete state snapshot, including the current position, the current task, and the current mode.
- The WCS compares the snapshot with its desired state. If there is a discrepancy, the WCS initiates a recovery procedure.
The state snapshot must include a version or timestamp to allow the WCS to detect stale snapshots. The snapshot should also include the last processed sequence_number for each active correlation_id. This information allows the WCS to determine if any events are missing.
Reconciliation is particularly important for mini-load cranes, where a missed event can lead to a discrepancy between the WCS’s inventory record and the physical location of a load. The Pearl Gateway article on Mini-Load Storage Cranes: Common Failure Modes and Diagnostic Evidence describes how such discrepancies can manifest as diagnostic evidence.
The reconciliation process must be designed to be safe. The WCS must not issue corrective commands based on a single snapshot if there is any risk of collision or unsafe movement. The reconciliation process should be integrated with the Restart Authorization logic, which defines the conditions under which equipment can be restarted after a fault.
Trace context propagation for end-to-end diagnosis #
Trace context is the set of identifiers that allow a single logical operation to be traced across multiple components. The W3C Trace Context specification [S3] defines a standard format for these identifiers. The specification defines two headers: traceparent and tracestate. The traceparent header carries the trace_id, the parent_id, and the trace flags. The tracestate header carries vendor-specific data.
In a warehouse control system, the trace context should be propagated from the WCS to the ECM and from the ECM to the PLC diagnostics. The trace_id is the same for all events that belong to the same logical operation. For example, a single task execution might generate a task.created event, a task.started event, and a task.completed event. All three events should share the same trace_id.
The trace_id is distinct from the correlation_id. The correlation_id groups events that belong to the same business conversation, such as a task. The trace_id groups events that belong to the same distributed trace, which may span multiple business conversations. For example, a single user request might trigger multiple tasks. All of these tasks would share the same trace_id but have different correlation_id values.
The message contract must include the trace_id in the envelope. The format should follow the W3C Trace Context specification [S3], which defines the trace_id as a 16-byte value represented as a 32-character hexadecimal string. The parent_id is an 8-byte value represented as a 16-character hexadecimal string.
Trace context is essential for diagnosing integration issues. When a task fails, the integration engineer can use the trace_id to retrieve all events related to the operation, including the command, the status updates, and the error events. This capability is particularly valuable when investigating issues that span multiple components, such as a timing issue between the WCS and the ECM.
Heartbeat and liveness detection #
Heartbeat messages are periodic signals that indicate that a component is alive and functioning. In an event-driven warehouse control system, heartbeats serve two purposes: they detect component failures, and they provide a baseline for latency measurement.
The Pearl Gateway article on Heartbeat and Watchdog Logic: Inspection Points and Early Warning Signs provides detailed guidance on this topic. The editorial recommendation is to define a heartbeat event type in the message contract. The heartbeat event should include the component ID, the current state, and a timestamp.
The heartbeat interval is an illustrative assumption, typically set to 5 seconds for an ECM. The WCS should consider the ECM to be unhealthy if no heartbeat is received within a timeout period. The timeout period is typically 3 times the heartbeat interval, or 15 seconds in this example. These values are illustrative assumptions and must be tuned based on the network latency and the criticality of the component.
Heartbeat messages should not be used for idempotency tracking. Heartbeats are high-frequency and low-value messages. Storing every heartbeat event_id in the deduplication store would consume excessive storage. Instead, the consumer should treat heartbeats as ephemeral and not apply the standard deduplication logic.
Heartbeat monitoring should be integrated with the reconciliation process. If the WCS detects a missing heartbeat, it should not immediately declare the ECM as failed. Instead, it should wait for the timeout period and then initiate a state query. If the state query succeeds, the heartbeat may have been lost due to a network issue. If the state query fails, the ECM is likely down, and the recovery procedure should be initiated.
Worked example #
This section provides a worked example of an event-driven task execution for a mini-load crane. The example illustrates the message contract, idempotency, ordering, and recovery behavior.
Inputs:
- Task ID:
T-2001 - Source location:
A-01 - Destination location:
B-03 - Crane ID:
crane_07 - Maximum retry count: 5 (illustrative assumption)
- Initial retry delay: 1 second (illustrative assumption)
- Retry base: 2 (illustrative assumption)
- Deduplication retention period: 7 days (illustrative assumption)
- Sequence gap timeout: 5 seconds (illustrative assumption)
Step 1: Task creation. The WCS publishes a task.created event to the topic warehouse/wh001/equipment/crane_07/command/task.assign. The envelope is:
event_id: 11111111-1111-1111-1111-111111111111
correlation_id: aaaa0000-0000-0000-0000-000000000001
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
sequence_number: 1
event_type: task.created
timestamp: 2026-08-14T14:30:00.000Z
schema_version: 1.0.0
Step 2: Task assignment. The ECM receives the event, checks its deduplication store, and finds no entry for event_id 11111111-1111-1111-1111-111111111111. The ECM processes the event and stores the event_id. The ECM publishes a task.assigned event:
event_id: 22222222-2222-2222-2222-222222222222
correlation_id: aaaa0000-0000-0000-0000-000000000001
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
sequence_number: 2
event_type: task.assigned
timestamp: 2026-08-14T14:30:00.050Z
Step 3: Task started. The ECM begins executing the task and publishes a task.started event:
event_id: 33333333-3333-3333-3333-333333333333
correlation_id: aaaa0000-0000-0000-0000-000000000001
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
sequence_number: 3
event_type: task.started
timestamp: 2026-08-14T14:30:01.000Z
Step 4: Duplicate event. The WCS receives a duplicate of the task.started event due to a broker redelivery. The WCS checks its deduplication store and finds the event_id 33333333-3333-3333-3333-333333333333. The duplicate is ignored.
Step 5: Task completed. The ECM completes the task and publishes a task.completed event:
event_id: 44444444-4444-4444-4444-444444444444
correlation_id: aaaa0000-0000-0000-0000-000000000001
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
sequence_number: 4
event_type: task.completed
timestamp: 2026-08-14T14:30:19.000Z
Step 6: Reconciliation. At the next reconciliation interval (60 seconds, illustrative assumption), the WCS requests a state snapshot from the ECM. The ECM responds with its current state: IDLE, last completed task T-2001, last processed sequence number 4. The WCS compares this with its desired state and finds no discrepancy.
Intermediate calculations:
- Retry delays:
delay_1 = 1 * 2^(1-1) = 1 second,delay_2 = 1 * 2^(2-1) = 2 seconds,delay_3 = 1 * 2^(3-1) = 4 seconds,delay_4 = 1 * 2^(4-1) = 8 seconds,delay_5 = 1 * 2^(5-1) = 16 seconds. - Total retry time:
1 + 2 + 4 + 8 + 16 = 31 seconds.
Result: The task was successfully created, assigned, started, and completed. The total task duration was 19 seconds (from 14:30:00 to 14:30:19). No retries were needed, and no dead-letter messages were generated.
Sensitivity analysis: The task duration is sensitive to the crane’s travel speed and the distance between locations. If the distance between A-01 and B-03 is 30 meters (illustrative assumption) and the crane’s average speed is 2 m/s (illustrative assumption), the expected travel time is 30 / 2 = 15 seconds. The observed duration of 19 seconds includes 4 seconds of overhead for load pickup and setdown. If the distance were 45 meters, the expected travel time would be 45 / 2 = 22.5 seconds, and the total task duration would be approximately 26.5 seconds.
Limitations: This example assumes a single task with no interruptions. It does not cover the case where a task fails and must be retried. It also does not cover the case where a sequence gap is detected. In a real system, the reconciliation process would detect a gap and trigger a recovery procedure. The example also assumes that the broker delivers messages in order, which may not hold in all network conditions.
Recovery patterns for sequence gaps and state divergence #
Recovery is the process of returning the system to a consistent state after a failure. The recovery pattern depends on the type of failure. This section covers two common failure types: sequence gaps and state divergence.
Sequence gap recovery. A sequence gap occurs when the consumer detects a missing sequence_number for a given correlation_id. The recovery procedure is as follows:
- The consumer waits for the sequence gap timeout (5 seconds, illustrative assumption).
- If the missing event does not arrive, the consumer publishes a
reconciliation.requestedevent to the WCS. - The WCS queries the ECM for the current state and the last processed sequence number.
- The WCS compares the ECM’s state with its own state and determines which events are missing.
- The WCS re-publishes the missing events or issues a corrective command.
This pattern is described in more detail in the Pearl Gateway article on Sequence Recovery Logic: Common Failure Modes and Diagnostic Evidence.
State divergence recovery. State divergence occurs when the WCS’s desired state and the ECM’s actual state do not match. This can happen due to a missed event, a manual intervention, or a PLC fault. The recovery procedure is as follows:
- The WCS identifies the divergence during the reconciliation process.
- The WCS determines the severity of the divergence. A minor divergence, such as a position offset, may be corrected by a new command. A major divergence, such as an unknown load location, requires manual intervention.
- The WCS issues a corrective command to the ECM.
- The ECM executes the corrective command and publishes a new status event.
- The WCS verifies that the divergence is resolved.
The recovery procedure must be integrated with the Restart Authorization logic. The equipment must not be restarted until the divergence is understood and the recovery procedure is authorized. The Pearl Gateway article on Safe Fault Investigation: Operating Principles and System Boundaries provides guidance on how to investigate faults safely.
When this guidance does not apply #
This guidance does not apply to safety-rated communication. Safety functions, such as emergency stops, light curtains, and safety-rated speed monitoring, must be implemented using safety-rated protocols and components that comply with the applicable machinery safety standards. The event-driven patterns described in this article are not suitable for safety functions because they introduce latency, potential for message loss, and non-deterministic timing.
This guidance does not apply to real-time motion control. The communication between the PLC and the servo drives, or between the PLC and the safety PLC, must use deterministic, low-latency protocols. Event-driven messaging is not appropriate for this layer.
This guidance does not apply to systems where the message broker is not reliable. If the broker does not provide persistent storage or if the network is prone to prolonged partitions, the patterns described here may not be sufficient. In such cases, a different architecture, such as a shared database or a file-based handshake, may be more appropriate.
This guidance does not apply to single-component systems. If the WCS and the ECM run on the same machine and communicate through in-memory calls, the overhead of event-driven messaging is unnecessary. The patterns described here are for distributed systems where components communicate over a network.
This guidance does not apply to systems that require exactly-once processing without consumer-side idempotency. The patterns described here rely on consumer-side deduplication. If the consumer cannot maintain a deduplication store, a different approach is required.
Implementation checklist for integration teams #
The following checklist summarizes the key implementation steps for an event-driven warehouse control integration. The checklist is a
Sources and standards #
- OASIS — MQTT Version 5.0 Specification. In “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns”, source [S1] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- OPC Foundation — OPC UA Online Reference. In “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns”, source [S2] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- W3C — Trace Context Recommendation. In “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns”, source [S3] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- NIST — Guide to Operational Technology Security, SP 800-82 Rev. 3. In “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns”, source [S4] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- NASA — NASA Systems Engineering Handbook. In “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns”, source [S5] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
Revision and editorial note #
The Pearl Gateway Editorial Team prepared “Event-Driven Warehouse Control: Message Contracts, Idempotency and Recovery Patterns” from the five linked source records. The published guide remains educational and requires site evidence before application.