# Sequence Recovery Logic: Failure Containment and Corrective Action Planning - Canonical: https://pearlgateway.net/docs/sequence-recovery-logic-failure-containment-and-corrective-action-planning/ - Updated: 2026-08-22T07:55:00+00:00 Sequence recovery logic (SRL) governs what happens after a commanded movement stops before its intended completion. In an automated warehouse, a sequence is not merely a list of moves; it is a binding relationship between the PLC’s real-world state, the WCS’s virtual order state, and the material tracking record that links them. When a pallet jams, a sensor misses a transition, or a communication timeout occurs, the physical system changes in ways that may not match the control model. SRL is the organized method for restoring that match, or for deciding that it cannot be restored without human intervention. It is deliberately broader than fault handling: a fault tells you what stopped; recovery logic tells you whether to re-drive, back out, cancel, or hand over to a maintenance procedure. This article describes the interaction between PLC state machines and WCS orchestration, the evidence needed for sound recovery decisions, common interpretation errors, and the documentation trail that demonstrates safe recovery. ## The Layer Between PLC States and WCS Orchestration Sequence recovery does not belong exclusively to the PLC or the WCS. It lives in the interface between them. Understanding this boundary is the first condition for troubleshooting recovery events without making them worse. The PLC executes a state machine at a fixed scan rate, mirroring the physical equipment with discrete states such as Standby, Running, Faulted, and Halted. These states are deterministic and safety-relevant: when a photoeye indicates an unexpected blockage, the PLC is designed to stop motion in the affected zone within a bounded time. The WCS, by contrast, operates at order level. It manages jobs, destinations, routing decisions, and the material tracking database that records which load is where and what its next intended move should be. It is not a real-time controller; it is a coordination engine that issues commands and receives acknowledgements. Between the two sits the message interface. A command flows down. An acknowledgement flows back. A status or completion message follows. Sequence recovery logic is the protocol for what both sides do when that flow is interrupted. For example, if a PLC is in a Faulted state and a WCS command is still pending, reissuing the command without first resolving the fault would create a second, contradictory instruction set. The PLC state machine should reject it, but a poorly designed recovery routine may not realize why. The recovery logic therefore needs to interpret PLC states as sequence events, not just as device alarms. This interpretation is what separates structured recovery from trial-and-error operations. ## Core Interactions and the Recovery Handshake Recovery is a dialogue, not a unilateral reset. The most reliable implementations use a defined handshake with explicit message types: a Recovery Request, a Recovery Acknowledgment, and a Recovery Complete. When either side detects a discrepancy, it initiates the handshake instead of simply pushing new commands into the system. Consider a common case: the WCS sends a transport command from Zone A to Zone B. The PLC acknowledges, motion begins, but a sensor fails to confirm the load arriving at Zone B. The WCS sees a timeout. At this point, the WCS should not re-drive the same command. It should issue a Recovery Request that asks the PLC for its current zone states, the last completed step, and any fault register that may have been set. The PLC enters a Recovering state, performs an internal consistency check, and replies with a snapshot of its tracking data. The WCS then compares that snapshot with its own database. The decision to re-drive, reverse, or reject the load depends entirely on whether those two views can be reconciled. Message sequence numbers and heartbeat counters support this handshake in both directions. A gap in sequence numbers often indicates a lost message, which may be the root cause of an apparent position mismatch. Similarly, a heartbeat loss during an active move should be treated as an incomplete sequence, not as a silent idle event. The recovery handshake is the mechanism that reconstructs intent after such interruptions, and it must be designed to fail safe when the reconstruction is ambiguous. ## Distinguishing Containment from Corrective Action Failure containment and corrective action planning solve different problems and must not be conflated. Containment is immediate, automatic, and narrow. Its purpose is to limit the propagation of an error: stop the affected zone, block upstream release, prevent a merge point from accepting a new load, and hold the state until further instruction. Containment is a PLC responsibility because it acts in real time and is bound by the equipment’s physical response time. Corrective action planning is deliberative. It answers the question: what is the next valid sequence? Should the load be re-driven, returned to origin, transferred to a reject lane, or assigned a new destination? This decision often involves the WCS, which holds order context that the PLC does not. The WCS can evaluate whether a downstream buffer has space, whether the original order is still active, and whether manual intervention is required. Good SRL designs keep these two functions separate. A recovery routine that combines containment and corrective planning into a single automatic action can inadvertently create new errors, such as releasing a blocked zone before the underlying tracking mismatch is resolved. ## Recognizable Symptoms and the Need for Evidence Operators and maintenance teams typically recognize an SRL event through a small set of symptoms. These include a zone that remains occupied with no active WCS order, a command that times out repeatedly, a discrepancy between the WCS load count and the physical pallet count at a transfer point, or a PLC that rebooted while the WCS held an active job. Each symptom points to a different layer of the system, and none should be treated as self-evident. Evidence collection is the foundation of any recovery decision. The following data should be gathered before a reset or re-drive is attempted: - Timestamps from both the PLC and the WCS logs, ideally synchronized to a common time source. - The last command message and its sequence number, including whether an acknowledgement was sent. - The PLC state transitions during the event, including the exact fault code and the time each state was entered. - The WCS order status and the material tracking record for the affected load. - Any record of manual intervention, such as a jog operation or a physically removed pallet, which is often the missing explanation for a tracking mismatch. - Video evidence from the affected area if available; it is frequently the fastest way to resolve a dispute between PLC and WCS records. The discipline of collecting evidence before acting applies equally to automatic and manual recovery. If the system’s own recovery logic is to make a decision, it must have been designed to capture this data. If a human is making the decision, the same evidence must be presented at an HMI or in a maintenance log. Acting without evidence is the most common cause of failed recovery attempts. ## The Diagnostic Decision Table The table below associates common observable symptoms with the containment question, the evidence that should be collected first, and the corrective planning question that remains open. It is intended as a field aid, not as a substitute for site-specific logic documentation. Observable Symptom | Containment Question | Evidence to Collect First | Corrective Planning Question Zone reports occupied, but no WCS order is in transit. | Are upstream zones blocked from releasing into this zone? | Image of the zone, the last PLC state change, and the last WCS order for that zone. | Should the load be reassigned to a new order, or removed from tracking as a phantom? WCS command times out before completion. | Did the PLC enter a Faulted state, or is the command still in progress at a slower rate? | PLC state at the moment of timeout, the command sequence ID, and the last status message sent by the PLC. | Can the command be safely reissued, or does the PLC require a full sequence reset? Pallet count mismatch at a merge or transfer point. | Is the transfer point held in a blocked state to prevent further accumulation? | Counts from both sides, timestamps of the last transfers, and any manual pallet movement records. | Which side of the transfer has the accurate tracking record, and how is the count reconciled? PLC rebooted while the WCS held an active order. | Are all outputs in a safe, de-energized state after the reboot? | Boot time, the WCS order state at that time, and the PLC’s stored tracking data ## Related Pearl Gateway Guides - [OPC UA Integration: Common Failure Modes and Diagnostic Evidence](/docs/opc-ua-integration-common-failure-modes-and-diagnostic-evidence/) - [Message Broker Interfaces: Common Failure Modes and Diagnostic Evidence](/docs/message-broker-interfaces-common-failure-modes-and-diagnostic-evidence/) - [PLC Conveyor State Machines: Selection Criteria and Application Boundaries](/docs/plc-conveyor-state-machines-selection-criteria-and-application-boundaries/) ## Site-Specific Review Worksheet This educational worksheet supports a structured review of sequence recovery logic: failure containment and corrective action planning. Begin by identifying the equipment boundary, control ownership, operating modes, material characteristics, upstream dependencies and downstream consequences. Record what the system is expected to do, what was actually observed and which evidence is time-aligned. Avoid changing several variables at once, because simultaneous changes make cause and effect difficult to establish. ### Evidence to collect - Operating mode, active mission or route, and the exact sequence state. - Alarm history, device state changes and controller timestamps. - Physical observations such as alignment, contamination, wear, obstruction and load condition. - Recent maintenance, software changes, parameter changes and recurring work orders. - Upstream and downstream readiness, including blocked, starved and unavailable conditions. ### Decision boundaries Use approved site procedures and competent engineering judgment before intervention. General information in the Controls, PLC & WCS Integration library cannot determine whether a specific machine is safe to enter, restart or modify. Preserve original settings, document authorized adjustments and establish a rollback point before controlled testing. When evidence conflicts, stop and resolve the timestamp, naming or measurement discrepancy before drawing a conclusion. ### Closeout record A useful closeout record states the symptom, confirmed cause, evidence, corrective action, validation method, residual risk and follow-up owner. It should also identify whether the event exposed a design weakness, maintenance gap, training issue, spare-parts issue or monitoring blind spot. This turns a single recovery into reusable reliability knowledge without treating one observation as universal. ## Evidence Matrix for Operational Review Evidence group \| Questions to answer \| Why it matters Sequence state \| What mode, step, mission and interlock state were active? \| Separates a physical problem from an expected control hold. Material condition \| Were load dimensions, orientation, stability and spacing within the intended envelope? \| Explains faults that appear random when only controller data is reviewed. Device evidence \| Which inputs changed, in what order, and against which timestamp? \| Supports repeatable diagnosis instead of component substitution by guesswork. Change history \| What maintenance, configuration, software or process change preceded the symptom? \| Helps define a useful comparison window and rollback boundary. For sequence recovery logic: failure containment and corrective action planning, the matrix should be completed with evidence from the same event window. Mixing observations from unrelated shifts can create a convincing but false causal story. If timestamps are inconsistent, establish which controller, server or operator record is authoritative before comparing event order. Trend evidence is more useful when the measurement definition remains stable. Record units, sampling interval, filtering, equipment mode and product family. A rising fault count may reflect increased throughput rather than deteriorating equipment, while a stable count can hide deterioration if production volume has fallen.