# Sequence Recovery Logic: Degraded-Mode Operations and Recovery Boundaries - Canonical: https://pearlgateway.net/docs/sequence-recovery-logic-degraded-mode-operations-and-recovery-boundaries/ - Updated: 2026-09-06T12:25:00+00:00 Sequence recovery logic is the control-layer capability that decides how an interrupted material flow is brought back to an orderly, trackable condition before full automatic operation resumes. In a warehouse automation environment, the term covers both the PLC code that reconstructs item order from sensor evidence and the WCS orchestration rules that reconcile logical inventory with physical reality. Degraded-mode operations complicate this task because the system is expected to keep moving product even when some devices are unavailable, manually controlled, or operating at reduced performance. Recovery boundaries mark the limits of what automatic recovery may safely assume and where the system must instead ask for human confirmation. This article describes how those boundaries are understood, measured, and defended. It is written for warehouse operators, maintenance engineers, and controls teams who must keep automated systems honest when the ideal picture of every item on every conveyor is no longer available. ## Defining the Recovery Boundary A recovery boundary is not a physical fence or a painted line on the floor. It is a logical threshold in the control and orchestration layers. Within the boundary, the PLC and WCS have enough reliable sensor data, sequence memory, and interlock state to reconstruct where every unit is and in what order. Outside the boundary, sensor truth is too thin, manual intervention has altered the physical layout, or the time gap between the last known state and the current moment is too large for automatic logic to make safe assumptions. Recovery boundaries exist in two dimensions. The first is spatial: a zone may be protected by dense photoeyes, accumulation sensors, and position encoders, so the logic can confidently say that if a carton is present at position A and not at position B, it must be between them. The second dimension is temporal. A zone that was trustworthy at 14:03 may become untrustworthy at 14:47 if an operator manually removed a jammed parcel and placed it back in a different orientation. Time erodes the accuracy of sequence memory. Recovery logic must therefore be written with explicit rules about how much time may pass before a state is considered stale. Control teams often make the mistake of treating the recovery boundary as a permanent configuration value. In practice, the boundary should be reviewed whenever the physical layout changes, when a sensor is added or removed, when a conveyor segment is replaced, or when the WCS begins tracking a new unit type. The boundary is a governance decision as much as a programming decision. It says: here the machine is allowed to guess, and here the machine must stop and ask. ### Sequence State Versus Machine State Machine state describes the physical condition of a device. A motor is running, a photoeye is blocked, a diverter is extended, a plc state is in fault. Sequence state describes the logical position of a unit within a flow. Two identical cartons side by side may have the same machine state but different sequence states. Sequence recovery logic exists primarily to reconstruct sequence state, not to repair machine state. A conveyor jam that clears itself changes machine state without necessarily harming sequence state. A manual operator who pulls a carton from a queue changes sequence state without changing any machine state. Understanding this distinction is the first step toward interpreting recovery events correctly. ## Operating Context: Normal, Degraded, and Recovery Sequence recovery logic is exercised differently in three operating modes. In normal mode, every sensor is reporting, every drive is healthy, and the WCS and PLC are in close agreement. Recovery logic is mostly dormant, only performing periodic consistency checks and heartbeat monitoring. In degraded mode, some components are out of service, operator overrides are active, or the system is running with reduced speed and bypassed zones. Recovery logic becomes more active because the PLC can no longer rely on every expected presence confirmation. The system may need to infer item position from partial evidence, and the WCS may need to accept delayed or alternative tracking messages. Recovery mode is the third state. It is entered after an unplanned stop, an unexpected loss of tracking continuity, or a manual intervention that the automation did not orchestrate. In recovery mode, the PLC executes a defined sequence that attempts to confirm the presence and order of units within the boundary. If the sequence completes successfully, the system returns to degraded or normal operation without operator entry into the guarded area. If the sequence cannot complete because evidence is insufficient, the system transitions to a hold state and requests operator confirmation through the HMI. Teams should define these three modes explicitly in their controls governance documentation. Vague language such as *the system will try to recover* is not enough. Operators need to know whether a particular alarm is asking them to confirm a carton location, to remove an item, or to authorize the PLC to assume that a zone is empty based on indirect evidence. ## Component Interactions That Shape Recovery Behavior Sequence recovery logic is not a single block of code. It emerges from the interaction of several components, and the interaction itself is where most weaknesses are found. The PLC is the real-time authority. It owns the interlock chain, reads the photoeyes and encoders, and executes the state machine that represents conveyor zones. During degraded-mode operations, the PLC must continue to enforce safe separation between units even when the WCS has not yet acknowledged the new sequence. The PLC also owns the timing parameters that define when a missing sensor change is considered a fault versus when it is merely a delayed product. The WCS is the inventory authority. It tracks each unit as it moves from induction to putaway or from storage to shipping. The WCS receives discrete messages from the PLC when a unit passes a tracking point, and it sends destination commands to the PLC ahead of physical arrival. In degraded mode, the PLC may be unable to send certain messages because a sensor is missing. The WCS must then decide whether to trust the PLC’s alternate evidence or to place the unit in an exception state. This decision is the heart of sequence recovery logic from the orchestration side. Physical sensing devices determine the boundary’s trust radius. A dense row of photoeyes gives the PLC the ability to track items closely. Long gaps between sensors force the logic to rely on timing, which is vulnerable to speed changes and slip on belt conveyors. When a sensor is known to be unreliable, the PLC should be configured to lower its confidence rather than to ignore the sensor entirely. A sensor that is offline must be treated as a missing evidence source, not as a device that has been proven clean. Motor controllers and drives contribute fault states that serve as recovery boundary markers. A conveyor segment that has tripped on overload may still contain product. The sequence logic must respect the drive fault and not attempt to move that segment until the fault is reset and the physical condition is verified. HMI terminals provide the interaction point where operators confirm or reject the PLC’s recovery proposal. Finally, the MES or ERP system sits above the WCS and has no direct role in sequence recovery; it only sees the results after the WCS declares a stable state. ## Symptoms That Point to Recovery Boundary Problems Recovery logic issues rarely produce a single dramatic alarm. More often, they show up as repeated, low-granularity problems that cost time across many shifts. Recognizing these symptoms early shortens troubleshooting and prevents teams from misdirecting effort toward mechanical repairs when the true cause is a poorly defined recovery boundary. - **Repeated item-not-found events.** The WCS expects a unit at a downstream tracking point, but the PLC has never registered its passage. This often means the recovery boundary is too narrow, and the PLC did not have permission to infer item movement across a gap that lacked sensor coverage. - **Ghost inventory in WCS.** The PLC has physically cleared a zone, but the WCS still shows one or more units in that zone. This indicates that the PLC completed a recovery action that the WCS did not acknowledge in the message flow. - **Manual removals creating cascade errors.** When an operator removes a jammed carton, the PLC may or may not be told. If the sequence logic has no formal manual-removal acknowledgment step, the boundary becomes corrupted from that moment onward. - **Self-clearing jams that leave the system locked out.** The physical jam clears itself due to vibration or gravity, but the PLC remains in a fault state because the recovery sequence requires an operator confirmation that no one ever issues. - **Partial restarts requiring multiple resets.** The operator resets a zone, the WCS clears its exception queue, and then the PLC faults again because the sequence state does not match what the WCS reset implied. This mismatch often comes from having different recovery boundaries in the PLC versus the WCS. - **Degraded mode becoming the default.** If the site runs for weeks with bypassed sensors and manual overrides, the recovery logic is permanently operating at its weakest confidence level. The system may recover less often and require more operator intervention, which is acceptable only if the degraded mode was formally declared and understood. When one or more of these symptoms appears consistently, the recovery boundary itself should be the primary suspect. Teams often look first at mechanical components, then at wiring, then at the PLC code. Only after all of those are cleared does someone ask whether the boundary was ever defined correctly for the actual physical layout. ## Evidence Collection Before Touching Logic Before any programmer changes sequence recovery logic, the site must capture evidence from the actual event. Changing logic without this evidence is guesswork that often fixes one zone while breaking another. The evidence collection process should be standardized so that every recovery event produces the same minimal dataset. The PLC state list must be captured at the moment of failure. This includes zone states, current sensor readings, pending messages to the WCS, and the values of any recovery timers. A timestamp must be aligned to the WCS clock. Many sites maintain separate clocks that drift, so a recovery event may appear to have different times in the PLC log and the WCS log. a small NTP alignment step, performed during maintenance windows, is worth the effort. The WCS log should include the unit-level tracking records, including the last known position, the expected destination, and the message sequence numbers. The operator actions surrounding the event must also be documented. What precisely did the operator see on the HMI? Did they acknowledge an alarm? Did they manually move a unit? Did they press a maintenance bypass button? These actions are the largest source of unrecoverable sequence errors, and they must be recorded with timestamps from the HMI itself. Physical evidence also matters. A digital photograph of the affected zone and a note of which units were physically present versus which were expected to be present can resolve disputes between the PLC and WCS. Finally, the speed and timing of the line at the moment of failure should be recorded. A belt conveyor moving at a speed that is faster than the recovery logic timing parameters expects will trip false losses of presence. Teams that capture this dataset routinely will find that the majority of recovery logic failures fall into a small number of repeatable patterns. ## Diagnostic Decision Table for Degraded-Mode Events The following table summarizes common degraded-mode events, whether the situation falls within the recovery boundary, the likely cause category, and the recommended first action. It is not a substitute for OEM-specific diagnostics, but it provides a structured starting point for a controls team. Symptom Observed | Within Recovery Boundary? | Likely Cause Category | First Action WCS shows unit present, PLC shows zone empty | No, if unit was manually moved; yes, if unit passed a sensor gap | Message loss or manual removal without acknowledgment | Confirm physical presence; if no unit exists, issue WCS inventory reconciliation with PLC confirmation PLC shows zone blocked, WCS shows no unit | Likely within boundary | Stale sensor or debris blocking photoeye | Inspect sensor and physical media output; verify sensor readability, not just status Recovery sequence runs but stops at the same position every time | Within boundary | Missing or misaligned tracking point in that segment | Compare PLC tracking map to physical layout; check sensor mapping and message trigger conditions Operator manually removed a jam, then system will not release | Outside boundary after removal | No formal manual-removal acknowledgment step | Use HMI manual-removal confirmation; do not force the PLC state until physical location is verified Auto recovery successfully shifts units downstream, but WCS still has old inventory | Within PLC boundary, outside WCS boundary | WCS did not receive or acknowledge the recovery confirmation message | Review message queue and handshake logic; verify that recovery events generate a return code to WCS Repeated faults only during degraded high-speed operation | Within boundary, but timing margin is too small | Speed exceeds recovery timing assumptions | Check speed profile against PLC timer values; either slow the degraded speed or expand timing tolerance with engineering approval The table’s first column describes what the operator or maintenance technician sees. The second column places the event in relation to the recovery boundary. The distinction is crucial because it tells the team whether the fix belongs in the logic or in the physical process. ## Common Interpretation Errors in Sequence Recovery Teams that work under time pressure tend to interpret ambiguous recovery events in predictable and often incorrect ways. The most common error is to blame the PLC when the WCS is wrong, or the reverse. A productive discussion requires the team to establish which component has the better evidence at the moment of the fault, not which component has a higher priority in the architecture. Another frequent error is treating every sensor loss as a product loss. When a photoeye fails in the blocked state, the PLC sees a steady presence signal. If the logic is poorly written, it may interpret this as a permanent jam and continue to hold the zone even after the physical conveyor is clear. The recovery boundary should include a rule that distinguishes a sensor fault from a true presence signal, usually through a combination of upstream and downstream sensor correlation. Teams also err by assuming that a recovery boundary is a hard stop. A boundary is not a wall. It is a confidence threshold. Within the boundary, the logic may still recover automatically. Outside the boundary, it must stop and ask. Conf ## Related Pearl Gateway Guides - [OPC UA Integration: Operating Principles and System Boundaries](/docs/opc-ua-integration-operating-principles-and-system-boundaries/) - [Message Broker Interfaces: Operating Principles and System Boundaries](/docs/message-broker-interfaces-operating-principles-and-system-boundaries/) - [PLC Conveyor State Machines: Inspection Points and Early Warning Signs](/docs/plc-conveyor-state-machines-inspection-points-and-early-warning-signs/) ## Site-Specific Review Worksheet This educational worksheet supports a structured review of sequence recovery logic: degraded-mode operations and recovery boundaries. Begin by identifying the equipment boundary, control ownership, operating modes, material characteristics, upstream dependencies and downstream consequences. Record what the system is expected to do, what was actually observed and which evidence is time-aligned. Avoid changing several variables at once, because simultaneous changes make cause and effect difficult to establish. ### Evidence to collect - Operating mode, active mission or route, and the exact sequence state. - Alarm history, device state changes and controller timestamps. - Physical observations such as alignment, contamination, wear, obstruction and load condition. - Recent maintenance, software changes, parameter changes and recurring work orders. - Upstream and downstream readiness, including blocked, starved and unavailable conditions. ### Decision boundaries Use approved site procedures and competent engineering judgment before intervention. General information in the Controls, PLC & WCS Integration library cannot determine whether a specific machine is safe to enter, restart or modify. Preserve original settings, document authorized adjustments and establish a rollback point before controlled testing. When evidence conflicts, stop and resolve the timestamp, naming or measurement discrepancy before drawing a conclusion. ### Closeout record A useful closeout record states the symptom, confirmed cause, evidence, corrective action, validation method, residual risk and follow-up owner. It should also identify whether the event exposed a design weakness, maintenance gap, training issue, spare-parts issue or monitoring blind spot. This turns a single recovery into reusable reliability knowledge without treating one observation as universal. ## Evidence Matrix for Operational Review Evidence group | Questions to answer | Why it matters Sequence state | What mode, step, mission and interlock state were active? | Separates a physical problem from an expected control hold. Material condition | Were load dimensions, orientation, stability and spacing within the intended envelope? | Explains faults that appear random when only controller data is reviewed. Device evidence | Which inputs changed, in what order, and against which timestamp? | Supports repeatable diagnosis instead of component substitution by guesswork. Change history | What maintenance, configuration, software or process change preceded the symptom? | Helps define a useful comparison window and rollback boundary. For sequence recovery logic: degraded-mode operations and recovery boundaries, the matrix should be completed with evidence from the same event window. Mixing observations from unrelated shifts can create a convincing but false causal story. If timestamps are inconsistent, establish which controller, server or operator record is authoritative before comparing event order. Trend evidence is more useful when the measurement definition remains stable. Record units, sampling interval, filtering, equipment mode and product family. A rising fault count may reflect increased throughput rather than deteriorating equipment, while a stable count can hide deterioration if production volume has fallen.