Heartbeat and Watchdog Logic: Degraded-Mode Operations and Recovery Boundaries #
In automated warehouses, the programmable logic controller (PLC) and the warehouse control system (WCS) are not one program. They are two independent machines that must agree on what is happening, what has already happened, and what should happen next. Heartbeat and watchdog logic is the mechanism that keeps that agreement honest. A heartbeat is a periodic signal that says, in effect, “I am still running.” A watchdog is the complementary logic that notices when that signal stops arriving. Neither proves that the other system is correct. Both prove only that the other system is still alive at the transport level. This distinction matters enormously when operations move into degraded mode. Degraded mode is not a failure state to be feared; it is a designed operating envelope for the period after communication is lost and before a controlled recovery is confirmed. The real risk lies in not knowing where that envelope ends.
This article discusses how heartbeat and watchdog logic behaves across PLC and WCS boundaries, what it can and cannot prove, how to interpret the symptoms of a missed beat, and where the recovery boundary sits between automatic resynchronization and mandatory human intervention. It is written for warehouse operators, maintenance engineers, and controls teams who must keep material moving without confusing liveness with correctness.
Normal Operation: The Implicit Contract Between PLC and WCS #
In a typical conveyor or automated storage and retrieval system (AS/RS) integration, the PLC is responsible for real-time physical control: stopping a carriage, verifying a photo-eye, driving a motor, and locking a transfer deck. The WCS is responsible for order-level decisions: which destination a carton should take, which order is due next, which storage slot to assign, and which two loads may simultaneously enter a junction. Neither system can perform its role without the other. The heartbeat is a low-level, high-cadence message that ensures the two systems remain logically present to each other.
Two common heartbeat directions exist. In a unidirectional design, the PLC writes a “status alive” counter or a timestamped word to a shared tag that the WCS reads over a network protocol such as OPC UA, EtherNet/IP, or a TCP socket. In a bidirectional design, the WCS echoes an acknowledgment, creating a closed loop. Bidirectional heartbeats are more expensive to configure but they allow both parties to detect stale communication. Some systems use a sawtooth counter rather than a timestamp, because a counter cannot be accidentally faked by a stale clock.
Normal operation also includes a message-flow contract. The WCS sends a destination request, the PLC accepts it, executes movement, and sends a completion event. The heartbeat rides alongside but does not replace this transactional flow. If the heartbeat fails while no material movement is active, the PLC may continue in a neutral state. If the heartbeat fails during an active transfer, the PLC must decide within one scan or one timeout window whether to continue to a safe stopping point, hold a carriage in place, or reverse a decision already taken. These decisions should be engineered in advance, not discovered on the day of the fault.
What a Watchdog Actually Monitors #
It is tempting to call a watchdog a “health monitor.” More precisely, a watchdog monitors liveness of a partner by measuring the absence of expected activity against a timeout. It does not monitor database integrity, logic correctness, or order state. A WCS can be fully alive, responding to every heartbeat, while its order queue points twenty cartons to a wall. A PLC can be fully alive while a sensor mapping is misconfigured. Heartbeat logic simply keeps the two applications present to each other; the rest of the diagnostic stack must be handled separately.
In a typical installation, the watchdog lives in three places. The PLC has a timer that is reset each time it receives an expected heartbeat message from the WCS. If the timer expires, the PLC sets a “WCS Comm Lost” bit and enters a degraded state. The WCS runs a polling routine that expects a response from the PLC within a certain number of milliseconds. If no response arrives, the WCS flags the PLC as offline or “non-responsive.” A network diagnostic tool may provide a third watchdog at the infrastructure level, but it only proves that packets move, not that the PLC application is scanning or the WCS service is processing.
The component interaction is important during root-cause analysis. A missed beat can originate in the PLC task timing (the CPU is too busy to scan the communication block), in the WCS service thread (a backlogged database query blocks the heartbeat routine), in the network (a saturated switch drops a packet), or in the application itself (the heartbeat writer is placed after a blocking instruction). Because the symptom is exactly the same in all four cases — a missed beat — the watchdog is only the beginning of diagnostics, never the end.
Degraded-Mode Operations: How the System Should Behave When the Heartbeat Stops #
Degraded mode is the set of deliberate behaviors that the PLC and the WCS execute after a heartbeat times out. It is not a chaotic shutdown. It is a narrower operating condition intended to protect personnel, protect product, and preserve the ability to recover without manual re-counting from scratch. There is a common false belief that the safest response is to stop everything immediately. Stopping everything can be more dangerous than continuing in a controlled manner because it may leave a load suspended on a transfer shuttle or at an unguarded gap. The correct response is defined by the risk of the physical motion and the state of the transaction, not by the communication status alone.
PLC-Side Degraded State #
When the PLC detects the loss of the WCS heartbeat, it should enter a state whose rules are explicit. For example, the PLC may allow a partially completed transfer to finish in its last known safe path, but it must refuse any new destination request. It should drive all controlled axes to a static, held condition where the motion is stopped but power is not necessarily removed, because removing power can drop the holding brakes on a lift in an uncontrolled manner. The PLC should blink or display a distinct alarm code that indicates “degraded due to WCS comm loss” rather than a generic fault, so that other systems and technicians do not interpret the state as a normal cycle pause.
WCS-Side Degraded State #
The WCS, on losing the PLC heartbeat, should stop issuing new assignments to that controller or to the zones that depend on it. It should mark the affected equipment as “comm unavailable” in its equipment status table, quarantine all in-flight orders in that zone, and continue to serve other zones served by other controllers if they are not logically dependent on the lost controller. The WCS should also set an interface connection state that prevents its connection pool from repeatedly hammering the PLC with retries, which can make the fault worse. Many WCS packages implement a “circuit breaker” pattern for interface links; once the miss count exceeds a threshold, the WCS stops sending and enters a backoff state.
The key to a defensible degraded mode is that both sides have well-defined entry and exit criteria. Entry criteria are met when the timeout expires. Exit criteria are met when a successful synchronized handshake is completed during recovery. Without explicit exit criteria, a connection may come back, but the two systems may have four different snapshots of the same zone inventory.
Recovery Boundaries: Where Automatic Resynchronization Ends #
The recovery boundary is the point beyond which the PLC and WCS are no longer allowed to guess. Before that boundary, automatic logic may reconcile the two sides. After that boundary, a human must physically or logically verify the state of the system and issue a deliberate acknowledgement.
There are three typical recovery boundary tiers. The first tier is the time-based boundary. If the heartbeat misses were brief (for example, less than five seconds) and no state-changing events occurred on either side during that window, the systems may automatically return to normal operation on the next successfully validated heartbeat exchange. The second tier is the event-based boundary. If the miss occurred while a load was transferable, the PLC and WCS must compare the last completed event IDs. If the event IDs match exactly, automatic recovery is acceptable. If they do not match, a human must inspect the physical position of each involved load before recovery. The third tier is the context boundary. If the PLC was restarted, if the WCS service was restarted, or if both lost power, automatic recovery is not permitted because counters and sequence numbers cannot be trusted. The time-based boundary is driven by clocks; the event-based boundary is driven by transaction state; the context boundary is driven by known loss of memory or loss of application state.
Boundaries should be implemented in the logic, not only in the operator procedures. The PLC should have a state variable that records whether a manual reconciliation is required before normal operation can resume. The WCS should present that state in its user interface and disable any “start all” button that would override it. The boundary must be hard-coded where the risk is high, such as the moment of resuming an interrupted transfer on a carousel or a cross-belt sorter.
Symptoms and First Evidence Collection #
Diagnosis begins with a disciplined evidence collection routine. The most common error during heartbeat faults is to reset or restart immediately, which destroys the very evidence needed to avoid a repeat fault. The table below presents the typical symptoms and the evidence items that should be collected before any reset action.
| Observed Symptom | Likely Recovery Boundary | Evidence to Collect | Initial Responsible Party |
|---|---|---|---|
| PLC display shows “WCS Comms Timeout” and equipment holds in place. | Time-based boundary (if no load in transfer); event-based if load present. | PLC alarm timestamp, heartbeat counter value, last accepted WCS command ID, load position states, WCS event log from the same minute. | Controls technician or shift maintenance. |
| WCS user interface marks a PLC as “offline” while the PLC appears healthy on its own display. | Context boundary; check whether the PLC restarted or the network switch rebooted. | PLC uptime counters, network switch port logs, WCS connection retry log, and both application logs including the last five pre-fault messages. | WCS administrator or integrator support. |
| Heartbeat returns but the WCS refuses to issue new jobs; supervisors see “system not ready for auto.” | Context or event boundary not yet cleared; manual reconciliation required. | The recovery handshake messages, sequence number gap values, and the list of in-flight orders at the time of the loss. | Controls and operations supervisors together. |
| Intermittent miss events occur every few hours with no load impact. | Time-based boundaries are being crossed repeatedly; underlying cause is likely latency or resource load, not a single hardware fault. | Round-trip latency statistics, historical message timestamps, CPU utilization of the PLC and WCS server at the time of each miss, and packet captures if available. | Controls engineer with network support. |
Evidence collection should be documented in an incident log that records the date, time source, the affected equipment, the sequence number or counter value of the last good message, and the first bad or missing expected message. The value of this evidence decays quickly; logs rotate, memory buffers are overwritten, and critical network captures are lost when interfaces are disabled. Make the first priority the preservation of all logs and trace files, and make the second priority the resumption of service, not the other way around.
Interpretation Errors That Lead to False Conclusions #
Even with good evidence, interpretation mistakes are routine. The most common mistake is equating an active heartbeat with system health. A heartbeat confirms only that both processes are alive. It says nothing about whether the PLC and WCS share the same inventory snapshot. This can be observed when the WCS shows a conveyor zone as “clear” while the PLC’s sensor state says the zone is “occupied.” Both systems are alive, both are responding to heartbeats, yet the control decision is being made on divergent data. The heartbeat should be treated as the login handshake of the control conversation, not as the full conversation itself.
A second mistake is assuming the timeout value is the root cause. A shorter timeout produces more frequent fault declarations; a longer timeout produces rarer faults with larger hidden periods. Neither is wrong per se; the timeout must be matched to the worst-case response time of both systems under maximum load. If the PLC has a 500 ms scan time at full capacity and the WCS has an occasional 800 ms database lag, a 1000 ms watchdog will fault only at the worst moment. Some teams respond by raising the timeout to 5000 ms, masking a real capacity deficit. Better to measure the response-time distribution under peak load and set the timeout to the 99th percentile plus a small margin, then give the interface hardware its own bandwidth reservation.
A third mistake is the belief that resetting the connection is a valid recovery action. Resetting the connection clears the socket, but it does not re-synchronize the state. After a reset, the PLC and WCS continue from whatever local memory they retained. If the PLC has already completed the last movement but the WCS still expects a completion event, the WCS will wait forever, and the operator will see a “ghost wait” condition. Resetting is only valid as part of a deterministic reconciliation routine that includes a state handshake. A fourth mistake is treating the watchdog as a safety function. Heartbeat logic is a control and communication diagnostic. It is not a replacement for safety-rated interlocks, light curtains, or emergency stop circuits. A watchdog may be used to invoke a safe state only when that safe state has been independently designed and verified under the site’s safety procedures. Site procedures, lockout requirements, OEM documentation, and competent engineering judgment always take priority over any generic recommendation.
Maintenance Implications and Lifecycle Care #
Heartbeat and watchdog logic requires deliberate maintenance, not just passive monitoring. In the lifecycle of a warehouse system, the heartbeat code is often written once and then forgotten. That is a mistake. The logic should be version-controlled just like any other part of the control application, and it should be included in pre-planned change reviews when network topology changes, when the WCS server is upgraded, or when the PLC program is modified to add a new zone. A change in the message scheduling, such as the addition of a new timer interrupt or a new remote I/O block, can alter the response time of the communication task and push a previously stable system past its timeout threshold.
Time synchronization is a maintenance task that is almost always underestimated. A PLC and a WCS that compare timestamps must share a common time source. Without Network Time Protocol (NTP) or an equivalent, the two logs may disagree by seconds, and the evidence collected during a heartbeat
