# Heartbeat and Watchdog Logic: Failure Containment and Corrective Action Planning - Canonical: https://pearlgateway.net/docs/heartbeat-and-watchdog-logic-failure-containment-and-corrective-action-planning/ - Updated: 2026-09-02T06:42:00+00:00 A heartbeat or watchdog signal in a warehouse controls environment is far more than a blinking tag on an HMI. It is a structured supervisory dialogue between the warehouse control system (WCS) and the programmable logic controllers (PLCs) that move conveyors, shuttles, cranes, and sorters. When implemented deliberately, heartbeat and watchdog logic provides early failure containment: the equipment stops in a known state instead of running blind while the WCS is stalled. It also creates a framework for corrective action planning, because the failure mode, the evidence trail, and the ownership of the fix all become traceable. This article explains how those mechanisms behave in a real distribution center, what the symptoms mean, and how controls and maintenance teams should interpret them before modifying code or scheduling downtime. ## The Operating Context: Why Warehouse Controls Need a Supervisory Pulse A typical automated warehouse has a WCS issuing high-level commands: release a carton from induction, advance a pallet to a stretch wrapper, move a shuttle to a storage lane. Each command is executed by local PLC logic that manages sensors, motors, and zone interlocks. The two systems must agree on a simple fact: *is the other side still thinking?* The heartbeat answers that question on a continuous, low-latency basis. Without a heartbeat, the PLC cannot distinguish between “the WCS has no new commands” and “the WCS has crashed.” If the WCS stops mid-transaction and the PLC continues running the last command stream, a downstream merge can accept a load without a release, a crane can index to a lane that was never assigned, or a sorter can close a divert before the WCS knows the carton has arrived. The watchdog exists to convert an ambiguous silence into a deterministic state transition. It is important to note what heartbeat logic is not. It is not a safety-rated function, and it does not replace guarding, light curtains, or safety-rated speed monitoring. It is a control-supervision mechanism. Any interaction with safety-related devices must follow site procedures, lockout requirements, and OEM documentation, and must only be performed by staff who are competent and authorized to do so. ## Where the Watchdog Logic Actually Lives A complete watchdog scheme spans three layers. On the WCS side, a periodic task writes a heartbeat value, usually a monotonically increasing counter or a UTC timestamp, into a tag in the PLC. On the PLC side, a dedicated logic routine compares the current wall-clock time or the incoming counter against the last received value. If the delta exceeds the configured watchdog timeout, the PLC enters a predefined hold state. On the same PLC side, if the PLC scan is healthy and the logic is executing, the PLC typically toggles a second heartbeat tag back to the WCS so that the WCS also knows the controller is alive. This bidirectional exchange is critical. A unidirectional heartbeat only confirms that the WCS can write; it says nothing about whether the PLC can read, run logic, or control outputs. In practice, many sites implement a “link running” bit on the PLC side and a “command active” acknowledgement on the WCS side. These are not redundant. They verify different layers of the same path: communication transport and application execution. The network between the two layers matters as much as the logic. The heartbeat may cross a managed Ethernet switch, a virtual LAN segment, or a wireless bridge to a shuttle. Any of those components can delay, drop, or corrupt a heartbeat message. A healthy watchdog design therefore logs more than the final timeout. It logs timestamps, sequence numbers, and the last known state of both sides. ## The Message Flow: From WCS Cycle to PLC State Transition Consider a typical carton conveyor zone controlled by a PLC that communicates with the WCS over Ethernet/IP. The WCS runs a 500 ms cycle. At the start of each cycle, it writes a 32-bit unsigned integer to a tag named *WCS_HB_COUNT*. The PLC reads that value in a quick peek inside its main scan, compares it to a stored copy in an internal register, and updates another tag, *PLC_HB_STATUS*, to true. If the difference between the current WCS counter and the stored counter is exactly one, the PLC advances its stored copy and considers the communication healthy. If the WCS counter jumps by more than one, the PLC recognizes that messages may have been missed, but the link may still be operational. If the counter does not change for the configured timeout, the PLC sets *PLC_HB_STATUS* to false, clears all software permits that allow motion, and holds all active zones in their current state. It does not simply stop energizing outputs; it latches a fault code that maintenance can read from the HMI. The WCS side performs a mirror test on the PLC’s return heartbeat. The sequence numbers matter. A toggle bit that simply flips between zero and one can be fooled by a single duplicated message. A counter is monotonically increasing and makes it obvious when messages have been lost or duplicated. Using the WCS cycle counter for command sequencing has an additional benefit: during an incident, engineers can compare the WCS’s last written counter against the PLC’s last stored counter to see exactly how many cycles were lost before the timeout was triggered. ### Timing Values Are a Design Decision There is no universal heartbeat interval. A 250 ms interval might be appropriate for a high-speed sorter, while a 2-second interval is acceptable for a slower pallet conveyor. The timeout must be longer than the worst-case normal network jitter, but short enough to stop motion before a jam causes damage. The right values come from measuring normal operation over a full shift, not from guessing. When a timeout is too aggressive, the system becomes a source of nuisance faults; when it is too loose, the system fails to contain damage. ## Symptom Patterns: Slow Loss, Fast Flapping, and Stale-but-Alive Watchdog faults do not all look the same, and the different looks point to different root causes. **Slow loss** describes a heartbeat that gradually drifts out of tolerance. The WCS may be writing every 500 ms for an hour, then the interval stretches to 600 ms, then 900 ms, and finally a fault occurs. This pattern often indicates scheduler starvation on the WCS server, such as a database query or a garbage collection routine that periodically steals the process’s CPU time. It may also indicate a network switch with a slowly filling buffer. **Fast flapping** is a heartbeat that fails for one or two cycles, recovers, then fails again minutes later. The PLC may enter a fault and immediately reset when the next heartbeat arrives. This pattern is frequently caused by marginal cabling, a failing network switch port, or a PLC communication module that resets under thermal or electrical noise. **Stale-but-alive** is the subtlest pattern. The WCS heartbeat counter is advancing, and the PLC heartbeat is returning, but the equipment is not executing commands. This may look like a watchdog fault to operators, but it is usually not a communication failure at all. The PLC logic may be stuck in a state machine step, a zone may have a sensor that never deasserted, or a maintenance override bit may be preventing the PLC from honoring releases. The heartbeat says both systems are breathing; it does not say they are acting correctly. ## Diagnostic and Decision Table for Watchdog Faults The table below summarizes the common heartbeat fault patterns, the evidence an engineer should expect to see, and the first decision to make before changing anything. Symptom Pattern | Evidence Signatures | Likely Domain | First Decision / Action Heartbeat timeout on one PLC only | WCS log shows successful writes, but PLC last-seen timestamp is frozen; other PLCs remain healthy | Network path to that PLC, PLC communication module, or PLC logic task hold | Inspect the specific port, cable, and PLC diagnostic counters before modifying WCS logic Simultaneous timeout across all PLCs | All last-seen timestamps stop at the same wall-clock time; WCS still responds to pings | WCS scheduler stall, database connection loss, or virtual machine host pause | Capture WCS thread dumps and database session state before restarting the WCS Heartbeat advances but no motion is commanded | Both heartbeat counters are healthy; zones remain in hold or commands remain pending | PLC state machine, interlock chain, or output inhibit logic | Read PLC status registers and trace the state machine logic path step by step PLC reports “WCS dead” while WCS reports “PLC dead” | Both directions missing; timestamp delta is large and identical on both sides | Switch, VLAN routing, time skew, IP address conflict, or concurrent media failure | Verify port statistics, duplicate address detection, and raw-socket reachability Intermittent flapping at a regular interval | Timeouts correlate with switch CRC error counters or WCS CPU busy graphs | Marginal termination, faulty media converter, or periodic resource contention | Schedule a controlled cable replacement or resource reallocation during planned maintenance ## Evidence Collection: What to Capture Before Changing Anything The biggest mistake in watchdog-fault investigation is restarting the WCS, resetting the PLC, or replacing hardware before the evidence has been preserved. The fault is the incident; the logs are the witness. A structured evidence snapshot should include the following: - The exact wall-clock time of the first fault and the last healthy heartbeat before it. - The WCS heartbeat count and the PLC last-stored count at the time of the fault, both captured from their respective tags. - The WCS scheduler log, including and CPU utilization or thread dump if available, for a window around the failure. - The PLC fault log, communications diagnostic counters, and any time-synchronization offset between the PLC clock and the WCS clock. - A list of every software change, configuration change, or firmware update made in the previous 24 hours on either side. - Network switch port statistics for the affected devices, including errors, discards, and port resets. - Screenshots are not enough. Export raw CSV or text logs with millisecond resolution wherever possible. Evidence must be labeled with the event identifier and a timestamp range, typically 30 minutes before and 30 minutes after the event. This practice lets multiple engineers review the same dataset and prevents the common error of comparing log excerpts from different time bases. If the site’s clocks are not synchronized, that is itself a finding; a sixteen-second offset between the WCS server and the PLC clock will produce misleading timestamps in every downstream analysis. ## Interpretation Errors That Mimic Real Watchdog Faults A heartbeat is a simple signal, but its interpretation is remarkably error-prone. One common error is assuming the network is healthy because a ping succeeds. ICMP reachability proves that IP packets can traverse the network, but it says nothing about the WCS application’s ability to write a tag or the PLC’s ability to read it. A busy PLC with an extended scan time can still respond to ping but may be too late to consume the WCS tag inside its communication window. A second error is treating the heartbeat as proof of correct equipment behavior. A PLC that is alive and receiving heartbeats may still be holding a zone in fault, or its outputs may be disabled while the logic scans normally. When a watchdog fault is accompanied by a physical jam, the jam is not necessarily the cause. The sequence of events must be established: did the heartbeat fail first and stop the conveyor, or did the conveyor jam cause the PLC scan to overrun and miss the heartbeat? The answer determines whether the corrective action is a mechanical fix or a controls modification. A third error is changing the timeout to eliminate nuisance faults without understanding why the fault occurred. Widening the timeout treats the symptom and delays detection. If the WCS scheduler is occasionally starving for 2 seconds, then widening the timeout to 10 seconds hides a performance problem that will eventually cause the WCS to lose track of active loads. The timeout should only be changed after the nominal performance envelope has been measured and the change has been tested in a controlled manner. Finally, a failed watchdog event must not be assigned to the most convenient suspect. The ## Related Pearl Gateway Guides - [Interlock Management: Common Failure Modes and Diagnostic Evidence](/docs/interlock-management-common-failure-modes-and-diagnostic-evidence/) - [Sequence Recovery Logic: Common Failure Modes and Diagnostic Evidence](/docs/sequence-recovery-logic-common-failure-modes-and-diagnostic-evidence/) - [Control-System Time Synchronization: Common Failure Modes and Diagnostic Evidence](/docs/control-system-time-synchronization-common-failure-modes-and-diagnostic-evidence/)