Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover

Key takeaways #

• Test containment is non-negotiable: Every failure-injection test must be physically and logically isolated so that a fault introduced into one zone cannot propagate to live operations, adjacent machinery, or the enterprise network.
• Expected state must be defined before injection: You cannot validate a fault response unless you have pre-declared the exact state the system should reach—including PLC outputs, conveyor speeds, and operator alarms—within a specified time bound.
• Abort criteria protect people and equipment: Each test plan must list explicit conditions under which the test is immediately halted and the system returned to a safe configuration, independent of the fault being studied.
• Evidence is the deliverable: Time-synchronized logs, network captures, and PLC trace data are the only acceptable proof that a failure-injection test actually exercised the intended fault path and produced the expected state.
• Restoration is a scripted procedure, not an afterthought: A documented, tested restoration sequence must be executed after every fault injection to return the system to normal operation without residual errors or hidden state.
• FIT complements, not replaces, standard commissioning: Factory acceptance testing and site acceptance testing verify nominal function; failure-injection testing verifies the degraded and faulted modes that those tests deliberately avoid.

Definition and scope of failure-injection testing #

Failure-injection testing is a verification method in which a fault is deliberately introduced into a system to observe its behavior under conditions that are not part of normal operation. In warehouse automation, the system under test typically includes programmable logic controllers (PLCs), conveyor drives, palletizers, stretch wrappers, automated storage and retrieval systems (AS/RS), and the industrial network that interconnects them. The fault can be a dropped network packet, a delayed response, a corrupted data value, a disconnected sensor, or a power interruption to a subsystem.

The scope of FIT is distinct from both functional testing and load testing. Functional testing verifies that the system performs its intended functions under nominal conditions. Load testing verifies performance under expected peak demand. FIT verifies the system’s behavior when something goes wrong—specifically, whether the system detects the fault, transitions to a safe state, and can be recovered without unsafe manual intervention. This is particularly important in automated warehouses where high-speed sortation systems and robotic palletizers can cause significant damage or injury if they fail in an uncontrolled manner.

FIT is not a single test but a campaign of tests, each targeting a specific fault mode. A complete campaign might include dozens of individual injections across different subsystems. The value of the campaign is not in the number of tests but in the rigor of the engineering controls applied to each test. Without these controls, a fault-injection test can easily become an uncontrolled incident that damages equipment, corrupts data, or injures personnel.

Test containment: physical and logical isolation #

Test containment is the set of measures that ensure a fault injected during a test cannot affect anything outside the defined test boundary. This is the first and most important engineering control because it determines whether the test is safe to perform at all. Containment has two dimensions: physical and logical.

Physical containment refers to the spatial and mechanical isolation of the equipment under test. In a warehouse environment, this typically means that the zone being tested is fenced off, interlocked, and cleared of personnel. The lockout/tagout procedures described in OSHA 29 CFR 1910.147 provide the regulatory framework for ensuring that hazardous energy is controlled during maintenance and testing activities [S4]. While FIT is not maintenance per se, the same principles apply: the test zone must be isolated from energy sources that could cause unexpected motion, and personnel must be protected from stored energy in accumulators, springs, or raised loads.

Logical containment refers to the isolation of the control system and network. A fault injected into a PLC’s communication port must not propagate to other PLCs, to the warehouse management system (WMS), or to the enterprise network. This is achieved through network segmentation, which is a core recommendation of the NIST Guide to Operational Technology Security [S2]. The guide emphasizes that OT networks should be segmented from enterprise networks and that internal segmentation should limit the blast radius of any single failure. For FIT, this means the test zone should have its own network segment, with managed switches configured to prevent broadcast storms or error frames from crossing into other segments.

Pearl Gateway editorial recommendation: For most warehouse automation systems, we recommend that FIT be performed on a dedicated test rig or on a production zone that has been fully isolated from adjacent zones by both physical interlocks and network-level access control lists. If the test must be performed on a live production line, the containment boundary must be documented in a signed test permit that names the specific equipment, network segments, and personnel involved.

The containment plan must also address time. A fault that persists for too long can cause thermal overload, mechanical stress, or data corruption even if it is logically contained. Therefore, the containment plan must specify the maximum fault duration and the mechanism by which the fault is automatically removed if the test engineer fails to do so manually.

Expected state: defining the target before injection #

Before any fault is injected, the test engineer must write down the expected state of the system after the fault is applied. This is not a vague description such as “the system should stop.” It is a precise, measurable specification of the system’s behavior, including:

• The exact state of each affected PLC program (e.g., which routine is active, which outputs are energized or de-energized).
• The position and velocity of each affected actuator (e.g., conveyor speed in meters per second, robot arm joint angles in degrees).
• The status of each safety device (e.g., light curtain state, emergency stop state, interlock state).
• The content of operator-facing alarms or messages, including the exact text and the time at which they appear.
• The state of data being processed (e.g., whether a pallet record is marked as “incomplete” or “held”).

The expected state must be defined with time bounds. For example, “the conveyor drive must decelerate to zero speed within 2.0 seconds of the fault injection” is a testable statement. “The conveyor must stop” is not, because it does not specify how quickly or what “stopped” means in terms of measurable speed.

The expected state definition serves two purposes. First, it forces the test engineer to think through the system’s fault logic before the test, which often reveals design flaws that would otherwise go unnoticed. Second, it provides the basis for pass/fail evaluation after the test. If the system reaches the expected state within the specified time bounds, the test passes. If it does not, the test fails and the fault-handling logic must be corrected.

It is important to distinguish between the expected state and the desired state. The expected state is what the system actually does in response to the fault, based on its current programming. The desired state is what the system should do, based on the safety and operational requirements. If these two differ, the test has revealed a defect. For example, if the expected state is “the palletizer continues to cycle with a missing pallet presence signal” and the desired state is “the palletizer halts and alarms,” then the test has identified a safety-relevant defect that must be corrected before the system is released for production.

Abort criteria: knowing when to stop the test #

Abort criteria are the pre-defined conditions under which a failure-injection test is immediately halted, regardless of whether the fault being studied has been fully exercised. These criteria are essential for protecting personnel, equipment, and data. They are distinct from the expected state because they are not about the system under test—they are about the test itself.

Abort criteria fall into three categories:

1. Safety-related abort criteria: Any condition that poses an immediate risk to personnel or equipment. Examples include unexpected motion of a robot arm, a conveyor starting when it should be stopped, a fire or smoke alarm, or a safety interlock failing to engage. These criteria are absolute and require immediate manual intervention, including emergency stop activation and power removal.
2. Containment-related abort criteria: Any condition that indicates the fault has escaped the test boundary. Examples include network traffic appearing on a segment that should be isolated, a PLC in an adjacent zone going into fault, or the WMS receiving corrupted data. These criteria require immediate termination of the test and a full assessment of the containment breach.
3. Data-integrity-related abort criteria: Any condition that indicates data corruption or loss that cannot be reversed. Examples include a database transaction being committed with incomplete data, a pallet record being duplicated, or a PLC memory area being overwritten with invalid values. These criteria require immediate test termination and a data-recovery assessment.

Each abort criterion must be written in a test plan with a specific, observable trigger. For example, “Abort if the conveyor speed exceeds 0.5 m/s while the stop command is active” is a valid criterion. “Abort if something looks wrong” is not.

The abort criteria must be communicated to all test participants before the test begins, and a designated test director must have the authority to execute an abort without seeking further approval. In practice, this means the test director has a physical emergency stop button and a network-level kill switch that can be activated from a single location.

Pearl Gateway editorial recommendation: We recommend that every FIT test plan include at least three abort criteria, one from each category above. If a test plan cannot identify a plausible safety-related abort criterion, the test is probably not exercising a meaningful fault path and should be redesigned.

Evidence collection: proving the test happened and what occurred #

Failure-injection testing produces valid results only if the test engineer can prove, after the fact, that the fault was injected at a specific time, that the system responded in a specific way, and that the response was causally linked to the fault. This requires time-synchronized evidence from multiple sources.

The minimum evidence set for a FIT test includes:

• Fault injection log: A timestamped record of the exact fault that was injected, the method of injection (e.g., network impairment device, physical disconnection, software fault injection), and the duration of the injection.
• PLC trace data: A timestamped record of the PLC program’s execution, including the values of key tags, the state of the program routines, and the timing of output changes. Most modern PLCs support built-in trace or logging functions that can capture this data at scan-rate resolution.
• Network capture: A timestamped packet capture from the network segment under test, showing the traffic before, during, and after the fault injection. This is essential for verifying that the fault was actually applied to the network and for detecting any unexpected traffic.
• Operator interface screenshots or logs: A record of what the operator saw, including alarm messages, status indicators, and any manual actions taken.
• Video recording: A time-synchronized video of the physical equipment, which is often the only way to verify mechanical behavior such as deceleration profiles or actuator positions.

All evidence must be synchronized to a common time reference. In practice, this means using a network time protocol (NTP) server or a dedicated time-synchronization system for all test instruments. The NIST Engineering Statistics Handbook notes that time-series analysis requires consistent time bases to draw valid conclusions [S3]. Without time synchronization, it is impossible to determine whether the PLC response occurred 100 milliseconds or 100 seconds after the fault injection.

The evidence must be preserved in a format that is readable by the engineering team and, if necessary, by third-party auditors. This typically means exporting logs to CSV or JSON format, saving packet captures in PCAP format, and storing video in a standard codec. The evidence should be archived with the test report and retained for the life of the system, as it may be needed for incident investigation or system upgrades.

Restoration: returning the system to a known-good state #

Restoration is the process of returning the system under test to its normal, pre-test configuration after a failure-injection test has been completed. This is a critical engineering control that is often neglected, leading to systems that operate with residual faults or corrupted state long after the test has ended.

Restoration must be a scripted procedure, not an ad-hoc sequence of actions. The restoration script must specify, in order:

1. Fault removal: The physical or logical removal of the injected fault. For example, reconnecting a disconnected cable, removing a network impairment device, or disabling a software fault-injection hook.
2. State reset: The resetting of all PLC programs, drives, and sensors to their initial state. This may involve a full PLC restart, a warm restart, or a targeted reset of specific tags.
3. Data verification: The verification that all data in the WMS, PLC memory, and databases is consistent and uncorrupted. This may involve comparing record counts, checksums, or transaction logs.
4. Functional check: A brief, scripted functional test to verify that the system operates normally after restoration. This is not a full functional test—it is a smoke test to confirm that the system is ready for production.
5. Documentation: The recording of the restoration actions, the time of restoration, and the results of the functional check in the test report.

The restoration procedure must be tested before the FIT campaign begins. This is typically done by performing a dry run in which the fault is injected and immediately removed, without allowing the system to reach its expected state. This validates that the restoration script is correct and that the test team can execute it under time pressure.

Restoration is particularly important for systems that use OPC UA for data exchange, as described in the OPC Foundation’s OPC UA specification [S5]. OPC UA sessions may need to be re-established, subscriptions may need to be re-created, and historical data buffers may need to be re-synchronized. The restoration script must account for these application-layer states, not just the network-layer connectivity.

Fault taxonomy for warehouse automation #

A useful failure-injection campaign is organized around a fault taxonomy that covers the major failure modes of warehouse automation systems. The taxonomy below is a Pearl Gateway editorial recommendation, derived from common failure modes observed in material-handling systems and from the operational technology security guidance in NIST SP 800-82 Rev. 3 [S2].

Each fault category requires a different injection method and has different expected-state definitions. The taxonomy helps the test engineer ensure that the campaign covers all major failure modes, rather than focusing on the easiest faults to inject.

Network fault injection: packet loss, latency, and corruption #

Network faults are among the most common and most impactful failure modes in warehouse automation, because modern systems rely on industrial Ethernet for real-time control, safety, and data exchange. The NIST Guide to Operational Technology Security emphasizes that OT networks have unique performance and reliability requirements that differ from enterprise IT networks [S2]. Failure-injection testing of the network must account for these requirements.

Packet loss is the most straightforward network fault to inject. A network impairment device is placed inline between the PLC and the device under test, and configured to drop a specified percentage of packets. The test engineer must specify the packet loss rate, the packet type (e.g., UDP, TCP, or a specific protocol), and the duration of the loss. For example, a test might inject 10% UDP packet loss for 30 seconds between the PLC and a variable frequency drive (VFD).

The expected state for a packet-loss test depends on the protocol and the application. For a safety-rated protocol such as PROFIsafe or CIP Safety, the expected state is typically a safe stop within a specified time. For a non-safety protocol such as OPC UA, the expected state may be a timeout and alarm after a specified period. The OPC UA specification defines timeouts and retry behavior for its services [S5], and the test engineer should reference these values when defining the expected state.

Latency injection adds a fixed or variable delay to network traffic. This is particularly relevant for systems that use OPC UA for supervisory control, where excessive latency can cause the HMI to display stale data or the WMS to time out on transactions. The test engineer should inject latency values that exceed the protocol’s timeout thresholds to verify that the system handles the timeout gracefully.

Packet corruption involves modifying the contents of network packets, typically by flipping bits in the payload or checksum. This is more complex to inject than loss or latency, but it is important for verifying that the system detects and rejects corrupted data. The expected state is typically an alarm and a retry or a safe stop, depending on the protocol.

All network fault injections must be performed with a device that can be removed from the network path without disrupting the physical connection. This is typically a managed switch with fault-injection capabilities or an inline network impairment appliance. The device must be configured to fail open (i.e., pass traffic normally) if it loses power, so that a test-equipment failure does not become a production outage.

Sensor fault injection: dropout, stuck, and out-of-range #

Sensor faults are a critical test category because sensors are the primary input to the PLC’s decision-making logic. A sensor that fails can cause the system to make incorrect decisions, such as sending a pallet to the wrong destination or failing to stop a conveyor when a person enters a hazardous area.

Sensor dropout is the loss of the sensor signal. This can be injected by physically disconnecting the sensor cable, by removing power from the sensor, or by using a software override in the PLC to force the input to a false state. The expected state depends on the sensor’s role. For a safety-related sensor such as a light curtain, the expected state is a safe stop. For a non-safety sensor such as a pallet presence detector, the expected state may be a halt of the affected zone and an alarm.

Sensor stuck is the condition where the sensor remains in a fixed state, either ON or OFF, regardless of the actual physical condition. This is more insidious than dropout because the system may not detect the fault immediately. For example, if a pallet presence sensor is stuck ON, the PLC may continue to send pallets into a zone that is already occupied, causing a collision. The expected state for a stuck-sensor test is that the PLC detects the lack of state change within a specified time and raises an alarm.

Out-of-range sensor values occur when a sensor provides a value that is outside its expected operating range. This is common with analog sensors such as ultrasonic distance sensors or load cells. The expected state is that the PLC detects the out-of-range value and treats it as a fault, rather than using the invalid value in its control logic.

Sensor fault injection requires careful coordination with the physical environment. For example, to test a stuck sensor on a palletizer, the test engineer must ensure that the palletizer is in a safe state and that no personnel are in the area, because the fault may cause unexpected motion. The lockout/tagout principles from OSHA 29 CFR 1910.147 apply here [S4].

Controller fault injection: PLC and drive failures #

Controller faults are failures of the computational devices that execute the control logic—primarily PLCs, but also motion controllers, robot controllers, and drive controllers. These faults are different from network or sensor faults because they affect the system’s ability to process information at all.

PLC CPU faults can be injected by triggering a watchdog timeout, forcing a divide-by-zero error, or writing an invalid value to a critical memory location. The expected state depends on the PLC’s fault-handling configuration. Many PLCs can be configured to transition to a safe state (e.g., de-energize all outputs) on a CPU fault. Others may have a redundant CPU that takes over. The test must verify that the configured behavior actually occurs.

Drive faults are failures of the motor drives that control conveyor speed, robot axes, or other motion. A drive fault can be injected by removing the drive’s enable signal, by simulating an overcurrent condition, or by disconnecting the motor feedback encoder. The expected state is that the drive enters a safe state (e.g., safe torque off) and that the PLC detects the drive fault and takes appropriate action.

Controller fault injection is the most risky category of FIT because it can cause unexpected motion or loss of control. The test containment measures described earlier are particularly important here. The test zone must be physically isolated, and all personnel must be outside the zone before the fault is injected.

Pearl Gateway editorial recommendation: We recommend that controller fault injection be performed only on a dedicated test rig, not on a production line, unless the production line has a fully redundant control system and the test is limited to the non-redundant components. The risk of an uncontrolled fault in a production PLC is simply too high to justify the benefit of testing on live equipment.

Power fault injection: interruptions and brownouts #

Power faults are interruptions or reductions in the electrical supply to the automation equipment. These are distinct from other fault categories because they affect the physical energy supply to the system, not just the information flow.

Power interruption testing involves removing power from a specific device or zone for a specified duration. The expected state is that the device shuts down safely, that the PLC detects the power loss (typically via a digital input monitoring a power-good signal), and that the system can be restarted without data corruption.

Brownout testing involves reducing the voltage to a level below the device’s rated minimum but not to zero. This is more difficult to inject and requires a variable voltage source. Brownouts can cause devices to behave erratically—PLCs may reset, drives may lose torque, and sensors may provide invalid readings. The expected state is that the system detects the abnormal voltage and transitions to a safe state.

Power fault testing requires careful coordination with the facility’s electrical safety procedures. The lockout/tagout standard [S4] requires that energy sources be controlled during maintenance and testing. For power fault testing, this means that the test engineer must have a clear understanding of the electrical distribution system and must be able to isolate the test zone from the rest of the facility.

Power fault testing also requires consideration of the restoration sequence. When power is restored, the system may go through a startup sequence that is different from a normal power-on. The restoration script must account for this, including the re-initialization of drives, the re-establishment of network connections, and the verification of data integrity.

Data integrity fault injection: corruption and loss #

Data integrity faults are failures that cause the system’s data to be corrupted, lost, or duplicated. These faults are particularly important in warehouse automation because the system’s decisions depend on the accuracy of data such as pallet IDs, inventory counts, and order statuses.

Data corruption can be injected by writing invalid values to PLC memory, by corrupting database records, or by modifying data in transit. The expected state is that the system detects the corruption and either corrects it or raises an alarm. For example, if a pallet ID is corrupted, the PLC should reject the pallet and raise an alarm, rather than processing it with an invalid ID.

Data loss can be injected by deleting records from a database, by clearing PLC memory, or by dropping messages in transit. The expected state is that the system detects the loss and either recovers the data from a redundant source or raises an alarm.

Data duplication can be injected by duplicating messages or records. The expected state is that the system detects the duplicate and ignores it, rather than processing the same pallet twice.

Data integrity fault injection is often performed in conjunction with network fault injection, because data corruption and loss are often caused by network faults. However, it can also be performed at the application layer, by directly manipulating the data in the PLC or the WMS.

The OPC UA specification provides mechanisms for data integrity, including sequence numbers and timestamps [S5]. The test engineer should verify that the system uses these mechanisms to detect data anomalies.

Test planning: from fault taxonomy to test cases #

A failure-injection campaign is only as good as its test plan. The test plan transforms the fault taxonomy into a set of specific, executable test cases, each with its own containment plan, expected state, abort criteria, evidence requirements, and restoration procedure.

Each test case must include the following elements:

The test plan should be reviewed by the engineering team, the safety team, and the operations team before the campaign begins. The review should focus on the completeness of the containment plan, the correctness of the expected state, and the adequacy of the abort criteria.

Execution procedure: step-by-step discipline #

The execution of a failure-injection test requires the same discipline as a surgical procedure. Each step must be performed in a specific order, with verification at each stage.

The recommended execution sequence is:

1. Pre-test briefing: All test participants meet to review the test plan, confirm the containment measures, and verify that all abort criteria are understood.
2. System verification: The system under test is verified to be in a known-good state. This includes checking that all sensors are reading correctly, all drives are enabled, and all network connections are active.
3. Evidence setup: All evidence collection systems are started and verified to be recording. The time synchronization is checked.
4. Baseline recording: A short period of normal operation is recorded to establish a baseline for comparison.
5. Fault injection: The fault is injected using the specified method. The time of injection is recorded.
6. Observation: The system is observed for the specified observation period. The test engineer records any deviations from the expected state.
7. Fault removal: The fault is removed at the specified time or when the expected state is reached.
8. Evidence collection: All evidence is collected and labeled with the test ID and time.
9. Restoration: The restoration procedure is executed.
10. Post-test debrief: The test team reviews the evidence, evaluates the pass/fail criteria, and documents the results.

Each step must be documented in a test log, with timestamps and the name of the person performing the step. The test log is part of the evidence and must be retained with the test report.

Worked example #

This worked example demonstrates the application of the FIT framework to a specific test case. All numbers are illustrative assumptions and are labeled as such.

Inputs #

• System under test: A conveyor zone (Zone 4) with a variable frequency drive (VFD-04) controlled by PLC-01.
• Fault to inject: 10% UDP packet loss between PLC-01 and VFD-04 (illustrative assumption).
• Injection method: Inline network impairment device (illustrative assumption).
• Fault duration: 30 seconds (illustrative assumption).
• Expected state: VFD-04 enters safe torque off (STO) within 500 ms of the first dropped packet (illustrative assumption). PLC-01 raises alarm “VFD-04 comm loss” within 1 second of the first dropped packet (illustrative assumption).
• Conveyor speed at time of injection: 1.5 m/s (illustrative assumption).
• Conveyor deceleration rate under STO: 2.0 m/s² (illustrative assumption).

Intermediate calculations #

Stopping distance: The distance the conveyor travels after STO is triggered, assuming constant deceleration, is given by:

d = v² / (2 × a)

where:

• d = stopping distance (m)
• v = initial speed (m/s) = 1.5 m/s (illustrative assumption)
• a = deceleration rate (m/s²) = 2.0 m/s² (illustrative assumption)

d = (1.5)² / (2 × 2.0) = 2.25 / 4.0 = 0.5625 m

Time to stop: The time to stop is given by:

t = v / a

t = 1.5 / 2.0 = 0.75 s

Total time from fault to stop: The total time from the first dropped packet to the conveyor coming to a stop is the sum of the STO activation time and the stopping time:

T_total = t_STO + t_stop

where:

• T_total = total time from fault to stop (s)
• t_STO = STO activation time (s) = 0.5 s (illustrative assumption)
• t_stop = stopping time (s) = 0.75 s (calculated above)

T_total = 0.5 + 0.75 = 1.25 s

Total distance traveled: The total distance traveled from the first dropped packet to the conveyor coming to a stop is:

D_total = v × t_STO + d

D_total = 1.5 × 0.5 + 0.5625 = 0.75 + 0.5625 = 1.3125 m

Result #

If the test is executed as specified, the conveyor will travel approximately 1.31 m (illustrative assumption) after the first dropped packet before coming to a complete stop. The total time from fault to stop will be approximately 1.25 s (illustrative assumption). The test passes if VFD-04 enters STO within 500 ms and PLC-01 raises the alarm within 1 s.

Sensitivity #

The result is most sensitive to the STO activation time (t_STO). If t_STO increases from 0.5 s to 1.0 s (illustrative assumption), the total distance increases to:

D_total = 1.5 × 1.0 + 0.5625 = 1.5 + 0.5625 = 2.0625 m

This is a 57% increase in stopping distance. The result is also sensitive to the initial speed. If the conveyor is running at 2.0 m/s instead of 1.5 m/s (illustrative assumption), the stopping distance increases to:

d = 2.0² / (2 × 2.0) = 4.0 / 4.0 = 1.0 m

This is a 78% increase in stopping distance.

Limitations #

This calculation assumes constant deceleration, which may not be accurate for all VFDs. It also assumes that the STO activation time is constant, which may vary with network conditions. The calculation does not account for the time required for the PLC to detect the fault and send the STO command, which is included in the t_STO parameter but may vary. The calculation is valid only for the specific fault injected (10% UDP packet loss) and may not be representative of other fault types.

When this guidance does not apply #

This guidance does not apply in several specific situations. First, it does not apply to safety-rated systems that are certified under functional safety standards such as IEC 61508 or ISO 13849. These systems have their own validation and verification requirements, and failure-injection testing must be performed in accordance with those standards, not the general framework described here. The test engineer must consult the relevant safety standard and the system’s safety case before performing any FIT on safety-rated equipment.

Second, this guidance does not apply to systems that are in active production and cannot be isolated. If a failure-injection test cannot be contained to a dedicated test zone, the risk of an uncontrolled fault is too high, and the test should not be performed. This is a Pearl Gateway editorial recommendation based on the principle that safety and operational continuity take precedence over testing objectives.

Third, this guidance does not apply to systems that are still in the design phase and have not yet been commissioned. Failure-injection testing is a verification activity that requires a working system. For design-phase systems, the appropriate activities are simulation, model checking, and design review.

Fourth, this guidance does not apply to fault injection performed for security testing, such as penetration testing or fuzzing. While there is overlap in the injection techniques, security testing has different objectives (e.g., identifying vulnerabilities that could be exploited by an attacker) and different legal and ethical considerations. The NIST Guide to Operational Technology Security [S2] provides guidance for security testing that is distinct from the FIT framework described here.

Finally, this guidance does not apply to systems that have not been through a proper factory acceptance test (FAT) or site acceptance test (SAT). Failure-injection testing assumes that the system has already been verified to function correctly under nominal conditions. If the system has not passed FAT and SAT, the appropriate next step is to complete those tests, not to proceed to FIT. The Pearl Gateway article on factory acceptance testing provides guidance on this prerequisite activity.

Integration with commissioning and lifecycle management #

Failure-injection testing should be integrated into the broader commissioning and lifecycle management process, not performed as a standalone activity. The Pearl Gateway article on factory acceptance testing describes the selection criteria and application boundaries for FAT, which is the natural precursor to FIT. FIT extends the verification performed during FAT by exercising the fault modes that FAT deliberately avoids.

During commissioning, FIT should be performed after the system has passed FAT and SAT, and before the system is handed over to operations. The results of the FIT campaign should be documented in the system handover records, as described in the Pearl Gateway article on system handover records. These records provide the evidence that the system has been verified to fail safely, which is essential information for the operations team.

During the operational lifecycle, FIT should be repeated when significant changes are made to the system, such as a PLC program update, a network reconfiguration, or the addition of new equipment. The Pearl Gateway article on lifecycle upgrade planning provides guidance on when and how to plan for these changes. FIT is a key verification activity for any change that affects the system’s fault-handling logic.

FIT is also relevant to the commissioning of supporting infrastructure. For example, the Pearl Gateway article on warehouse Wi-Fi coverage describes the commissioning and acceptance checklist for wireless networks. If the warehouse uses Wi-Fi for any control or monitoring functions, FIT should include tests for Wi-Fi packet loss and latency. Similarly, the article on control network capacity describes the data signals and condition monitoring for the control network, which is the primary target for network fault injection.

Roles and responsibilities for FIT campaigns #

A failure-injection campaign requires clear roles and responsibilities to ensure safety and accountability. The following roles are recommended for any FIT campaign:

• Test director: The person with overall authority for the test campaign. The test director approves the test plan, authorizes each test, and has the authority to abort any test. The test director is typically a senior engineer or engineering manager.
• Test engineer: The person who designs and executes the individual test cases. The test engineer is responsible for the injection method, the expected state definition, and the evidence collection.
• Safety officer: The person responsible for verifying that all safety measures are in place before each test. The safety officer has the authority to halt any test if a safety condition is violated.
• Operations representative: The person representing the operations team that will operate the system after commissioning. The operations representative verifies that the test plan does not conflict with operational requirements and that the restoration procedure is practical.
• Evidence recorder: The person responsible for collecting, labeling, and archiving all evidence. This role is often combined with the test engineer role, but for large campaigns, a dedicated evidence recorder is recommended.

Each role must be assigned to a named individual, and the assignments must be documented in the test plan. The test director must not be the same person as the safety officer, to ensure that safety decisions are independent of test objectives.

Documentation and reporting standards #

The documentation of a failure-injection campaign is as important as the tests themselves. The documentation serves as the evidence that the system has been verified to fail safely, and it provides the basis for future changes and upgrades.

The minimum documentation set for a FIT campaign includes:

• Test plan: The overall plan for the campaign, including the fault taxonomy, the list of test cases, the roles and responsibilities, and the schedule.
• Test case specifications: A detailed specification for each test case, including the elements described in the test planning section above.
• Test execution log: A timestamped log of each test execution, including the names of the participants, the actions taken, and the observations made.
• Test report: A summary of the campaign results, including the pass/fail status of each test case, the evidence collected, and any defects identified.
• Defect reports: A detailed description of any defects found during the campaign, including the fault that revealed the defect, the system behavior, and the recommended corrective action.

The documentation should be stored in a version-controlled repository and retained for the life of the system. The Pearl Gateway article on system handover records describes the types of records that should be handed over to operations, and the FIT documentation should be included in this set.

Common pitfalls and how

Related Pearl Gateway guides #

• Presence Detection: Capacity Planning and Bottleneck Analysis
• Warehouse Wi-Fi Coverage: Commissioning and Acceptance Checklist
• Control Network Capacity: Data Signals and Condition Monitoring
• Stretch Wrapper Integration: Selection Criteria and Application Boundaries
• Pallet Dimension Checks: Data Signals and Condition Monitoring

Sources and standards #

1. NASA — NASA Systems Engineering Handbook. In “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover”, source [S1] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
2. NIST — Guide to Operational Technology Security, SP 800-82 Rev. 3. In “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover”, source [S2] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
3. NIST — Engineering Statistics Handbook. In “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover”, source [S3] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
4. OSHA — The Control of Hazardous Energy (Lockout/Tagout), 29 CFR 1910.147. In “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover”, source [S4] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
5. OPC Foundation — OPC UA Online Reference. In “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover”, source [S5] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.

Revision and editorial note #

The Pearl Gateway Editorial Team prepared “Failure-Injection Testing for Warehouse Automation: What to Test and How to Recover” from the five linked source records. The published guide remains educational and requires site evidence before application.