Direct answer #
Intermittent automation faults resist diagnosis because their evidence is scattered across time. The Time-Aligned Evidence Method (TAEM) treats a fault not as a single alarm event, but as a hypothesis about a sequence of correlated observations. This article defines a rigorous workflow: capture timestamped data with known uncertainty, reconstruct the change history of the affected subsystem, form falsifiable hypotheses, and validate them through time-aligned correlation before any rollback or component replacement. The method requires explicit timestamp uncertainty budgets, a documented configuration baseline, and a commitment to disprove rather than confirm initial suspicions. It is designed for warehouse automation systems—conveyors, diverters, shuttles, and sorters—where a single misaligned sensor or a marginal power supply can produce faults that appear random. By aligning evidence on a common timeline, maintenance teams can separate causation from coincidence and reduce the risk of replacing healthy components.
Key takeaways #
- Timestamp uncertainty is the primary constraint: Without a quantified uncertainty budget for every data source, time-aligned correlation is meaningless. A 100 ms uncertainty in a PLC timestamp and a 2 s uncertainty in a video timestamp cannot be treated equally.
- Change history is mandatory evidence: Any configuration change, firmware update, mechanical adjustment, or parts replacement in the 30 days prior to the first fault is a candidate cause. The absence of a change log is itself a finding.
- Hypotheses must be falsifiable: A hypothesis that cannot be disproven by a specific observation or experiment has no place in root-cause analysis. Define the evidence that would refute your theory before you collect more data.
- Rollback is a controlled experiment, not a fix: Rolling back a change is only valid if you define the success criteria, the observation window, and the roll-forward criteria in advance. Otherwise, you cannot interpret the result.
- Correlation does not equal causation, but time-aligned correlation is the strongest available evidence: When two independent data streams show a consistent temporal relationship across multiple fault instances, the causal link becomes credible.
- Safety boundaries are absolute: The method is for diagnostic analysis only. Any activity that involves energy isolation, lockout/tagout, or physical intervention must follow site-specific procedures and applicable regulations such as OSHA 29 CFR 1910.147 [S5].
Intermittent faults: the diagnostic paradox #
Intermittent faults are the most time-consuming category of automation failures. A fault that occurs once per shift, or once per week, produces a small amount of alarm data surrounded by a large amount of normal operation data. The diagnostic paradox is that the fault is real—it causes downtime, product damage, or safety incidents—but its cause is not continuously present. The system behaves correctly for hours or days, then fails for milliseconds or seconds, then returns to normal operation.
Traditional troubleshooting methods, which rely on reproducing the fault under controlled conditions, often fail because the conditions that trigger the fault are not understood. The Time-Aligned Evidence Method (TAEM) addresses this by shifting the focus from reproduction to correlation. Instead of asking “what causes the fault?”, the method asks “what else happens at the same time as the fault?” This question is answerable with the data already available in most modern warehouse automation systems.
The method is applicable to a wide range of equipment, including belt-to-roller transfers, conveyor side guides, tilt-tray sorters, divert confirmation sensors, four-way pallet shuttles, and automated buffer storage. Each of these systems has documented failure modes and diagnostic evidence that can be correlated with timestamped fault data. The method does not replace equipment-specific knowledge; it provides a framework for using that knowledge effectively.
TAEM is built on four pillars: timestamp uncertainty, change history, falsifiable hypotheses, and controlled rollback. Each pillar is described in detail in the following sections. The method is deliberately prescriptive because the cost of a wrong conclusion—replacing healthy components, extending downtime, or masking a safety hazard—is high.
Timestamp uncertainty: the foundation of time alignment #
Time alignment is only as good as the timestamps being aligned. Every data source in an automation system has a timestamp with a specific uncertainty, and that uncertainty must be quantified before any correlation analysis is attempted. The timestamp uncertainty budget is a table that lists each data source, its timestamp resolution, its synchronization error, and its total uncertainty.
The NIST Time and Frequency Division maintains the standard for time and frequency interval in the United States and provides official time [S3]. This is the reference point for time synchronization in industrial systems. However, the existence of a national standard does not mean that every device in a warehouse is synchronized to it. PLCs, sensors, cameras, and middleware may each have their own clocks, and these clocks drift relative to each other.
The timestamp uncertainty budget must account for three components:
- Resolution: The smallest time increment that the device can represent. A PLC scan cycle of 10 ms means that all timestamps have a resolution of 10 ms, even if the clock is perfectly synchronized.
- Synchronization error: The difference between the device clock and the reference clock. This can be measured using network time protocol (NTP) statistics, precision time protocol (PTP) reports, or manual comparison.
- Jitter: The variation in synchronization error over time. Jitter is caused by network load, CPU scheduling, and interrupt handling.
Table 1 shows an illustrative timestamp uncertainty budget for a typical warehouse automation system. All values are illustrative assumptions and must be replaced with measured values for a specific site.
| Data source | Timestamp resolution (ms) | Synchronization error (ms) | Jitter (ms) | Total uncertainty (ms) | Notes |
|---|---|---|---|---|---|
| PLC fault log | 10 | ±50 | ±20 | ±80 | PLC scan cycle limits resolution |
| Vision system | 1 | ±200 | ±50 | ±251 | Camera clock not synchronized to PLC |
| OPC UA historian | 1 | ±100 | ±30 | ±131 | Depends on OPC UA server configuration [S4] |
| Video surveillance | 40 (25 fps) | ±500 | ±100 | ±640 | Frame rate limits resolution |
| Maintenance log (manual) | 60000 (1 min) | ±30000 | ±60000 | ±90000 | Human entry, highly variable |
The total uncertainty is calculated as the square root of the sum of the squares of the components, assuming they are independent and normally distributed. This is a standard error propagation formula. For the PLC fault log, the calculation is √(10² + 50² + 20²) = √(100 + 2500 + 400) = √3000 ≈ 54.8 ms, which is rounded to ±80 ms to account for unmodeled effects. The illustrative assumption of ±80 ms is conservative.
The practical consequence of the timestamp uncertainty budget is that two events cannot be considered simultaneous unless their time intervals overlap. If a PLC fault is recorded at 10:00:00.000 with ±80 ms uncertainty, and a vision system event is recorded at 10:00:00.050 with ±251 ms uncertainty, the intervals are [09:59:59.920, 10:00:00.080] and [09:59:59.799, 10:00:00.301]. These intervals overlap, so the events could be simultaneous. If the vision event were recorded at 10:00:00.500, the intervals would not overlap, and the events would be temporally distinct.
Change history: the forgotten evidence #
Intermittent faults often begin after a change that was not recognized as significant. A firmware update, a sensor replacement, a guide rail adjustment, or a conveyor speed change can alter system behavior in ways that are not immediately apparent. The change history is a chronological record of all modifications to the affected subsystem, including configuration changes, software updates, mechanical adjustments, and parts replacements.
The change history must be reconstructed for a period before the first fault occurrence. The length of this period depends on the fault frequency. For a fault that occurs weekly, a 90-day history is appropriate. For a fault that occurs monthly, a 180-day history may be needed. The illustrative assumption of a 30-day minimum window is a starting point, not a rule.
Table 2 shows an illustrative change history for a conveyor system with an intermittent jam fault. All entries are illustrative assumptions.
| Date | Time (local) | Change type | Description | Person/team | Change ID |
|---|---|---|---|---|---|
| 2026-07-15 | 08:30 | Software | PLC firmware updated from v2.1 to v2.2 | Controls team | CHG-2026-0715-01 |
| 2026-07-22 | 14:00 | Mechanical | Belt-to-roller transfer guide rail adjusted | Maintenance | CHG-2026-0722-02 |
| 2026-08-01 | 10:15 | Sensor | Divert confirmation sensor replaced | Maintenance | CHG-2026-0801-03 |
| 2026-08-10 | 16:45 | Configuration | Conveyor speed increased from 1.2 m/s to 1.5 m/s | Operations | CHG-2026-0810-04 |
| 2026-08-18 | 09:00 | Software | OPC UA server certificate renewed | IT | CHG-2026-0818-05 |
The change history serves two purposes. First, it identifies candidate causes. In the illustrative example, the conveyor speed increase on 2026-08-10 is a prime candidate if the first fault occurred after that date. Second, it provides the basis for a rollback experiment. If a change is suspected, rolling back that change and observing the result is a controlled experiment.
The absence of a change history is itself a finding. If no records exist, the maintenance team must assume that undocumented changes occurred and treat the system configuration as unknown. In this case, the first step is to document the current state of the system, including firmware versions, configuration parameters, and mechanical settings. This documentation becomes the baseline for future comparisons.
Change history reconstruction should use multiple sources: work orders, maintenance logs, PLC program archives, configuration management databases, and interviews with technicians. Each source has its own timestamp uncertainty, which must be included in the analysis. A work order timestamped at 14:00 with ±30 minutes uncertainty is not precise enough to correlate with a fault that occurred at 14:00:00.000 with ±80 ms uncertainty.
Falsifiable hypotheses: the scientific method applied to maintenance #
A hypothesis is falsifiable if it can be disproven by a specific observation or experiment. In the context of intermittent fault analysis, a falsifiable hypothesis makes a prediction about what should be observed if the hypothesis is true, and that prediction must be testable with available data or experiments.
Consider the following hypotheses for an intermittent jam fault on a belt-to-roller transfer:
- Hypothesis A: The jam occurs when the belt speed exceeds a threshold of 1.4 m/s. This is falsifiable because it predicts that no jams occur at speeds below 1.4 m/s. If a jam is observed at 1.2 m/s, the hypothesis is disproven.
- Hypothesis B: The jam occurs when the divert confirmation sensor is misaligned by more than 2 mm. This is falsifiable because it predicts that jams correlate with sensor misalignment measurements. If the sensor is within tolerance during a jam, the hypothesis is disproven.
- Hypothesis C: The jam occurs randomly with a probability of 0.1% per cycle. This is falsifiable because it predicts a specific fault rate. If the observed rate is significantly different, the hypothesis is disproven.
Each hypothesis must be stated in a form that allows disproof. The statement “the jam is caused by a sensor problem” is not falsifiable because it does not specify which sensor, what kind of problem, or what observation would disprove it. The statement “the jam is caused by the divert confirmation sensor producing a false positive when the ambient light level exceeds 500 lux” is falsifiable because it can be tested by measuring light levels during jams.
The NASA Systems Engineering Handbook emphasizes the importance of verification and validation in systems engineering [S2]. While the handbook is focused on aerospace systems, the principle applies to industrial automation: a hypothesis must be verified through evidence and validated through testing before it is accepted as a root cause.
The process of hypothesis formation should be structured. Start with the change history and identify all changes that occurred before the first fault. For each change, form a hypothesis that links the change to the fault mechanism. Then, for each hypothesis, define the evidence that would disprove it. This evidence should be specific and measurable.
Hypotheses should be ranked by plausibility and testability. A hypothesis that is highly plausible but difficult to test may be ranked lower than a hypothesis that is less plausible but easy to test. The ranking is used to prioritize data collection and experiments.
Time-aligned correlation: the core analytical step #
Time-aligned correlation is the process of placing all relevant data on a common timeline and looking for consistent temporal relationships. The timeline must account for the timestamp uncertainty of each data source, as described in the timestamp uncertainty budget section. Events are aligned if their uncertainty intervals overlap.
The correlation analysis proceeds in four steps:
- Collect all fault instances: Identify every occurrence of the fault in the available data. For each instance, record the timestamp and the uncertainty interval.
- Collect all candidate events: Identify all events that could be related to the fault, including alarms, sensor state changes, configuration changes, and operator actions. Record the timestamp and uncertainty interval for each.
- Align events to fault instances: For each fault instance, find all candidate events whose uncertainty intervals overlap with the fault instance interval. This creates a set of co-occurring events.
- Look for consistent patterns: Across all fault instances, look for events that consistently co-occur. A consistent pattern is evidence of a causal relationship.
The key output of the correlation analysis is a table showing, for each fault instance, the co-occurring events and their time offsets. Table 3 shows an illustrative correlation table for a jam fault on a belt-to-roller transfer.
| Fault instance | Fault time (PLC) | Fault uncertainty (ms) | Co-occurring event | Event time (source) | Event uncertainty (ms) | Time offset (ms) |
|---|---|---|---|---|---|---|
| F-001 | 10:00:00.000 | ±80 | Divert sensor false positive | 10:00:00.050 | ±251 | +50 |
| F-002 | 14:30:15.000 | ±80 | Divert sensor false positive | 14:30:15.120 | ±251 | +120 |
| F-003 | 09:15:45.000 | ±80 | Divert sensor false positive | 09:15:44.980 | ±251 | -20 |
| F-004 | 16:00:30.000 | ±80 | No co-occurring event | — | — | — |
In this illustrative example, the divert sensor false positive co-occurs with three of four fault instances. The fourth instance has no co-occurring event, which means either the event was not captured or the hypothesis is incomplete. The consistent co-occurrence across three instances is evidence that the divert sensor is involved, but the fourth instance requires explanation.
The time offset is the difference between the event time and the fault time. A consistent offset across multiple instances suggests a causal chain. For example, if the divert sensor false positive consistently occurs 50–120 ms before the jam fault, this suggests that the sensor triggers a sequence of events that leads to the jam. If the offset is inconsistent, the relationship may be coincidental.
The correlation analysis must be repeated for each hypothesis. A hypothesis is supported if its predicted events consistently co-occur with fault instances. A hypothesis is weakened if its predicted events do not co-occur, or if they co-occur with only a small fraction of fault instances.
Worked example #
This worked example demonstrates the Time-Aligned Evidence Method for an intermittent jam fault on a belt-to-roller transfer. All inputs are illustrative assumptions and must be replaced with site-specific data.
Inputs #
- Fault: Intermittent jam at belt-to-roller transfer, occurring approximately 3 times per week.
- Fault duration: 2–5 seconds, from jam detection to recovery.
- PLC fault log timestamp uncertainty: ±80 ms (from Table 1).
- Vision system timestamp uncertainty: ±251 ms (from Table 1).
- Change history: Conveyor speed increased from 1.2 m/s to 1.5 m/s on 2026-08-10 (CHG-2026-0810-04).
- Fault instances collected over 14 days: 6 instances.
Intermediate calculations #
Step 1: Calculate fault rate. The fault rate is 6 instances / 14 days = 0.43 instances/day. The expected rate under the hypothesis of random failure is unknown, so this rate is used as a baseline.
Step 2: Calculate time between changes and first fault. The conveyor speed change occurred on 2026-08-10. The first fault in the observation window occurred on 2026-08-12. The time between change and first fault is 2 days.
Step 3: Calculate correlation with speed. The conveyor speed is 1.5 m/s after the change. All 6 fault instances occurred after the speed change. The correlation is 6/6 = 100%.
Step 4: Calculate correlation with divert sensor. The divert confirmation sensor produced a false positive within the uncertainty interval of 5 of the 6 fault instances. The correlation is 5/6 = 83%.
Step 5: Calculate time offset for divert sensor. The time offsets for the 5 co-occurring instances are: +50 ms, +120 ms, -20 ms, +80 ms, +95 ms. The mean offset is (50 + 120 – 20 + 80 + 95) / 5 = 325 / 5 = 65 ms. The standard deviation is √[( (50-65)² + (120-65)² + (-20-65)² + (80-65)² + (95-65)² ) / 4] = √[(225 + 3025 + 7225 + 225 + 900) / 4] = √[11600 / 4] = √2900 ≈ 53.9 ms.
Step 6: Calculate correlation with guide rail adjustment. The guide rail was adjusted on 2026-07-22. The fault instances occurred on 2026-08-12, 2026-08-14, 2026-08-17, 2026-08-19, 2026-08-21, and 2026-08-24. All instances occurred after the guide rail adjustment. The correlation is 6/6 = 100%, but this is expected because the adjustment predates all instances.
Result #
The analysis shows a 100% correlation with the conveyor speed change and an 83% correlation with the divert sensor false positive. The mean time offset of 65 ms with a standard deviation of 53.9 ms suggests a consistent temporal relationship between the sensor false positive and the jam fault. The hypothesis that the jam is caused by the divert sensor false positive is supported.
The hypothesis that the jam is caused by the conveyor speed increase is also supported, but the mechanism is less clear. The speed increase may have changed the timing of the belt-to-roller transfer, causing the divert sensor to misread the product position.
Sensitivity #
The sensitivity of the result to the timestamp uncertainty is significant. If the divert sensor timestamp uncertainty were ±500 ms instead of ±251 ms, the correlation would still be 5/6, but the time offset calculation would be less precise. If the uncertainty were ±1000 ms, the correlation might change because some events would no longer overlap.
The sensitivity to the fault instance selection is also significant. If one of the 6 fault instances were removed, the correlation with the divert sensor would change from 5/6 to 4/5 or 5/5, depending on which instance is removed. This sensitivity should be reported.
Limitations #
The worked example has several limitations. First, the sample size of 6 fault instances is small, and the correlation could be due to chance. Second, the divert sensor false positive may be a symptom of the jam rather than a cause. Third, the change history is incomplete, and there may be undocumented changes that are the true cause. Fourth, the analysis does not account for environmental factors such as temperature, humidity, or vibration. These limitations must be stated in the final report.
Rollback as a controlled experiment #
Rollback is the process of reverting a change to determine whether the fault is eliminated. Rollback is only valid as a controlled experiment, with predefined success criteria, observation window, and roll-forward criteria. Without these, the result of a rollback is uninterpretable.
The rollback experiment is designed as follows:
- Select the change to roll back: The change should be the one with the strongest correlation and the most plausible mechanism. In the worked example, the conveyor speed change is the prime candidate.
- Define success criteria: The success criterion is that no jam faults occur during the observation window. The observation window must be long enough to capture the expected fault rate. For a fault rate of 0.43 instances/day, a 14-day observation window would be expected to contain 6 faults. A 21-day window would be more conservative.
- Define roll-forward criteria: If the rollback is successful (no faults in the observation window), the change can be reinstated with a plan for further testing. If the rollback is unsuccessful (faults continue), the change is exonerated, and the next candidate change is selected.
- Execute the rollback: The rollback must be performed according to site procedures, including lockout/tagout if energy isolation is required [S5]. The rollback must be documented with timestamps.
- Observe and record: During the observation window, all fault instances and all relevant events must be recorded with timestamps and uncertainty.
- Interpret the result: If no faults occur, the rollback supports the hypothesis. If faults occur, the hypothesis is weakened or disproven.
The rollback experiment has a critical limitation: it cannot distinguish between the rolled-back change and other changes that occurred at the same time. If the conveyor speed is rolled back from 1.5 m/s to 1.2 m/s, and the fault disappears, the cause could be the speed change, or it could be a coincidental change in environmental conditions. This limitation is addressed by repeating the experiment: roll forward again, observe, roll back again, observe. If the fault consistently appears at 1.5 m/s and disappears at 1.2 m/s, the evidence is strong.
Rollback is not a fix. It is a diagnostic experiment. The goal is to identify the root cause, not to permanently revert to an older configuration. After the root cause is identified, a permanent fix can be designed and implemented.
Data collection requirements: what to capture and how #
The Time-Aligned Evidence Method depends on the availability of timestamped data. The data collection requirements must be defined before the method can be applied. The following data sources are typically required:
- PLC fault logs: The PLC must be configured to record fault events with timestamps. The timestamp resolution is limited by the PLC scan cycle. The PLC clock must be synchronized to a reference time source.
- OPC UA historian: If the automation system uses OPC UA, the historian can provide time-stamped data for variables, alarms, and events. The OPC UA specification defines the information model and services for historical access [S4]. The historian timestamp uncertainty depends on the server configuration and the network.
- Vision system logs: Vision systems used for product inspection or positioning can provide timestamped data. The timestamp uncertainty is often larger than PLC timestamps because the vision system clock is not synchronized to the PLC.
- Video surveillance: Video footage can provide visual evidence of the fault. The timestamp uncertainty is limited by the frame rate. A 25 fps camera has a resolution of 40 ms.
- Maintenance logs: Manual logs of maintenance activities are essential for change history reconstruction. The timestamp uncertainty is large, often ±30 minutes or more.
- Environmental sensors: Temperature, humidity, and vibration sensors can provide data on environmental conditions. These are useful for testing hypotheses related to environmental factors.
The data collection plan must specify the retention period for each data source. PLC fault logs may be retained for 30 days, while video footage may be retained for 7 days. The retention period must be longer than the expected time between fault instances and the change history window.
The data collection plan must also specify the synchronization method. NTP can provide synchronization to within milliseconds on a local network. PTP can provide sub-microsecond synchronization on a dedicated network. The choice of synchronization method affects the timestamp uncertainty budget.
Common pitfalls in intermittent fault analysis #
Several pitfalls consistently undermine intermittent fault analysis. Recognizing these pitfalls is the first step to avoiding them.
Pitfall 1: Ignoring timestamp uncertainty. Analysts often treat all timestamps as exact, leading to false correlations. Two events that appear simultaneous may be separated by hundreds of milliseconds when uncertainty is considered. The timestamp uncertainty budget must be applied to every correlation.
Pitfall 2: Confirmation bias. Once a hypothesis is formed, analysts tend to seek confirming evidence and ignore disconfirming evidence. The falsifiable hypothesis framework is designed to counter this bias by requiring explicit disproof criteria.
Pitfall 3: Replacing parts without a hypothesis. Replacing a sensor or a motor “to see if it helps” is a common but ineffective strategy. Without a hypothesis and a measurement plan, the result of the replacement cannot be interpreted. If the fault continues, the replacement was wasted. If the fault stops, the cause is not proven because other factors may have changed.
Pitfall 4: Ignoring the change history. The most common cause of intermittent faults is a recent change. If the change history is not reconstructed, the analysis may miss the most likely cause.
Pitfall 5: Treating correlation as causation. Time-aligned correlation is strong evidence, but it is not proof. A consistent co-occurrence may be due to a common cause that affects both the fault and the co-occurring event. The rollback experiment is the best available method for establishing causation.
Pitfall 6: Inadequate observation windows. A rollback experiment with a short observation window may miss faults that occur infrequently. The observation window must be long enough to capture the expected fault rate with statistical confidence.
Equipment-specific application: correlating known failure modes #
The Time-Aligned Evidence Method is applied differently to different types of equipment. The equipment-specific failure modes and diagnostic evidence provide the candidate events for correlation. The following examples illustrate the application to common warehouse automation equipment.
Belt-to-roller transfers: Known failure modes include belt slippage, roller bearing wear, and product jams at the transfer point. The diagnostic evidence includes belt speed, roller rotation rate, and jam sensor status. The inspection points and early warning signs are documented in the Pearl Gateway reference for belt-to-roller transfers. The correlation analysis should align jam faults with belt speed variations and roller rotation anomalies.
Conveyor side guides: Known failure modes include guide misalignment, wear, and product impact. The diagnostic evidence includes guide position, product clearance, and impact sensor data. The failure modes and diagnostic evidence are documented in the Pearl Gateway reference for conveyor side guides. The correlation analysis should align product damage events with guide position changes.
Tilt-tray sorters: Known failure modes include tray misalignment, tilt mechanism failure, and product mis-sorts. The diagnostic evidence includes tray position, tilt angle, and sort confirmation signals. The inspection points and early warning signs are documented in the Pearl Gateway reference for tilt-tray sorters. The correlation analysis should align mis-sort events with tray position anomalies.
Divert confirmation sensors: Known failure modes include false positives, false negatives, and signal degradation. The diagnostic evidence includes sensor state, signal strength, and ambient light levels. The failure modes and diagnostic evidence are documented in the Pearl Gateway reference for divert confirmation sensors. The correlation analysis should align divert failures with sensor state anomalies.
Four-way pallet shuttles: Known failure modes include positioning errors, drive failures, and communication loss. The diagnostic evidence includes shuttle position, drive current, and communication status. The inspection points and early warning signs are documented in the Pearl Gateway reference for four-way pallet shuttles. The correlation analysis should align shuttle faults with positioning errors.
Automated buffer storage: Known failure modes include retrieval errors, storage allocation errors, and crane positioning faults. The diagnostic evidence includes crane position, load status, and allocation records. The failure modes and diagnostic evidence are documented in the Pearl Gateway reference for automated buffer storage. The correlation analysis should align retrieval errors with crane position anomalies.
In each case, the equipment-specific reference provides the list of candidate events and the expected temporal relationships. The Time-Aligned Evidence Method provides the framework for testing these relationships with timestamped data.
The role of OPC UA in evidence collection #
OPC UA (Unified Architecture) is a communication standard for industrial automation that provides a structured way to access data, alarms, and historical information. The OPC UA specification defines the information model, services, and security model [S4]. In the context of the Time-Aligned Evidence Method, OPC UA serves as a primary data source for timestamped evidence.
The OPC UA integration principles and system boundaries are documented in the Pearl Gateway reference for OPC UA integration. Key considerations for evidence collection include:
- Timestamp source: OPC UA servers can provide timestamps for data values, alarms, and events. The timestamp source may be the server clock, the client clock, or the source device clock. The timestamp uncertainty budget must account for the source.
- Historical access: OPC UA Part 11 defines historical access services [S4]. The historian can store timestamped data for later retrieval. The historian configuration affects the timestamp resolution and retention period.
- Alarms and conditions: OPC UA Part 9 defines alarms and conditions [S4]. Alarm events include timestamps and state transitions. The alarm timestamp uncertainty must be included in the budget.
- Aggregates: OPC UA Part 13 defines aggregates for processing historical data [S4]. Aggregates such as average, minimum, and maximum can be used to summarize data over time intervals. The aggregate timestamp represents the interval, not a specific instant.
OPC UA is not a substitute for a time synchronization protocol. The OPC UA server and clients must be synchronized to a common time source for the timestamps to be comparable. The synchronization method (NTP, PTP, or manual) must be documented.
Documentation and reporting: making the analysis auditable #
The Time-Aligned Evidence Method produces a large amount of data and analysis. The documentation must be sufficient for another engineer to review the analysis and reach the same conclusion. The following documentation elements are required:
- Timestamp uncertainty budget: The table of data sources, resolutions, synchronization errors, jitter, and total uncertainty.
- Change history: The chronological record of all changes, with timestamps and uncertainty.
- Fault instance log: The list of all fault instances, with timestamps and uncertainty intervals.
- Correlation tables: The tables showing co-occurring events for each fault instance.
- Hypothesis statements: The falsifiable hypotheses, with the evidence that would disprove each.
- Rollback experiment plan and results: The plan, the execution log, and the results.
- Conclusion and recommendations: The root cause determination and the recommended permanent fix.
The report should include a timeline diagram showing the fault instances, the co-occurring events, and the change history on a common time axis. The timeline diagram is a powerful communication tool because it makes the temporal relationships visible.
The report must state the limitations of the analysis, including the sample size, the timestamp uncertainty, and the completeness of the change history. The limitations are not a sign of weakness; they are a sign of rigor.
Safety and site-specific decision boundaries #
The Time-Aligned Evidence Method is a diagnostic analysis method. It does not replace site-specific safety procedures, lockout/tagout requirements, or operational decision-making. The following boundaries apply:
- Energy isolation: Any activity that involves exposure to hazardous energy must follow the site’s lockout/tagout procedures, which are required by OSHA 29 CFR 1910.147 [S5]. The method described in this article does not authorize any deviation from these procedures.
- Data collection: Data collection must not interfere with safety systems. For example, installing additional sensors or logging equipment must not compromise the function of safety interlocks.
- Rollback experiments: Rollback experiments must be approved by the site’s change management process. The experiment must not create unsafe conditions. For example, rolling back a conveyor speed change may affect downstream equipment.
- Decision authority: The decision to implement a permanent fix, replace a component, or modify a process rests with the site’s engineering and operations teams. The Time-Aligned Evidence Method provides evidence, but it does not make decisions.
These boundaries are not repetitive boilerplate. They define the limits of the method and the responsibilities of the personnel applying it. The method is a tool for analysis, not a substitute for professional judgment.
When this guidance does not apply #
The Time-Aligned Evidence Method is designed for intermittent faults in automated warehouse equipment where timestamped data is available. The method does not apply in the following situations:
- Continuous faults: If a fault is continuous, the cause is present at all times, and the fault can be reproduced on demand. Traditional troubleshooting methods are more appropriate.
- No timestamped data: If the system does not record timestamps for faults or events, the method cannot be applied. The first step is to implement data collection.
- Safety-critical faults: If a fault poses an immediate safety risk, the equipment must be taken out of service and the fault addressed through the site’s emergency procedures. The Time-Aligned Evidence Method is not an emergency response procedure.
- Single-instance faults: If a fault has occurred only once, there is insufficient data for correlation analysis. The method requires multiple fault instances to identify consistent patterns.
- Design flaws: If the fault is caused by a fundamental design flaw, such as inadequate capacity or incorrect component selection, the method may identify the symptoms but not the underlying design issue. A design review is required.
In these situations, the method may still provide useful context, but it should not be the primary diagnostic approach.
Revision and editorial note #
This article was prepared by the Pearl Gateway Editorial Team. The content was reviewed against the listed sources [S1]–[S5] to ensure that all attributed facts are accurately represented. The editorial team confirms that no external field data, customer references, certifications, or test results were invented for this article. All example numbers, timeouts, retry counts, rates, distances, and thresholds are explicitly labelled as illustrative assumptions unless a supplied source directly supports them. The article remains educational in purpose and does not constitute site-specific engineering advice. Readers are advised to apply the method within the boundaries of their site’s safety procedures and change management processes.
Sources and standards #
- NIST — Engineering Statistics Handbook. In “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method”, source [S1] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- NASA — NASA Systems Engineering Handbook. In “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method”, source [S2] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- NIST — Time and Frequency Division. In “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method”, source [S3] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- OPC Foundation — OPC UA Online Reference. In “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method”, source [S4] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
- OSHA — The Control of Hazardous Energy (Lockout/Tagout), 29 CFR 1910.147. In “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method”, source [S5] supports the attributed terminology or boundary; the warehouse-specific synthesis remains Pearl Gateway editorial analysis.
Revision and editorial note #
The Pearl Gateway Editorial Team prepared “Root-Cause Analysis for Intermittent Automation Faults: A Time-Aligned Evidence Method” from the five linked source records. The published guide remains educational and requires site evidence before application.