Advancing QE Competence (30) | Reliability Verification Testing and Growth Management

1. A "Qualified but Rejected by the Customer" Qualification Test

A manufacturing company conducted a reliability qualification test before mass-producing a new platform product: 8 sample machines were tested, accumulating 20,000 machine hours, during which 3 failures occurred (power board crash, loose terminal, software watchdog reset). The QE calculated the observed MTBF as approximately 6,667 hours using "total test time ÷ number of responsible failures," and since the design target was 3,000 hours, they concluded that the "reliability criteria were met" and issued a report.

During the customer review, only three questions were asked: What is the inspection lower limit of this test plan? What is the discrimination ratio? What are the producer's risk and the user's risk? The report could not answer any of these questions. More troubling was that the discussion on whether "loose terminal" should be classified as a responsible failure or an assembly issue only took place after the test was completed and the results were seen. The report was rejected, and the test had to be redone according to a new plan, delaying the project by six weeks.

The problem was not in the data but in treating "verification" as "measurement." The essence of reliability verification testing is a hypothesis test: its conclusion strength is entirely determined by the inspection lower limit, discrimination ratio, and both parties' risks set before the test, not by a post-test calculated observed value.

2. Key Principles: Growth Testing and Verification Testing Are Two Different Things

（1）Different Purposes, Different Rules.

The purpose of reliability growth testing (TAAF) is "discover—analyze—improve—retest," allowing design changes during the test and using a growth model to measure the improvement rate. The purpose of reliability verification (qualification/acceptance) testing is to "prove that the product has reached the specified level," prohibiting design improvements during the test and only allowing repairs according to the original design. Mixing these two processes means that the word "verification" loses its meaning.

（2）Statistical Logic of Verification Testing.

Under the assumption of an exponential lifetime, the number of failures within the cumulative test time T follows a Poisson distribution, with the expected number of failures μ = T/θ. Given the inspection lower limit θ1 (the lowest acceptable MTBF), the discrimination ratio d = θ0/θ1, the producer's risk α, and the user's risk β, the acceptance criteria are derived from two equations:

• When the true value θ = θ1 (unacceptable level), the acceptance probability = β;
• When the true value θ = θ0 = d·θ1 (acceptable level), the acceptance probability = 1 − α.

The solution is a set of "total test time T (in multiples of θ1) + acceptance number of failures c." Without this set of parameters, the test is left with only a post-test point estimate.

（3）Confidence Lower Limit (Criteria for Few Failures or Default Plan).

For a time-truncated test with r responsible failures, the one-sided confidence lower limit for MTBF, θL, is given by θL = 2T / χ²(1−α, 2r+2). For zero failures, it simplifies to θL = T / 2.3026 (90% confidence). This formula is the most practical: to prove that MTBF ≥ 1,000 hours (90% confidence) with zero failures, the cumulative test time must be 2,303 hours—it only depends on the cumulative machine hours, not on the number of machines tested.

3. Practical Steps (5 Steps, Each with Criteria)

Step 1: Fix the 4 Parameters in the Test Plan. θ1 should be the design target value (not the estimated value or the customer's "desired value"); the discrimination ratio d should be between 2 and 3, with a larger d reducing test time but increasing coarseness of discrimination; α and β should each be between 10% and 20%, with 10% for safety-related or customer-mandated items and 20% for general industrial items. Once signed, these parameters cannot be modified during the test.

Step 2: Calculate the Total Test Time and Acceptance Number of Failures. The table below is derived for β = 10% and d = 3 (θ1 is the inspection lower limit):

Example: θ1 = 500 hours, d = 3, α ≈ β ≈ 10%, choose c = 5, then T = 9.27 × 500 ≈ 4,635 machine hours. Testing 8 machines requires continuous operation for 580 hours; if only 3 machines can be tested, it requires 1,545 hours. Criteria: Time periods for disassembly, standby, and test equipment failures are not counted in machine hours. The cumulative effective machine hours must be ≥ T.

Step 3: Pre-define the Criteria for Responsible Failures. The criteria must be signed by both parties before the test and not modified during the test; any necessary modifications must have a written reason and be signed by both parties. It is recommended to solidify the classification criteria as follows:

Step 4: Judge According to the Plan, Not by "Feeling." For a time-truncated plan: when T is reached, if the number of responsible failures ≤ c, accept; if ≥ c+1, reject. If only 1-2 sample machines are available, use a sequential plan, judging according to the acceptance line/rejection line, which can end the test early, but each judgment time point must fall on the specified criteria line in the plan.

Step 5: Use Instantaneous MTBF for Growth Management, Not Cumulative Values. Under the Crow-AMSAA model, the cumulative number of failures N(t) = λ·t^β, cumulative MTBF = t / N(t), instantaneous MTBF = cumulative MTBF ÷ β. Criteria: β < 1 indicates that MTBF is growing; the growth slope α = 1 − β, with 0.3 to 0.5 indicating effective improvement, and < 0.2 indicating growth stagnation, requiring a reopening of root cause analysis. Calculate once at each stage node (e.g., every 500 machine hours), and if the actual curve falls below the planned growth curve and exceeds the confidence band for two consecutive nodes, stop the test for a review, and do not allow "let's see how it goes."

Algorithm for a set of actual data (cumulative machine hours and cumulative responsible failures):

Fitting yields β ≈ 0.65, α = 1 − β ≈ 0.35, falling within the effective growth range; the current instantaneous MTBF = cumulative 100 hours ÷ 0.65 ≈ 154 hours. The "current reliability level" reported to management should be this 154 hours, not 100 hours—it is also still short of the target of 200 hours, requiring further improvement actions.

4. Common Misconceptions

Misconception 1: Judging Qualification Based on Observed MTBF. The observed value is just a point estimate. In the example above, with T = 20,000 and r = 3, the observed value of 6,667 hours has a 90% confidence lower limit of only about 2,994 hours—below the target value of 3,000 hours. The same data set can lead to opposite conclusions when described differently.

Misconception 2: Improving While Verifying. Changing the design during the test means that the data before and after the change are not from the same product, making the test criteria invalid. The test's validity must be redefined or the test must be restarted.

Misconception 3: Post-test Definition of Responsible Failures. Revisiting the classification of failures after seeing the number exceed the plan is the easiest point to be challenged by customers and auditors.

Misconception 4: Zero Failure Plan is the Cheapest. The zero failure plan has the shortest test time (2.30 θ1), but the producer's risk is as high as 53.6%, equivalent to a coin toss. Many companies choose the zero failure plan and then complain that "the test fails," which is actually due to selecting the wrong plan.

Misconception 5: Reporting Current Level Using Cumulative MTBF. During the growth phase, the cumulative value is always lower than the instantaneous value, leading to underestimation if used to judge "current capability." Reporting the cumulative curve of the growth test as the verification conclusion will misjudge the test's validity.

5. Self-Check List

1. Are θ1, θ0 (or discrimination ratio d), α, β, acceptance number of failures c, and total test time T all clearly stated and signed in the test plan?
2. Are the criteria for responsible and non-related failures defined and traceable before the test?
3. Does the cumulative effective machine time meet T? Have time periods for disassembly, standby, and test equipment failures been deducted?
4. Are design changes during the test documented? Have the changes been re-evaluated for test validity?
5. Does the report provide the observed MTBF, confidence lower limit, and judgment conclusion, rather than just the observed value?

Without setting risks before the test, there will be no conclusions after the test.

Knowledge code: 8.2.3

Version: v20261012

Author: QTank QTank is dedicated to providing quality management professionals with systematic knowledge, methodologies, and practical tools to continuously enhance corporate quality capabilities.