Advancing QE Skills (36) | Determining Whether Two Sets of Data Differ — Test Selection Table, Minimum Sample Size Calculation, and Equivalence Testing

A precision injection molding company changed the mold temperature and measured the critical dimensions (specification 12.00±0.03) of 12 products before and after the change. The mean before the change was 12.008 with a standard deviation of 0.0091; the mean after the change was 12.002 with a standard deviation of 0.0088. The quality engineer (QE) wrote in the improvement report: "The dimension center shifted down by 0.006mm, and the capability improved." During the management review, the customer engineer asked one question: "Is this 0.006mm shift due to the mold temperature change or sampling variation?" The report did not include any tests or confidence intervals, and the QE could not answer.

A more common and opposite pitfall is: another company conducted a two-sample t-test, with p=0.03, and immediately declared the improvement effective. However, the actual difference was only 0.0004mm, which is less than the resolution of the measuring instrument. These two incidents highlight the same issue: whether a difference exists (statistical judgment) and whether the difference is worth acting on (engineering judgment) are two separate questions, and neither can be answered without the other.

1. Principle: What Does the p-Value Really Answer?

The approach of hypothesis testing is to first establish a null hypothesis H0 (e.g., the means of two groups are equal, the defect rates of two groups are equal), and then see how likely the data in hand would occur if H0 were true. This probability is the p-value.

Three key points to remember:

1. The p-value is not the probability that H0 is true. It is the probability of observing the current or more extreme results if H0 is true. A p=0.11 means "if there is no difference, there is an 11% chance of drawing such two groups," not "there is an 11% probability of no difference."
2. α and β are two types of errors with very different costs. α=0.05 means "mistaking noise for improvement" (wasting money, going in the wrong direction); β means "mistaking a real difference for noise" (missing the true cause, the problem continues to flow to the customer). In quality scenarios, the cost of β is usually higher, so the power 1−β is generally set to 0.8, and for safety/regulatory characteristics, it is set to 0.9.
3. Non-significant ≠ No difference. When the sample size is insufficient, the test simply "lacks the ability to distinguish," not "proves sameness." To prove "no impact from the change," an equivalence test must be conducted.

2. Test Selection: Identify Data Type, Then Choose the Test

Choosing the wrong test is worse than not doing a test at all, as it provides a seemingly professional but incorrect conclusion.

Three common mistakes to avoid:

• "Measuring the same batch twice" is paired, not independent. Treating it as independent samples will overestimate the variance and bury real differences.
• Proportions cannot be tested with a t-test. Defect rates are 0/1 data, and the assumptions for a mean test do not hold.
• To compare variability, use variance tests. To determine if the new process is more stable, comparing means is irrelevant.

3. Sample Size: Calculate from the Desired Difference, Not Just 30

The approximate formula for the sample size per group in a two-sample mean test is:

n ≈ 2 × (z₁₋α/₂ + z₁₋β)² × σ² ÷ δ²

where δ is the difference you want to detect (it must come from engineering significance, not the difference already present in the data), and σ is the process standard deviation. When α=0.05 (two-tailed) and power 1−β=0.8, the coefficient is approximately 15.7, so:

For the company mentioned at the beginning: σ=0.009, if they want to detect a 0.005mm shift (δ/σ=0.56), they need about 51 pieces per group, but they only took 12 pieces. Using n=12, the minimum detectable difference is (1.96+0.84)×0.009×√(2/12) ≈ 0.0103mm—larger than the 0.006mm they claimed. The report should state "The minimum detectable difference in this test is 0.010mm," not "No significant difference."

The sample size for proportion tests is much larger because detecting defect rates requires a large number of "non-occurrences":

When the sample size is insufficient, there are three correct approaches: switch to a paired design (reduce σ from 0.009 to 0.004, with the same δ=0.005, only about 11 pieces per group are needed), switch to measuring continuous data (continuous data requires fewer samples than count data), and clearly accept a lower power and document it in the report. The wrong approach is: start the test, and if the sample size is insufficient, add more data.

4. Judgment: Three Steps, Significance is Just the First

Relying solely on the p-value will inevitably lead to problems. With a sufficiently large sample, a 0.0004mm difference can have p<0.05; with a sufficiently small sample, a 0.02mm difference can have p>0.05. Therefore, the judgment process should follow three steps:

1. Pre-determine the threshold. Write the engineering significance threshold Δ in the test plan (in this example, Δ=0.005mm, based on the combination of fit tolerance and instrument resolution).
2. Calculate the difference and 95% confidence interval. The interval provides the range where the difference might fall.
3. Three-step judgment:
   • If the entire interval is outside Δ and p<α → the difference is significant and worth acting on.
   • If the interval crosses ±Δ → the evidence is insufficient; increase the sample size or change the measurement method; do not write "improvement is effective."
   • If the entire interval is within ±Δ → the difference is less than the engineering significance, and it can be concluded that "no action is necessary," but this is an equivalence conclusion and requires support from the test in section 5.

5. To Prove "No Difference," Conduct an Equivalence Test

Conclusions such as "no impact from the change" or "the new supplier is consistent with the old one" cannot be derived from a regular t-test—non-significance may simply be due to an insufficient sample size. The correct approach is equivalence testing (TOST): pre-set equivalence boundaries ±Δ, conduct two one-sided tests, and check if the 90% confidence interval is entirely within ±Δ.

Using the data from the beginning: the difference is 0.006mm, the standard error is 0.00367, and the 90% interval is approximately (−0.0003, 0.0123). The upper limit 0.0123 exceeds Δ=0.005, so equivalence cannot be concluded—the conclusion for this data set can only be "insufficient evidence," meaning neither improvement nor no change can be claimed.

This criterion is particularly valuable in supplier changes, material substitutions, and process simplification reviews: when the customer asks "Does the material change have an impact?" without a TOST conclusion, the only option is to add more verification batches.

6. Common Pitfalls

1. Claiming p≥0.05 as "proof of no difference". This is mistaking "no evidence" for "evidence of no difference." Either report the minimum detectable difference or conduct a TOST.
2. Continuously adding samples until significance is achieved. Each additional look is another judgment, and α will continue to inflate, making the final p=0.04 unreliable. The sample size must be determined in advance; interim assessments should use sequential designs.
3. Treating paired data as independent samples. Comparing before and after on the same batch or two measurements by the same operator are paired; treating them as independent will double the variance and miss real differences.
4. Multiple pairwise comparisons without correction. Conducting 15 pairwise t-tests on 6 groups increases the overall error rate to about 54% when α=0.05. Either conduct ANOVA followed by Tukey, or set the single-test threshold to 0.05/15≈0.0033.
5. Reporting only p, not the difference, interval, and n. Readers cannot determine whether the "significant" result is 0.02mm or 0.0004mm, and thus cannot decide whether to invest in the change.

7. Self-Check List

• Data type matches the test method (mean, proportion, count, variability, paired), and t-tests are not used for defect rates.
• The engineering significance threshold Δ is determined and documented in writing before sampling, with clear justification (tolerance, instrument resolution, customer requirements).
• Sample size is calculated based on δ/σ, and the report provides the minimum detectable difference, not just "30 pieces were taken."
• Assumptions are verified and recorded: normality (Anderson-Darling p>0.05), homogeneity of variances (Levene), and independence of sampling.
• Conclusions include the difference, 95% confidence interval, p, n, method, and multiple comparison correction approach.
• For conclusions stating "no difference" or "no impact from the change," TOST results and equivalence boundaries are provided.

Determine the size of the difference first, then discuss significance.

Knowledge code: 6.1.1

Version: v20261028

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.