Advancing QE Skills (38) | Analysis of Variance (ANOVA): Multi-Level Comparison and Interaction Effects

A manufacturing company's welding line was experiencing excessive spatter. The process engineer planned two actions: reducing the welding current and changing the electrode. Before making the changes, he verified with 16 test welding data points, conducting two pairwise comparisons—once by current level, p=0.62; and once by electrode A/B, p=0.31. Both comparisons were not significant, so the report stated, "Neither current nor electrode has a significant effect on spatter; the cause needs further investigation." Consequently, he shifted focus to the welding wire batches, spending two months on it. Meanwhile, the quality engineer rearranged the 16 data points into four combinations and drew an interaction plot: the data points fell into four corners, forming a clear cross—electrode A performed well at low current, while electrode B performed poorly, and the situation was reversed at high current. The two "no effect" main effects were actually the result of two opposite effects canceling each other out, leaving an average value. This led to a completely different approach to process improvement: not changing the welding wire, but locking the current and electrode as a pair.

1. Key Principles: Variance Decomposition, Two Degrees of Freedom, One Hard Limit

ANOVA does not ask "how much the means differ," but rather "what explains the total variation." The total sum of squares of 16 observations is broken down into several parts. Each part is divided by its own degrees of freedom to get the mean square, and the ratio of two mean squares is the F value:

• F = [SS_factor / (k−1)] / [SS_error / (N−k)]

A single factor answers whether "three or more levels are the same level"; a two-factor design breaks down the total variation into four parts: the main effect of A, the main effect of B, the interaction of AB, and error.

The degrees of freedom for the interaction term are (a−1)(b−1), and it must have replication. In a 2×2 design with 4 replications per cell, the total degrees of freedom are 15 = 1 (current) + 1 (electrode) + 1 (interaction) + 12 (error). If only 1 replication is done per cell, the interaction degrees of freedom are 1, and the error degrees of freedom are 0—there is no place for the interaction term, so it must be merged with the error. In this case, "interaction is not significant" is a result of the design, not the data. This is a hard limit of a two-factor design without replication, which cannot be overcome by changing software options.

When the interaction is significant, the main effects lose interpretability. Three criteria are fixed:

1. Interaction term p<0.05 and interaction sum of squares >30% of the total sum of squares → Stop reporting main effects, analyze each cell separately, and conclude in a conditional format such as "Under the condition B=A, the effect of X is..."
2. Interaction is not significant → remove the interaction term and recalculate, merging its degrees of freedom into the error. The power of the main effect test will increase.
3. Interaction is significant but <10% of the total sum of squares → main effects can be used to draw trend conclusions, but the report must include an interaction plot and not just a single average difference.

Sample size is calculated based on "what size difference needs to be detected." For a single factor, α=0.05, and power 0.8, the number of replications per group (approximate values) required to detect a difference of "1 standard deviation between the maximum and minimum levels" is as follows:

For a difference of 0.5σ, the required replications are approximately four times the above table; for a difference of 2σ, they can be halved. The power of the interaction test in a two-factor design is naturally lower than that of the main effects: at least 3 replications per cell are needed to detect interaction, and 4-5 replications per cell are required to detect an interaction at the 1σ level. Most cases where "a two-factor design did not detect interaction" are due to only 1-2 replications per cell.

2. Practical Steps: Five Steps to Implementation

Step 1: Determine levels and replications, calculate before doing. Write the number of replications in the experimental plan according to the table above, rather than asking "is it enough" after the data is collected. When the number of levels exceeds 5, clearly define whether each level represents a truly executable process state—more levels mean thinner samples per group.

Step 2: Block known sources of interference to prevent them from falling into the error. Products produced from the same batch of materials, by the same operator, or during the same time period have higher correlation than random. Criterion: if the range of a known source of interference exceeds 2 times the within-group standard deviation, it must be set as a blocking factor (block) or balanced within each cell. Otherwise, the error is inflated, and the real factor effects are drowned out—such "undetected" effects are not due to the absence of effects but to the design allowing the error to consume the effects.

Step 3: Check all three assumptions together; missing any one will discount the conclusion. ① Homogeneity of variances: Levene's test p>0.05; a practical criterion is the ratio of the maximum to the minimum standard deviation across cells ≤2. ② Normality of residuals: the residual probability plot does not show significant curvature or a funnel shape. ③ Independence: randomize the experimental order, and do not count the same sample multiple times. Only if all three assumptions are met should the conventional F test be used.

Step 4: Read the ANOVA table in the order of interaction → main effects → post hoc. The complete table for the 16 test welds is as follows:

Reading: the interaction term sum of squares is 196, accounting for 94% of the total sum of squares, and the F value far exceeds the critical value of 4.75 (α=0.05, df=1,12). At this point, the statement "electrode p≈0.03, significant" cannot be written alone—it means "the average spatter difference between the two electrodes is 1 point," but the interaction term reveals the true structure as ±7 points. This is precisely where misinterpretation begins.

Step 5: Use Tukey HSD for post hoc tests, not individual t-tests.

• HSD = q(α, k, df_error) × √(MSE / n)

In this example, k=4, df=12, q=4.20, MSE=0.667, and n=4: HSD = 4.20 × 0.408 ≈ 1.71. The means of the four cells are 8, 16, 15, and 9, and differences exceeding 1.71 are considered significant. The result is two clusters: {8, 9} and {15, 16}. The difference within clusters of 1 is not significant, while the difference between clusters of 7 is significant. The conclusion can only be written as "the spatter level is determined by the combination of current and electrode, with each combination forming a cluster," not "electrode affects spatter," which would lose all the information. When there is a control group (such as the current process), use Dunnett's test, comparing only with the control, which has a more relaxed critical value.

3. Common Pitfalls

Pitfall 1: Using multiple pairwise t-tests instead of ANOVA. For 4 levels, 6 comparisons are needed; for 5 levels, 10 comparisons are needed. When α=0.05 for each comparison, the probability of at least one false positive in 10 comparisons is 1−0.95¹⁰≈40%. ANOVA uses a unified error mean square and a single F test to guard against this, and post hoc tests control the familywise error rate.

Pitfall 2: Reporting interaction effects with only 1 replication per cell. Software may either report an error or quietly treat the interaction term as error, producing a seemingly reasonable F value. The judgment is simple: check if the error degrees of freedom equal "total number of observations − degrees of freedom of each main effect − 1." If it equals 0, it means the interaction was not estimated separately.

Pitfall 3: Reporting only averages when the interaction is significant. The statement "current has no average effect" is invalid when interaction exists, as it is good under electrode A and bad under electrode B. This is especially dangerous in cross-functional communication: the process team may abandon parameter optimization upon hearing "no effect," while the R&D team may blame the material. This is the most common starting point for disputes among the three parties.

Pitfall 4: Treating "not significant" as "no difference." Not significant only means "the difference was not detected with this sample size and variability," which could be due to a real difference existing but the sample size being insufficient. To claim "equivalence," equivalence testing or power calculation must be performed. The report should state the "minimum detectable difference" rather than just the p-value.

Pitfall 5: Forcing conventional F tests when variances are clearly unequal. When the ratio of standard deviations across cells >2 and the sample sizes are unequal, the conventional F test becomes too lenient. Use Welch's ANOVA and Games-Howell for post hoc tests, or apply logarithmic or Box-Cox transformations before analysis, and note the transformation method in the report, otherwise others will not be able to reproduce the same results.

4. Self-Check List

• The experimental plan specifies the number of levels, replications per cell, and the minimum detectable difference (multiple of σ)?
• The number of replications per cell ≥2 (≥3 if checking interaction), and the error degrees of freedom ≥10?
• Known sources of interference, such as batches, machines, and operators, have been blocked or balanced and not mixed into the error?
• The interaction term is checked first: when p<0.05 and it accounts for >30% of the total sum of squares, the conclusion is rewritten in a conditional format, and main effects are not interpreted separately?
• Tukey/Dunnett is used for post hoc comparisons, not multiple pairwise t-tests; the report includes the results of variance homogeneity and residual checks?

When the interaction is significant, the main effects are just the averaged values of offsetting effects.

Knowledge code: 6.4.1

Version: v20261102

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.