Six Sigma Hypothesis Testing in Practice: Three Case Studies from Data Comparison to Statistical Decision Making

1. Introduction: Why Do We Need Hypothesis Testing?

In September 2025, a car electronics company, Hengda Electronics (a pseudonym), encountered a typical quality issue on one of its SMT assembly lines. The process engineer changed the reflow soldering temperature profile from "ramp-up" to "soak" and hypothesized that this would reduce the BGA solder joint voiding rate. After producing 200 samples each with the old and new processes, the data showed: the defect rate for the old process was 4.0%, and for the new process, it was 2.5%. It seemed that the new process was better, but the quality manager asked a critical question: "Is the 1.5% difference due to the process change, or is it a false impression caused by sampling variation?"

This question is at the core of every improvement project—whether the observed differences are "real" or "random noise."

The answer lies not in intuition but in hypothesis testing (Hypothesis Testing). Hypothesis testing is one of the most essential statistical tools in the Analyze phase of the Six Sigma DMAIC methodology. It provides quality engineers with a rigorous mathematical framework to determine whether the differences in data are statistically significant. Without hypothesis testing, improvement teams can easily fall into two extremes: either mistaking random variation for "improvement results" or overlooking true critical factors.

This article will systematically demonstrate the complete application path of hypothesis testing in Six Sigma improvement projects through three real-world cases from the manufacturing front line—covering two-sample t-test, ANOVA, and chi-square test—allowing readers to understand the statistical principles and apply them directly in their own projects.

2. Case Study One: Two-Sample t-Test—Is the Process Parameter Optimization Really Effective?

2.1 Background

Hengda Electronics' SMT line had a persistently high BGA voiding rate, fluctuating between 38,000 and 42,000 PPM monthly. The client's ICT (In-Circuit Test) and AOI (Automated Optical Inspection) frequently raised alarms. The Six Sigma project team defined Y as the "BGA solder joint voiding rate (PPM)" in the Define phase, completed MSA confirmation (AOI equipment GR&R = 8.3%, acceptable) in the Measure phase, and identified three candidate factors in the Analyze phase, with the "reflow soldering temperature profile type" being the highest priority factor.

The team designed a comparative experiment: 200 units were produced using the old process (original ramp-up curve), and 200 units were produced using the new process (soak curve), with all other conditions kept consistent. The results were as follows:

• Old Process Group: 8 voids out of 200 units, defect rate 4.0%
• New Process Group: 5 voids out of 200 units, defect rate 2.5%

2.2 Establishing Hypotheses

This is a typical scenario for a two-sample proportion test. The team used Minitab to establish the hypotheses:

• Null Hypothesis H0: p_old = p_new (no difference in defect rates between the old and new processes)
• Alternative Hypothesis H1: p_old ≠ p_new (a significant difference in defect rates between the old and new processes)
• Significance level α = 0.05

2.3 Testing Process and Results

The team calculated the Z statistic to be 0.86, with a p-value of 0.39. Since p-value > 0.05, the null hypothesis cannot be rejected. This means that the existing data is insufficient to prove that the new process is better than the old process—the observed 1.5% difference could entirely be due to sampling variation.

This result was disheartening for the process engineer, but the quality manager was relieved—thankfully, they conducted the hypothesis test; otherwise, directly implementing the "new process" in mass production and finding no change in the defect rate three months later would have been a greater loss.

2.4 Follow-Up Actions

The team did not give up. They reviewed the sample size—200 units were too small for a proportion test. Using power and sample size calculations, they found that to detect a 1.5% difference in defect rates (at α = 0.05 and power = 0.8), each group needed approximately 1200 samples.

The team expanded the experiment to 1500 units per group and retested. This time, p = 0.023 < 0.05, rejecting the null hypothesis and confirming that the new process indeed reduced the voiding rate. The new process was eventually standardized and incorporated into the SOP, reducing the voiding PPM from 40,000 to 18,000.

2.5 Case Insights

This case highlights a very common misconception: many people see numerical differences and assume "effectiveness," ignoring the issue of statistical power (Statistical Power) when the sample size is insufficient. In Six Sigma projects, hypothesis testing is not an optional step but a "mandatory procedure" for verifying improvement effects—without it, you cannot distinguish between signals and noise.

3. Case Study Two: One-Way ANOVA—Are There Differences in Incoming Quality from Three Suppliers?

3.1 Background

In January 2026, a home appliance manufacturing company, Tianyu Appliances (a pseudonym), discovered an abnormal increase in the noise level dispersion of a batch of motors on the assembly line. The motors came from three suppliers A, B, and C, and IQC sampling inspections were all qualified (according to GB/T 2828.1, AQL = 0.65). However, some motors exceeded the noise test limits after being put into production. The quality department suspected differences in incoming quality from the three suppliers, but the batch qualification rates were all between 96% and 98%, making it difficult to judge.

3.2 Data Collection

The team decided to use continuous data—noise levels (dB)—for precise comparison. They randomly selected 20 motors from the recent batches delivered by each of the three suppliers and measured the noise levels under standardized test conditions:

• Supplier A: Mean 68.3 dB, Standard Deviation 2.1 dB
• Supplier B: Mean 71.5 dB, Standard Deviation 3.4 dB
• Supplier C: Mean 67.9 dB, Standard Deviation 1.8 dB

Intuitively, Supplier B had a higher mean and greater variation, but Suppliers A and C were close. The question was: Is there a statistically significant difference between Supplier B and the other two?

3.3 Establishing Hypotheses and Testing

The team used one-way ANOVA (One-way ANOVA):

• Null Hypothesis H0: μA = μB = μC (the noise means of the three suppliers are equal)
• Alternative Hypothesis H1: At least one supplier is different
• α = 0.05

Before conducting the analysis, the team performed three key preliminary tests:

First: Independence Test. Confirm that the samples are independent and that there is no association between batches from different suppliers.

Second: Normality Test. Use the Anderson-Darling test to verify the normality of the three data sets. The results were: Supplier A p = 0.38, Supplier B p = 0.21, Supplier C p = 0.45—all > 0.05, meeting the normality assumption.

Third: Homogeneity of Variances Test. Use the Bartlett test, p = 0.09 > 0.05, indicating no significant difference in variances among the three groups, satisfying the homogeneity of variances requirement for ANOVA.

The ANOVA results were as follows:

• Between-group Sum of Squares (SSB) = 178.5, Degrees of Freedom = 2, Mean Square = 89.25
• Within-group Sum of Squares (SSW) = 562.3, Degrees of Freedom = 57, Mean Square = 9.86
• F statistic = 89.25 / 9.86 = 9.05
• p-value = 0.0004

The p-value is much less than 0.05, rejecting the null hypothesis—there is a significant difference in the noise means of at least one supplier compared to the others.

3.4 Post-Hoc Multiple Comparisons

ANOVA told the team that "there is a difference," but it did not specify which group. The team further conducted Tukey HSD post-hoc tests:

• A vs B: p = 0.002, significant difference
• A vs C: p = 0.88, no significant difference
• B vs C: p = 0.001, significant difference

The conclusion was clear: Supplier B's motors had significantly higher noise means compared to Suppliers A and C, and the standard deviation was also the largest. The team initiated a supplier improvement process.

3.5 Improvement Measures and Results

After providing the analysis results to Supplier B, the team discovered that Supplier B used different batches of magnetic steel materials in production, leading to significant fluctuations in magnetic flux density and increased noise dispersion. Supplier B changed the magnetic steel batch and adjusted the process, reducing the noise mean to 68.1 dB and the standard deviation to 1.9 dB. Follow-up data over the next three months showed that the noise exceedance rate on the assembly line dropped from 4.7% before the improvement to 0.3%.

3.6 Case Insights

ANOVA is a powerful tool for "multiple group comparisons" in Six Sigma projects. Many quality engineers, when faced with more than three sample groups, habitually perform multiple t-tests—this can lead to multiple comparison issues (Family-wise Error Rate inflation). For example, comparing three groups requires three t-tests, inflating the overall α from 0.05 to about 0.14. ANOVA controls the overall α with a single test, which is the standard practice in Six Sigma methodology.

4. Case Study Three: Chi-Square Test—Is There a Relationship Between Defect Types and Shifts?

4.1 Background

In March 2026, a precision injection molding factory, Ruichuang Precision (a pseudonym), reported in its quality monthly report that the defect rate for March had risen from the usual 1.2% to 2.8%, primarily in the phone mid-frame products. The production supervisor believed that the difference was due to operational differences between the day and night shifts, but the night shift supervisor countered that "the products made by the night shift are exactly the same as those made by the day shift." The team decided to let the data speak.

4.2 Data Collection

The team collected nonconforming product data for the entire month of March, categorized by shift and defect type:

4.3 Establishing Hypotheses

This is a typical scenario for a chi-square test of independence (Chi-square Test of Independence):

• Null Hypothesis H0: Defect types and shifts are independent (i.e., the defect distribution is unrelated to the shift)
• Alternative Hypothesis H1: Defect types and shifts are not independent (i.e., the defect distribution is related to the shift)
• α = 0.05

4.4 Testing Process and Results

The team calculated the expected frequencies and the chi-square statistic:

For "Day Shift—Shrinkage": Expected frequency = (60 × 70) / 160 = 26.25 Actual observed value 28, expected 26.25, the difference is small.

However, for "Day Shift—Flash": Expected frequency = (60 × 50) / 160 = 18.75, actual 15; For "Night Shift—Flash": Expected frequency = (100 × 50) / 160 = 31.25, actual 35.

After calculating the contributions of each cell, the chi-square statistic χ² = 5.84, with degrees of freedom (4-1) × (2-1) = 3. Referring to the chi-square distribution table, the critical value χ²(0.05,3) = 7.81. Since 5.84 < 7.81, the corresponding p-value is 0.12 > 0.05.

Conclusion: The null hypothesis cannot be rejected, indicating no statistically significant association between defect types and shifts.

4.5 Deep Dive Investigation

This result was unexpected by the production supervisor. The team then analyzed "why the overall defect rate increased in March?"—comparing March data with January and February using a chi-square test, they found that the shrinkage rate in March was significantly higher than in the previous two months (p = 0.008). Further investigation revealed that the raw material batch was changed at the beginning of March, and the flow properties of the new batch did not match the process parameters well. The team optimized the injection molding temperature and holding pressure through DOE, reducing the shrinkage defect rate from 4.4% to 1.1%.

4.6 Case Insights

The chi-square test is the most essential hypothesis testing tool for handling attribute/count data. In Six Sigma projects, a large amount of quality data is recorded in forms such as "conforming/nonconforming," "defect types," and "defect locations," which cannot be directly analyzed using t-tests or ANOVA. The chi-square test is designed for this purpose. This case also reminds us that hypothesis testing can tell you "where the difference is," but it does not directly tell you "what the cause is"—it is a compass that points the direction, and further root cause analysis is still necessary.

5. Practical Guidelines for Hypothesis Testing in Six Sigma Projects

Through the above three cases, we can summarize six practical points for applying hypothesis testing in Six Sigma improvement projects:

5.1 Determine Data Type Before Selecting Test Method

The first step in data analysis is not to open the software but to understand the data type:

• Continuous Data (such as dimensions, pressure, time, temperature): Consider t-tests (for two groups) or ANOVA (for multiple groups)
• Discrete Data/Count Data (such as nonconforming counts, defect counts): Use chi-square tests or proportion tests
• Ordinal Data (such as rating scores): Use non-parametric methods (Mann-Whitney, Kruskal-Wallis)

5.2 Verify Assumptions Before Testing

Each hypothesis testing method has its own assumptions, and ignoring these can lead to invalid conclusions:

• t-tests and ANOVA require data to be approximately normally distributed (the Central Limit Theorem can relax this requirement for sample sizes > 30)
• ANOVA also requires homogeneity of variances among groups (verified using Bartlett or Levene tests)
• Chi-square tests require expected frequencies to be not too small (generally, all expected frequencies ≥ 5)

5.3 Sample Size Determines Test Power

As demonstrated in Case Study One, insufficient sample size can reduce the statistical power (Statistical Power) of the test, leading to "real differences not being detected." It is recommended to perform power analysis in the Analyze phase of Six Sigma projects to determine the minimum sample size before collecting data.

5.4 p-Value is Not the Only Measure

The p-value only tells you the probability of observing the current data or more extreme data if the null hypothesis is true. It does not equal the "size of the difference" or the "practical business significance." A small p-value, if the difference Δ itself is small (e.g., defect rate dropping from 0.5% to 0.48%), may be meaningless in business terms. Therefore, when reporting hypothesis testing results, it is essential to also report the effect size (Effect Size) and confidence intervals.

5.5 Correct for Multiple Comparisons

As mentioned in Case Study Two, when comparing multiple groups, use ANOVA rather than multiple t-tests. If pairwise comparisons are indeed necessary, use correction methods such as Bonferroni, Tukey HSD, or Dunnett to control the overall α level.

5.6 Hypothesis Testing is the Starting Point, Not the End

This is one of the most important insights in the Six Sigma methodology. Hypothesis testing tells you "whether a factor has a significant impact on the output," but it does not tell you "why it has an impact" or "how to improve it." After identifying significant factors, it is still necessary to return to engineering judgment, root cause analysis, DOE optimization, and improvement verification in a complete closed loop.

6. Conclusion

In the Six Sigma DMAIC system, hypothesis testing serves as the statistical bridge connecting "data" and "decision-making." It transforms quality improvement from "guesswork" to "data-driven" and from "I think it's effective" to "statistically significantly effective." The three cases in this article—two-sample t-test for process comparison, ANOVA for supplier evaluation, and chi-square test for defect attribution—correspond to the three most common data analysis scenarios in Six Sigma projects.

For every Six Sigma practitioner, mastering hypothesis testing is not only a hard requirement for Green Belt and Black Belt certification but also a core capability to ensure that improvement projects are robust and results are reproducible. When your improvement proposal is questioned by management, "why do you say this solution is effective," hypothesis testing is your most powerful response.

Speak with data, decide with statistics—hypothesis testing is the most worthwhile foundational skill to invest in for Six Sigma improvement projects.

Knowledge code: 6.1.1

Version: v20260725

Author: Quality Think Tank Quality Think Tank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.