Is the Improvement Valid Just by Comparing Before and After Data? —— Five Steps to Confirm Effectiveness Using the Seven QC Tools

"After introducing new tooling, the defect rate dropped from 3.2% to 0.8%, indicating effective improvement." The report was submitted, the customer signed off, and the project was closed. Three months later, the defect rate for the same process quietly returned to 2.9%, and the customer returned the report with just one question: What made you say it was effective back then?

This kind of rework is quite common in the quality department. The improvement itself is often genuine, but the issue lies in the "effectiveness confirmation" step—most people reduce it to comparing two numbers before and after the improvement, which is the least effective way to prove improvement.

1. Why "Before and After Comparison" Cannot Prove Improvement Effectiveness

There are three reasons, any one of which can invalidate the conclusion.

The process is inherently variable. The difference between 3.2% and 0.8% might just be two samples within the normal range of process variation. Using the average of one day or one week to represent the process level, the "effectiveness" is often just luck; if you sample another week, the conclusion might be the opposite.

Other variables are also changing. Material changes, shift changes, equipment maintenance, and order structure changes can all occur simultaneously with your measures. Looking only at the result numbers cannot attribute the changes to your measures—this is precisely what the customer will question when reviewing the report.

The criteria may have changed. Did the inspection criteria become stricter during the improvement period? Did the judgment standards change? Were reworked products counted as repaired products? Any change in criteria can make the numbers look better, but the better numbers do not reflect the process.

Therefore, effectiveness confirmation must address three questions: Has the process really changed (beyond normal variation)? Is this change due to our measures (causality)? Can it be sustained (next month)? The value of the Seven QC Tools lies in their ability to answer each of these questions.

The "causality" question is the easiest to overlook. A practical method is to conduct a turn-on/turn-off verification when conditions permit—temporarily stop the measures and see if the metrics rebound; or implement the measures on only a portion of the machines and compare the differences between the lines. For projects where such verification is not possible, the report must clearly state the basis for causality (time alignment plus mechanism analysis) rather than assuming it.

2. Five-Step Method for Effectiveness Confirmation

Step 1: Check Sheet — Align the Criteria Before Comparing.

Align the data collection methods for the baseline period and the improvement period: the same process scope, the same inspection method and judgment criteria, the same sampling frequency and sample size, and the same statistical criteria (whether defects include rework, whether they include trial production). Any discrepancy must be noted and explained in the report, rather than directly comparing two numbers. This step is the least "technical" but can easily invalidate the entire report.

Step 2: Stratification — Analyze Layers Before Looking at the Whole.

Do not look at the overall numbers; instead, draw a line for each shift, machine, model, and material batch. In real scenarios, improvements often only take effect under certain conditions: the day shift improves but the night shift does not (lack of implementation); Model A improves but Model B does not (measures only cover part of the range); one machine improves while another remains the same (tooling only changed on one machine). By stratifying, the boundaries of the measures become immediately apparent, and the report gains direction for the next steps.

Step 3: Control Chart — Use Time to Speak, Evaluate Both Center and Variation.

Connect the data from before and after the improvement into a time series, using either an I-MR chart or an Xbar-R chart. Look at four signals: whether the center line has moved, whether the range or standard deviation has narrowed, whether the points before and after the improvement fall outside each other's control limits, and whether there is a continuous shift. Do not rush to conclusions if there are fewer than 15-20 sample points; extend the observation period or supplement with hypothesis testing (t-test, Mann-Whitney test) to ensure that "significance" is statistically supported.

Step 4: Histogram + Pareto Chart — Analyze Distribution Shape and Problem Structure.

The histogram answers "Has the process shape changed?": effective improvements show an overall shift in the distribution and a narrowing, indicating reduced variation. If the distribution shifts but does not narrow, it suggests that there are still sources of variation that need to be addressed. If a bimodal or long-tailed distribution appears, it indicates that the improvement only covers part of the population. The Pareto chart answers "Has the problem structure changed?": if the top issue before the improvement was shrinkage and the top issue after the improvement is a different one, it indicates that the main problem has been resolved. If the top issue remains the same, the previous improvement was likely due to temporary measures, not a fundamental solution.

Step 5: Write the Conclusion into Standards — Confirm Sustainability.

The final step is not just "confirmation" but "standardization": write the effective parameters into the work instruction, include the monitoring items in the control plan, and clearly define the response plan for anomalies. Set a maintenance observation period (e.g., 4-8 consecutive weeks or 30 consecutive batches). If the metrics do not return to the original levels during this period, the project can be considered truly closed. Failing to do this step significantly increases the probability of reverting to the original state after three months. If the metrics rebound during the maintenance period, the correct approach is to return to Step 4 and re-evaluate the problem structure, rather than quietly extending the observation period and writing the conclusion as "basically effective."

3. Three Common Misuses

Concluding Based on Average Numbers Alone. Averages hide distribution and trends; at a minimum, look at the histogram and control chart.

Declaring Effectiveness with Too Few Samples. With small sample sizes, it is impossible to determine the distribution shape, and the conclusion cannot withstand scrutiny.

Mixing Trial Production, Temporary Measures, and Stricter Inspection Periods into the Improvement Period. Unclear criteria are equivalent to creating a favorable result for yourself. Once the customer reviews the original data, the report will not hold up.

4. One-Sentence Summary

Effectiveness confirmation does not rely on "numbers getting smaller," but on "changes exceeding variation, causality pointing to measures, and conclusions being written into standards"—the role of the Seven QC Tools is to help you clarify these three points, not to make the report look prettier.

Improvement effectiveness does not depend on comparing two numbers before and after, but on clarifying three things: variation, causality, and standardization.

Knowledge code: 5.2.4

Version: v20261006

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously enhance their quality capabilities.