How Long Have Averages Fooled You? — Five Practical Steps of Stratification to Identify the True Source of Problems

1. Why Does the Customer Still Complain When the Nonconforming Rate is 1.8%?

In a stamping workshop of an automotive parts company, the same scene plays out at every monthly quality meeting: the quality manager flips to the slide showing "This month's nonconforming rate is 1.8%, the same as last month," and the meeting room remains calm, the meeting ends. However, in the same month, the customer returned goods twice, both times due to excessive burrs. The workshop supervisor feels wronged: the report clearly states that the nonconforming rate is within the control range.

Quality Manager Li decided to get to the bottom of it. He asked the inspector to export all 856 nonconforming product records from the past two weeks and to statistically analyze them by shift, machine, mold, material batch, and operator. The results made his back cold: the nonconforming rate for the day shift was 1.1%, and for the night shift, it was 3.2%; the nonconforming rate for Press No. 3 was 6.8%, while the average for the other machines was less than 1%; when combining the "night shift" and "Press No. 3" dimensions, the nonconforming rate for the night shift on Press No. 3 was as high as 11.5%—over 60% of the workshop's nonconforming products were concentrated in this single intersection.

The overall nonconforming rate of 1.8% was not fabricated, but it diluted the truth. The average number smoothed out all differences, making it impossible to see where the problem really lies. This is the most common and hidden cognitive trap in quality management: every number on the report is true, but the conclusions drawn from them can be entirely wrong.

Averages can mislead decision-making in three typical ways:

1. Differences within groups are smoothed out, mixing good and poor groups, making it impossible to distinguish them.
2. Imbalanced sample structures, where one group has a disproportionately large amount of data, skewing the average.
3. Outliers distorting the average, where a few extreme data points make the average inaccurate. To uncover the false impressions created by averages, one relies on the most straightforward and often overlooked tool among the seven QC tools—stratification.

2. Stratification: The Skill of "Separating Data" for Observation

Stratification, also known as the stratification method, is one of the seven QC tools (the old seven techniques) and is a prerequisite for the other six tools. Its principle is simple: categorize data by different types and then observe them separately. Before drawing a Pareto chart, you must stratify; before creating a histogram, you must stratify; and after a control chart alarms, you must stratify—without stratification, all subsequent analyses may be based on incorrect premises.

Why is stratification necessary? Because any set of quality data is the result of multiple factors working together. If data from different shifts, machines, and batches are mixed, it is like blending "normal fluctuations" and "abnormal signals" into a pot, where they cancel each other out and dilute the differences. For example, if Machine A has a 95% pass rate and Machine B has a 75% pass rate, the combined pass rate is 85%—looking at this number alone, you cannot see that Machine A is actually very stable, nor that Machine B is out of control. The role of stratification is to first break down the mixed data into their respective "homogeneous groups" to make the differences visible.

An even more extreme case is the "Simpson's Paradox": the conclusion drawn from combined data is completely opposite to the conclusions within each stratified group. A famous case in statistical history involved two hospitals treating the same disease. When the data were combined, Hospital A had a higher cure rate; however, when patients were stratified by the severity of their condition, both the mild and severe groups had higher cure rates at Hospital B—the reason was that Hospital A treated a larger proportion of mild cases, which skewed the overall numbers. The same reversal can occur in quality scenarios: a supplier may have a lower overall nonconforming rate, but when stratified by material type, it may have the highest nonconforming rate for key materials. Without stratification, you might not even realize that the conclusion is reversed.

The dimensions for stratification are typically summarized as "people, machines, materials, methods, environment, and measurement," supplemented by time, suppliers, and batches. How to stratify depends on the specific questions you want to answer:

A special reminder about the "measurement" dimension: it is the easiest to overlook but often provides false clues. Sometimes an "increase in the nonconforming rate" is not due to a decline in product quality but to a change in inspectors, uncalibrated measuring tools, or a change in the judgment criteria. Exclude measurement factors first, then investigate production factors—do not reverse the order.

3. Five Practical Steps of Stratification

Returning to Li's stamping workshop, stratification can be broken down into five steps, each with a clear output. Follow these steps to implement stratification effectively.

Step 1: Define the specific question to answer. Stratification is not just about "categorizing data," but about categorizing data with a specific question in mind. The question should be specific, such as "Which dimension is the nonconforming rate concentrated in?" rather than a general "Why are there nonconformities?" Li's question was: "Where are the burr nonconformities concentrated? Is it a people issue, a machine issue, or a material issue?"

Step 2: List all candidate dimensions and screen them based on difference hypotheses. First, list all possible dimensions using the "people, machines, materials, methods, environment, and measurement" framework, and then ask yourself: Is there a possible difference between groups in this dimension? If a machine is brand new and all operators use the same method, the "machine" and "people" dimensions can be set aside, focusing on the most likely dimensions. During screening, you can score each candidate dimension: dimensions with a high likelihood of differences and recent changes (such as mold changes, personnel changes, material changes, parameter adjustments) should be prioritized. Li's candidate dimensions were five: shifts, machines, molds, material batches, and operators. The molds and material batches were marked as "key suspects" because they had recently been changed. Note: it is better to include more dimensions than fewer. Dimensions that do not show differences initially should be retained until the data is analyzed, rather than being eliminated based on initial feelings.

Step 3: Design data collection to ensure sufficient samples in each stratum. The biggest fear in stratification is "realizing the data is incomplete when you want to analyze it." If the inspection records do not include machine numbers, shifts, or material batches, no one can stratify the data afterward. Therefore, data collection must be designed in advance: the record form must include all fields you plan to use for stratification, and ensure that each stratum has enough samples—generally, at least 30 data points per stratum are recommended to avoid overly random conclusions. Li had the inspector fill in all the fields, and each of the 856 nonconforming products could be traced back to the machine, shift, and batch.

Step 4: Stratify and compare, first by single dimension, then by cross-dimension. This is the core step. First, stratify by each dimension separately, using the nonconforming rate or the number of nonconformities for comparison, and identify dimensions with significant differences. Then, cross-stratify two suspicious dimensions to pinpoint the problem. Li first stratified by single dimensions and obtained a clear comparison table:

The single-dimension results pointed to shifts and machines, but it was still unclear whether the issue was with the "night shift overall" or "Press No. 3 overall." Therefore, Li performed a "shift × machine" cross-stratification: the nonconforming rate for the day shift on Press No. 3 was 1.2%, and for the night shift on Press No. 3, it was 11.5%, while other combinations were around 1%—the problem was immediately pinpointed. Cross-stratification is the most valuable action in this step: single-dimension stratification can only tell you "the night shift is poor" or "Press No. 3 is poor," but cross-stratification reveals the true problem coordinate as "night shift × Press No. 3."

Step 5: Verification and Action. After identifying the stratum, immediately conduct a root cause analysis (using a fishbone diagram or 5Why method) and develop measures. Then, use stratification again to verify: after implementing the measures, re-stratify and re-statistically analyze to see if the nonconforming rate for "night shift × Press No. 3" has truly decreased. Li ultimately found the root cause: the guide pillars of Press No. 3 were worn out, and the wear increased when the temperature was low and lubrication was insufficient during the night shift, leading to more burrs. After replacing the guide pillars and adjusting the lubrication frequency during the night shift, re-stratification showed that the nonconforming rate for Press No. 3 dropped from 6.8% to 0.9%, and the overall nonconforming rate dropped from 1.8% to 0.7%. Customer returns disappeared. A key detail to note: during the verification period, do not just focus on the overall nonconforming rate; continue to stratify by the original dimensions for at least two weeks— if the overall nonconforming rate decreases but the "night shift × Press No. 3" cell does not show significant improvement, it indicates that the measures did not address the root cause, and you need to return to Step 4 to re-investigate, rather than being misled by the improvement in the overall numbers.

These five steps determine the direction in the first two steps, the data quality in the third step, the analysis depth in the fourth step, and the improvement loop in the fifth step—skipping any step will compromise the effectiveness of stratification.

4. Five Common Misunderstandings of Stratification

Misunderstanding 1: Choosing the wrong dimensions, stratifying in vain. The most typical mistake is "habitual stratification": always stratifying by shift or machine, without adjusting dimensions based on the problem's characteristics. In an electronics factory, the welding nonconforming rate remained high for two months, and the quality department stratified by shift and line without reaching a conclusion. Later, someone suggested stratifying by "solder batch," and they immediately identified a batch of solder paste—the problem was in the material from the beginning, not in the people or the equipment. The dimensions for stratification must serve the specific question: asking "Is it a material issue?" but only stratifying by shift will not yield an answer. Solution: before each stratification, list all candidate dimensions in the "people, machines, materials, methods, environment, and measurement" framework, and screen them based on difference hypotheses, rather than using a fixed routine.

Misunderstanding 2: Stratifying without action, treating analysis as a conclusion. Many teams perform beautiful stratification analyses, but after the PPT presentation, the problem remains unchanged. Stratification is just a means; after identifying the significant strata, root cause analysis and corrective actions must follow, otherwise, it is "using analysis to replace action." The only criterion for evaluating the effectiveness of stratification is whether the numbers in the problem coordinates have decreased.

Misunderstanding 3: Over-stratifying, leading to insufficient sample sizes and inaccurate conclusions. If dimensions are too finely divided, each cell may contain only three to five samples, and a single anomaly can multiply the nonconforming rate, leading to noisy conclusions. General principle: each stratum should have at least 30 samples; if the sample size is insufficient, combine similar strata or extend the data collection period, rather than rushing to a conclusion.

Misunderstanding 4: Focusing only on single dimensions and ignoring cross-effects. This is the most hidden pitfall. Individually, the day shift and night shift may not seem poor, but the "night shift × first hour after mold change" combination may be extremely poor—such interaction effects are invisible in single-dimension stratification. When the data volume is large, always perform cross-stratification on suspicious dimensions; when the data volume is limited, at least cross-verify the "two most suspicious dimensions."

Misunderstanding 5: Not re-measuring stratification results, assuming the stratification was wrong when measures are ineffective. After implementing measures, re-stratify and compare to confirm the effectiveness of the improvement. Some companies implement measures but see no results, and instead of reflecting on the measures themselves, they doubt the initial stratification—often, the stratification was not wrong, but the measures did not truly address the root cause or were not fully executed. Re-measurement is the final step in the stratification loop; without it, the previous four steps are in vain.

5. In a Nutshell

Stratification is the first step in making data speak the truth—separate the data first, then draw conclusions, and don't let averages do your thinking for you.

Data must be stratified for the truth to emerge.

Knowledge code: 5.2.4

Version: v20260827

Author: Quality Think Tank Quality Think Tank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.