Advancing QE Skills (39) | Stratification and Simpson's Paradox: Two Conclusions from the Same Data

At a quarterly supplier review meeting of a manufacturing company, the procurement department presented a summary table: Supplier A's incoming material batch nonconformity rate was 18.5%, while Supplier B's was only 7.5%. According to the 15% red line, Supplier A should be downgraded. However, Supplier A's SQE was not convinced and recalculated the same data by breaking it down into two categories based on order volume: for small batches, Supplier A's rate was 5.0% and Supplier B's was 6.0%; for large batches, Supplier A's rate was 20.0% and Supplier B's was 21.0%. In both categories, Supplier A performed better, yet the overall rate showed Supplier A to be more than twice as bad. The meeting was stuck for half an hour, and the final conclusion was that "the data is problematic."

The data is not the issue; the problem lies in the weighting. Out of 1000 batches, 900 of Supplier A's batches were large (the baseline nonconformity rate for large batches is inherently higher), while 900 of Supplier B's batches were small. The summary rate is a weighted average, and the structure of the orders affects the overall rate. This phenomenon, where each stratum shows a win but the overall result shows a loss, is called Simpson's Paradox. It is not limited to supplier comparisons but is more commonly seen in capability indices—three machines each with a Cpk of over 1.48, but combined, the Cpk is only 1.07, leading to a direct disqualification by the customer during an audit. The same data can lead to two different conclusions, depending on whether the strata are separated and the weights are clearly presented.

1. Principle: Weighted Averages and Variance Decomposition, Reversals Have Only Two Sources

For ratio metrics, reversals come from weighted averages. The summary nonconformity rate ( p ) is calculated as ( p = \sum \left( \frac{n_i}{N} \right) \cdot p_i ), where ( n_i ) is the sample size of the ( i )-th stratum, and ( N ) is the total sample size. To reverse the ranking, two conditions must be met simultaneously: first, the baseline rates of different strata must differ (in this example, 5% and 20%); second, the grouping must be related to the strata composition (Supplier A's orders are concentrated in the high-baseline large batch category). If either condition is missing, the ranking will not reverse—identical baseline rates will yield the same result regardless of weight changes; identical compositions will result in a simple average.

The solution is direct standardization: apply the same set of standard weights ( w^* ) (usually based on the combined structure of the two groups or industry benchmarks) and recalculate ( p^* = \sum w^* \cdot p_i ). In this example, using the combined structure of 0.5/0.5, Supplier A's rate is 12.5%, and Supplier B's is 13.5%, making Supplier A the better choice. The standardized rate is not the "true value," but it is the only consistent basis for horizontal comparison, and the report must specify which set of weights is used.

For measurement metrics, reversals come from variance decomposition. This is a more common pitfall for QEs:

• ( \sigma^2_{\text{total}} = \sigma^2_{\text{within}} + \sigma^2_{\text{between}} )

The denominator of Cpk uses the total standard deviation. As long as the means of different strata do not overlap, the between-strata shift will be included in the denominator. For example, the specification is 10.0±0.6, and the means of three machines are 9.80, 10.00, and 10.20, respectively, with a standard deviation of 0.09 for each machine. The individual Cpk values are 1.48, 2.22, and 1.48, all exceeding the threshold of 1.33. However, the between-strata standard deviation is 0.163, and the combined standard deviation is 0.187, resulting in a combined Cpk of only 1.07—不合格. The between-strata variance accounts for 76.7% of the total variance, and the ratio of between-strata σ to within-strata σ is 1.81.

The criteria for setting the red line are as follows:

A third derived issue: control charts are widened. Mixing data from three machines on a single Xbar-R chart, with control limits calculated based on the combined σ of 0.187, makes the control limits more than twice as wide as those calculated for a single machine (0.09). As a result, all points outside the 3σ range within a machine fall within the control limits, making the chart appear "all normal." The premise for control charts is that the groups are homogeneous, allowing only common causes within a group. Mixing data with different means is equivalent to using between-strata shifts to mimic process variation.

2. Five-Step Operation: From Selecting Dimensions to Defining Reporting Bases

Step 1: List candidate dimensions, screen and stratify. Six commonly used dimensions: machine/line, shift, batch/material batch, customer/order type, operator, time (month/quarter). Two hard constraints: a maximum of 3 dimensions simultaneously (more will fragment the sample size in each cell); for continuous data, ( n \geq 30 ) per stratum, and for count data, at least 5 nonconforming and 5 conforming items per stratum. Screening criteria: retain a dimension if the single-factor ANOVA ( p < 0.05 ) or the between-strata mean difference is greater than 0.5 within-strata σ.

Step 2: Draw graphs before calculating. Use side-by-side box plots to check three things: whether the data is bimodal or multimodal, the ratio of between-strata mean differences to within-strata spread, and whether there are outliers within each stratum. Criteria: if the between-strata mean difference is greater than 1 within-strata σ and the distributions are approximately symmetric, there is a substantial between-strata shift, and proceed to Step 3; if the between-strata mean difference is less than 0.3σ, this dimension does not need further stratification, as it will not change the conclusion.

Step 3: Calculate variance components. Compute the mean and standard deviation for each stratum, use the pooled within standard deviation as ( \sigma_{\text{within}} ), and the standard deviation of the stratum means (weighted by sample size) as ( \sigma_{\text{between}} ). Determine the reporting basis according to the red line criteria. This step can be done with a single command in software, but two things must be manually confirmed: whether the within-strata data is approximately normal (transform if not) and whether the sample size is sufficient for each stratum.

Step 4: Standardize ratio metrics. List the weight table (each stratum's ( n_i ) and proportion) and use the same set of standard weights to calculate the standardized rate, presenting the stratified table alongside the standardized summary table. Criteria: if the ratio of the maximum weight to the minimum weight exceeds 3:1, the unstandardized summary rate cannot be used for horizontal comparison—this ratio is 9:1 in the example, making the summary comparison methodologically invalid.

Step 5: Attribute conclusions to the responsible party. If the between-strata shift dominates (ratio > 0.5) → the issue lies in the consistency of equipment, tooling, and procedures across machines. The action is to align, standardize positioning benchmarks, and lock procedure parameters, with the KPI recorded as "between-strata shift." If the within-strata shift dominates → the issue lies in the parameters of individual machines and incoming material variation, and the action is to return to the process window of the specific machine. The report should always include three fixed components: the stratified table (each stratum's ( n ), mean, σ, Cpk), the variance components table, and the reporting basis statement (combined, weighted, or worst stratum, clearly indicating which one is used).

3. Five Common Misunderstandings

1. Using the arithmetic mean of each stratum's Cpk to represent overall capability. In the example, the average Cpk of three machines is 1.73, but the actual combined Cpk is 1.07. The arithmetic mean is neither weighted nor reflective of the tail risk brought by shifts. Overall capability should either be weighted by production volume or the worst stratum value should be used as the guarantee.
2. Giving Cpk conclusions with insufficient samples. Cpk itself has sampling error: when the true Cpk is 1.33, the half-width of the 90% confidence interval is approximately 0.31 for ( n = 50 ), 0.22 for ( n = 100 ), and 0.16 for ( n = 200 ). For a single stratum with ( n < 50 ), only report the mean and range, not the capability; "all machines are不合格" is often noise, not a signal.
3. Treating Simpson's Paradox as data falsification and pursuing accountability. It is a problem of weighting and composition. The correct action is to standardize and recompute, breaking down the composition ratio, not to investigate individuals. Conversely, when suppliers use "we are cheaper in every category" to resist price cuts, the same set of weights should be used to verify.
4. The other direction of error: stratifying without combining. Insisting on six strata when there is no significant difference between them, with insufficient samples in each stratum, reduces the power to detect a real 0.3σ shift in the overall data. The criteria are clear: if the between-strata σ / within-strata σ < 0.3, combine and report.
5. Combining data to draw control charts. The control limits are widened by between-strata shifts, making the rules for detecting anomalies ineffective. Either draw charts separately for each stratum or eliminate between-strata shifts before combining.

4. Self-Check List

• Are the stratification dimensions no more than 3, with ( n \geq 30 ) for continuous data and at least 5 nonconforming and 5 conforming items for count data in each stratum?
• Are the stratified table and summary table presented together in the report, with the weight column (each stratum's ( n_i ) and proportion) clearly listed?
• Have the between-strata σ / within-strata σ been calculated, and the reporting basis determined according to the red line criteria?
• Has the overall Cpk not been substituted by the arithmetic mean of each stratum's Cpk, ensuring the worst stratum value or production volume-weighted value is used as the guarantee?
• Are control charts and capability analysis based on homogeneous groups, without mixing data with different means on a single chart?

Stratifying shows a win in each stratum, but combining shows a loss—first, look at the weights.

Knowledge code: 6.3.2

Version: v20261104

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.