Advancing QE Skills (11) | Capability Analysis for Non-Normal Data: Transformation, Box-Cox, and Non-Parametric Methods

A mechanical processing company submitted an initial process capability report to a client, focusing on the roundness of shaft components. The data concentration looked good: most parts fell within 8 to 12 μm, with only a few reaching 18 to 22 μm. The quality engineer (QE) calculated a Cpk of only 0.78 using the standard formula, and the client immediately issued a rectification requirement. The process team changed the cutting tools, modified the cooling parameters, and readjusted the clamping method, but after two months of effort, the data distribution remained right-skewed, and the Cpk was still around 0.8. The real issue was not with the process but with the algorithm: characteristics such as roundness, roughness, position, and impurity content, which have a physical lower bound of zero and can only grow in one direction, are naturally right-skewed. Applying the normal distribution formula results in a number that is neither truly high nor truly low. The two most common reactions at the site are to force the process to meet an unattainable Cpk or to use "non-normal data" as an excuse to avoid analysis. Both are incorrect.

1. Key Principles: Where Does the Normality Assumption Lie?

Capability indices are intermediate tools that convert process variation into defect rates. For bilateral characteristics:

Cp = (USL − LSL) / (6σ)

Cpk = min(USL − μ, μ − LSL) / (3σ)

The implicit assumption in these formulas is that the data follows a normal distribution. Only under normality does "μ ± 3σ" correspond to a 0.135% one-sided exceedance rate, and there is a fixed conversion relationship between Cpk and defect rates: Cpk = 1.00 corresponds to approximately 1350 PPM, 1.33 to approximately 32 PPM, and 1.67 to approximately 0.29 PPM.

Once the data is skewed, this conversion chain breaks. The same σ will have a tail probability that does not correspond to the 0.135% of a normal distribution: when the data is right-skewed and the upper specification limit (USL) is tight, the actual exceedance rate will be much higher than the index suggests (the index is falsely high, and the risk is underestimated); when the data is right-skewed but the lower limit is far from the specification, the index will be significantly low (the index is falsely low, and the process is unfairly criticized). The roundness of the shaft components in the aforementioned company falls into the latter category—converting the Cpk of 0.78 to a defect rate results in several thousand PPM, while the actual exceedance rate above the USL is less than 300 PPM.

There are three non-mutually exclusive approaches to handling non-normal data:

Transformation Method. Map the data to a scale that is approximately normal and then calculate the capability. The Box-Cox transformation is a standard tool: when λ ≠ 0, y = (x^λ − 1) / λ; when λ = 0, y = ln x. λ is determined by maximum likelihood estimation, and common values have clear meanings: λ = 0 corresponds to log-normal (life, roughness, dimensional skew), λ = 0.5 corresponds to square root (counts, defect counts), and λ = −1 corresponds to reciprocal (cycle, rate). For more complex shapes, the Johnson transformation (SU, SB, SL families) can be used.

Distribution Fitting Method. Directly assume that the data follows a log-normal, three-parameter Weibull, Gamma, or exponential distribution, and use the fitted distribution to calculate the probability of exceeding the specification, expressed in PPM.

Non-Parametric (Percentile) Method. Make no assumptions about the distribution and directly use the empirical distribution to calculate the proportion of data exceeding the specification. This is the most general fallback method and the only one that does not rely on distribution assumptions.

Regardless of the approach taken, the final output should be converted to equivalent Cpk: Cpk_eq = z_p / 3, where z_p is the standard normal quantile corresponding to the actual one-sided exceedance rate p (for bilateral characteristics, take the poorer one-sided rate). For example, if the actual PPM = 300, i.e., p = 0.0003, z_p ≈ 3.43, and the equivalent Cpk ≈ 1.14. This conversion ensures that the index corresponds one-to-one with the actual defect rate and can be compared with the client's requirements.

2. Five Practical Steps: From Data Shape Determination to Reporting Standards

Step One: Determine Data Shape. It is recommended that a capability study have a sample size of at least 100 consecutive data points or 25 subgroups (4-5 items per group). Calculate skewness and kurtosis, perform the Anderson-Darling normality test, and plot the Q-Q graph. Criteria: if |skewness| ≤ 0.5, |kurtosis| ≤ 1.0, and the AD test p ≥ 0.05, handle it as a normal distribution and directly report the conventional Cpk; if |skewness| > 1.0 or the AD test p < 0.05 and the Q-Q graph shows a clear bend, switch to the non-normal path.

Step Two: Attempt Transformation. Use maximum likelihood to search for the optimal λ, and re-perform the normality test after transformation. Criteria: if the AD test p ≥ 0.05 after transformation and λ falls within the −1 to 1 range, which has engineering interpretability, use the transformation method; if λ falls outside the |λ| > 2 extreme values or the AD test p < 0.05 after transformation, abandon the transformation and proceed to Step Three. Ensure that the specification limits (USL, LSL) are also transformed using the same λ, otherwise the entire result is invalid.

Step Three: Switch to Distribution Fitting. Consider candidate distributions such as log-normal, three-parameter Weibull, Gamma, and exponential, and select the one with the smallest AD statistic. Criteria: if the AD test p ≥ 0.05 for the selected distribution and the fitting deviation at the 5% upper quantile does not exceed 10% (the tail is the critical area for capability analysis, and a well-fitted median is not useful).

Step Four: Non-Parametric Fallback and Confidence Intervals. Directly use the empirical distribution to count the number of items exceeding the upper and lower specification limits, convert to PPM, and use Bootstrap (resample 1000 times) to provide a 95% confidence interval. Criteria: if the point estimate PPM does not exceed the client's target value and the upper confidence limit does not exceed twice the target value, the capability is deemed acceptable; if the upper confidence limit exceeds twice the target value, the capability is insufficient, and improvement is needed rather than just supplementing the report. The sample size must be verified: to discuss the tail at the 1000 PPM level, n must be at least 1000 for a stable estimate (expecting only one point in the tail); if n is between 100 and 125, only report the actual exceedance count and the PPM point estimate, and do not report distant quantiles.

Step Five: Standardize Reporting. The report should include three components on one page: the conclusion of the normality test (AD statistic, p-value, Q-Q graph), the method used and the rationale, and the actual PPM and equivalent Cpk. Criteria: if the results of the normal algorithm and the selected algorithm differ by more than 0.30 Cpk units, both numbers should be listed, and it should be clearly stated that "the actual PPM is the basis for decision-making."

3. Common Pitfalls

Pitfall One: Removing "Outliers" to Make Data Normal. Isolated large values in right-skewed data are precisely evidence of the true process capability. The sequence of handling should be to trace the cause first: is it due to burrs not being removed, insufficient gauge resolution (e.g., resolution greater than 1/10 of the tolerance), or the mixing of materials from different batches. Measurement/recording errors with clear causes can be removed and noted in the report; all extreme values without a clear cause must be retained.

Pitfall Two: Transforming Data but Forgetting to Transform Specification Limits. This is the most hidden and common error. If the transformation only affects the data column and the specification limits are copied from the original values, the calculated capability index will be either unrealistically good or unrealistically poor, and this will not be apparent from the report.

Pitfall Three: Declaring "Data is Non-Normal, Cannot Perform Capability Analysis" Upon Seeing p < 0.05. The AD test is extremely sensitive to sample size: when n exceeds 1000, even a slight skewness of 0.2 will be judged as "significantly non-normal." In such cases, the degree of deviation should be examined: if |skewness| ≤ 0.5 and the Q-Q graph basically follows the straight line, the data can still be treated as normal, and the test results should be noted in the report. Non-normality is never a reason to avoid capability analysis; it just requires a different algorithm.

Pitfall Four: Using Small Samples for Non-Parametric Tail Inference. Reporting a 0.1% quantile or 1000 PPM level capability with n = 30 is essentially a numbers game—the tail that does not exist in the sample cannot be estimated. The sample size must match the precision of the conclusion.

Pitfall Five: Combining Batch Data for Shape Determination. Mixing data from multiple molds, shifts, or batches can disguise a mixed distribution as non-normal or even bimodal. The transformation parameters fitted in this case are incorrect "mixed artifacts." The correct approach is to stratify the data first, check if each layer is approximately normal, and then decide whether to perform capability analysis on each layer or combine them.

Pitfall Six: Copying Bilateral Formulas for Unilateral Characteristics. Characteristics such as roughness, cleanliness, and impurity content only have an upper limit. The correct approach is to calculate the equivalent Cpk based on the one-sided exceedance rate, rather than artificially creating a lower specification limit to fit the bilateral formula.

4. Self-Check List

• Before performing capability analysis, have skewness, kurtosis, and the AD normality test been calculated, and has the Q-Q graph been retained?
• If a transformation is used, have the specification limits been transformed synchronously, and have the λ value and selection criteria been included in the report?
• Is the capability conclusion based on the actual defect rate (PPM) as the final criterion, rather than just presenting a capability index?
• When reporting distant PPM, does the sample size match the precision of the conclusion (n ≥ 1000)?
• Has it been confirmed that the data is not mixed (from multiple molds, shifts, or batches), and has stratification been performed if necessary?

Non-normality does not mean not calculating capability, but rather using a different scale.

Knowledge code: 6.3.2

Version: v20260921

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping enterprises continuously improve their quality capabilities.