GR&R Passes, but the Gauge is Not Sufficient? —— Five-Step Method for Identifying and Addressing Insufficient Gauge Resolution
1. A Vernier Caliper That "Can't See the Difference"
A parts manufacturer was inspecting a shaft with a tolerance of 0.02mm using a digital vernier caliper with a resolution of 0.02mm. MSA was conducted annually, and the %GRR was only 12%, firmly within the "acceptable" range. However, problems persisted on the shop floor: for the same batch of parts, Employee A read 9.98mm, while Employee B read 10.00mm, leading to some parts being approved and others being reworked. Even more puzzling, the process capability calculated a Cpk of 1.6, but the customer's re-inspection resulted in batch after batch of returns.
The problem wasn't with the people, but with the caliper. This caliper could only read to a minimum of 0.02mm, and the entire tolerance range for the measured characteristic was also 0.02mm—leaving only two or three graduations for the entire tolerance band. The true differences between parts were being flattened by the measurement system. With readings quantized into a few graduations, the repeatability was often close to zero, making the %GRR "look good" and the report pass, but the caliper couldn't distinguish between good and bad parts.
This is the easiest aspect of MSA to overlook: resolution, or the number of distinguishable categories. GR&R answers whether the variation is small enough, while resolution answers whether the differences can be seen. If differences can't be seen, even the smallest variation is meaningless.
2. Two Criteria, Enough to Recognize
Criterion One: The Tenth-Part Rule. The resolution of the gauge (minimum readable graduation) should not exceed 1/10 of the tolerance of the measured characteristic. For a tolerance of 0.02mm, a gauge with a resolution of 0.002mm is required; using a 0.02mm caliper disqualifies the gauge from being used. If the process variation is used instead of the tolerance, the stricter side should be taken.
Criterion Two: Number of Distinguishable Categories (NDC). NDC = 1.41 × (part variation / measurement system variation). AIAG MSA requires NDC ≥ 5, and in practice, it is typically hoped to achieve 10 or more. It measures how many categories the measurement system can divide the parts into. When NDC is only 2 or 3, a pile of good and bad parts will cluster together in the data.
These two criteria are two sides of the same coin: insufficient resolution will inevitably lead to a low NDC. Focusing only on the %GRR value and ignoring the NDC is like measuring body temperature without diagnosing the cause.
3. Three Typical Consequences of Insufficient Resolution
Conflicting Judgments. Coarse graduations make it difficult to judge borderline parts, leading to inconsistent conclusions by different people or shifts, causing endless disputes.
Data Distortion Propagates Downstream. Insufficient resolution can cause the process capability index to be artificially high or low, leading to "staircase" patterns on control charts and misdirected SPC alarms and improvement efforts.
Improvements Diluted. Improvement actions that reduce actual variation are not detected by the gauge, resulting in zero effect in the data and wasted effort by the team.
4. Five-Step Implementation Method
Step One: Back-Calculate Required Resolution from Tolerance. List the tolerances of key characteristics and divide each by 10 to determine the minimum readable graduation for the gauge. This step should be done during the control plan compilation phase, not waiting until mass production.
Step Two: Verify the Readings and Graduations of Existing Gauges. Check the gauge register and actual measurement records: what is the minimum graduation? Are the digital readings only in whole numbers? When the same position is measured ten times, are there only two or three different values? Highly concentrated readings are a warning sign.
Step Three: Calculate NDC to Assess True Classification Ability. Review NDC along with %GRR: if %GRR is acceptable but NDC is less than 5, the gauge is deemed to have insufficient resolution and the measurement system is still unacceptable. This step is crucial for correcting the "only look at %GRR" mindset.
Step Four: Choose One of Three Actions. Replace the gauge (upgrade to a higher resolution), change the method (use magnification or specialized inspection tools, such as a dial caliper or pneumatic gauge), or add controls (establish a rule for rejudging borderline parts with a more precise gauge). Prioritize actions based on cost and impact, focusing on key characteristics and high-risk processes.
Step Five: Document the Conclusions. Update the control plan and gauge list with the replaced gauge; add new judgment rules to the work instruction; and set "resolution verification" as a mandatory step for new gauge procurement and first article inspection.
Returning to the initial case: the company replaced the caliper for the shaft with a digital micrometer with a resolution of 0.002mm. After redoing the MSA, the %GRR increased to 18%, and the NDC rose from 3 to 9. The differences in judgment between shifts disappeared, and the customer's re-inspection returns ceased—while the %GRR became "worse," the measurement system actually became more effective.
5. Three Common Misconceptions
Misconception One: GR&R Pass Means the Measurement System is Qualified. A %GRR within the acceptable range is a necessary condition, not a sufficient one. When NDC is不合格 (unacceptable), the "qualified" status is often a false impression created by the measurement system's inability to distinguish differences.
Misconception Two: Calibration Qualification Ensures Resolution. Calibration only ensures that the gauge is accurate and traceable; it does not guarantee that the gauge can "see" the required changes in the measured characteristic.
Misconception Three: Higher Resolution is Always Better. Excessively fine resolution can amplify noise, increase reading instability, and extend measurement time. Setting the resolution to 1/10 of the tolerance is sufficient; there is no need to pursue unnecessary precision.
6. One-Sentence Summary
First, set the resolution to 1/10 of the tolerance, then verify the measurement system's classification ability with NDC ≥ 5, and choose one of the three actions: replace the gauge, change the method, or add controls. Finally, document the changes in the control plan and work instruction—the first threshold for a measurement system is not how small the variation is, but whether the differences can be seen.
Resolution is the entry ticket for a measurement system: first, see the differences, then discuss whether the variation is small enough.
Knowledge code: 6.2.1
Version: v20260911
Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.