Advancing QE Skills (35) | Verification and Misjudgment Management of Online Automatic Inspection

A visual inspection device was installed on the assembly line of an automotive parts company to detect incorrect assembly, missing parts, and surface scratches, with the judgment results directly driving release. On the day of acceptance, the supplier tested 500 sample pieces, 40 of which were deliberately made nonconforming. The device alarmed for all 40, and the report stated a "detection rate of 100%." The acceptance form was signed on the spot, and the production line subsequently removed two final inspectors. Three months later, the customer complained about two consecutive batches: one with missing clips and another with scratched shells. Upon reviewing the device logs, both batches were judged as OK. Meanwhile, the complaints from the other end of the production line were completely opposite—this device was "too aggressive," with over 200 pieces being judged as NG daily, and 80% of them being good after re-inspection, leading to constant overtime at the re-inspection station. The QE was called to "fine-tune the device," but no one could answer a basic question: what is the actual detection capability of this device for each type of defect? The 40 sample pieces were long gone, and the 100% detection rate from the acceptance test could no longer be replicated.

1. Principle: Both Types of Errors Cannot Be Eliminated Simultaneously

Online automatic inspection is essentially an attribute judgment device: it converts measurement values or image features into a score, compares it with a threshold, and outputs whether the product is conforming or nonconforming. This process inherently involves two types of errors—missed detection (nonconforming judged as conforming, defects flow to the customer) and overkill (conforming judged as nonconforming, good pieces are stopped). Tightening the threshold reduces missed detection but increases overkill; loosening the threshold has the opposite effect. These two errors are inversely proportional, and there is no single "best" setting; only trade-offs based on cost and risk.

Trade-offs can be calculated. Let the missed detection rate be p₁, the escape cost per piece be C₁, the overkill rate be p₂, and the overkill cost per piece be C₂. The total judgment cost per piece is:

Total Cost = p₁ × C₁ + p₂ × C₂

A company's shell line measured the following: under the current threshold, p₁ = 0.5%, p₂ = 3%; the comprehensive loss per nonconforming piece found at the customer's assembly end is C₁ ≈ 800 yuan (line stoppage, sorting, and apportioned claims), and the cost per overkill piece within the factory is C₂ ≈ 6 yuan (re-inspection labor, handling, and minor mis-scraping). Total Cost = 0.5% × 800 + 3% × 6 = 4.00 + 0.18 = 4.18 yuan per piece. After tightening the threshold by one level, the overkill rate increased to 8%, and the missed detection rate decreased to 0.2%. Total Cost = 1.60 + 0.48 = 2.08 yuan per piece, saving 2.1 yuan per piece. The conclusion is counterintuitive: at this cost structure, an overkill rate of 8% is still cost-effective. Conversely, if the escape cost is only 20 yuan (nonconforming can be detected in the next in-house process), the same data would lead to the opposite conclusion—tightening the threshold would be a loss.

From this, two engineering judgments can be derived:

First, detection capability is not an inherent attribute of a device but varies by defect type. Dimensions, contrast, position, posture, and batch color differences can all change the detection rate. Announcing a "detection rate of 99.8%" without specifying the type of defect, its size, and the sample size provides no meaningful information. Capability must be reported in layers—critical/severe/minor—because the acceptable levels for each layer differ by an order of magnitude.

Second, "zero missed detection" in small samples does not prove capability. If n real nonconforming pieces are all detected, the upper limit of the missed detection rate at 95% confidence is approximately 3/n:

To declare a "missed detection rate ≤1%," at least 300 nonconforming samples of that type must be detected. It is unrealistic for most production lines to gather this sample size for critical defects—this is the basis for the decision: critical defects cannot rely solely on inspection; physical poka-yoke (limiting, non-standard tooling, sensor interlocks) must be used to prevent errors at the source, with inspection as a secondary measure. Severe defects should be verified through inspection and regular challenge samples. Minor defects can rely on inspection and accept a higher missed detection rate.

2. Five Practical Steps: From Equipment Acceptance to Daily Monitoring

Step One: Build a Defect Sample Library, Do Not Use Temporary Samples.

Stratify by "defect type × severity × size gradient," with no fewer than 30 samples for each type of defect. Include 10 borderline samples (sizes at the edge of the judgment threshold, with the poorest contrast) and mix in 150 conforming samples for comparison. The true value of samples cannot be determined by the device or a single person: it should be independently judged by three experienced inspectors, with a Kappa consistency ≥0.75. Controversial cases should be resolved using microscopic measurement or destructive analysis. Re-evaluate the samples every six months, and immediately replace any that are rusted, worn, or deformed—high scores from old samples are misleading.

Step Two: Conduct Acceptance Tests Blindly, with Predefined Criteria.

Randomly mix the defect samples with conforming samples (a ratio of about 1:5 nonconforming to conforming), without informing the operators or equipment engineers. Run the samples through the station at normal production speed, repeating the process at least three times. Suggested acceptance criteria:

The criteria must be included in the equipment acceptance agreement and control plan, not just stored on the QE's computer. Once set, there is no room for "let it pass this time."

Step Three: Three-Level Monitoring After Deployment, Unify the Metrics.

• Daily: Run 5 key defect challenge samples at the start of each shift. Any missed detection should trigger a shutdown for investigation. Simultaneously, calculate the proportion of re-inspected NG pieces that are judged as conforming, to serve as the overkill rate for the day.
• Weekly: Plot the missed detection rate and overkill rate for each defect type on a p control chart. Ensure the sample size is sufficient—do not plot trend lines if the weekly judgment volume is less than 2000 pieces or the re-inspection volume is less than 50 times, as small sample fluctuations can mislead.
• Monthly: Compare each client escape piece with the device logs. Any piece that the device judged as OK but the client judged as NG should be added to the defect sample library. This is the only way to calibrate the "perceived capability" of the device with the "actual capability."

Step Four: Set Clear Re-Inspection Triggers, Re-Run the Entire Library Within 24 Hours.

Triggers include: changing models or tooling, changing lenses/light sources/cameras, algorithm or software version upgrades, operator changes, environmental limits (temperature, illumination), challenge sample failures, and client escape complaints. Algorithm upgrades are the most dangerous—suppliers often push a "optimized version" remotely, which can shift the judgment behavior as a whole. This must be treated as a new device for re-acceptance, and change management procedures must be followed. Without clear triggers, the device's capability will silently degrade.

Step Five: Misjudgment Handling Process, Prevent "Overkill" from Becoming Habitual Release.

Three-color handling: for pieces judged as NG due to critical or severe defects, isolate and re-inspect all, trace back to the previous piece to confirm the conforming point, and resume production only after confirming no missed detections. For pieces judged as NG due to minor defects, use limit samples for comparison, with independent judgments by two people. In case of disagreement, escalate to a third person. Overkill pieces must not be released by operators on their own; they must be stored in a separate area, re-inspected with signatures, and counted in the overkill rate.

If the overkill rate exceeds the target for three consecutive shifts, threshold adjustments must be approved by the QE, process engineers, and customer representatives (for special characteristics). Parameter changes must be documented and traceable, and the control plan must be updated accordingly. It is even more valuable to establish an overkill cost ledger: convert daily re-inspection labor, handling, and mis-scraping into monetary terms and summarize monthly. Without this ledger, the QE's statement that "overkill is too high" in meetings will always be just a feeling; with it, there is leverage to adjust the threshold.

3. Common Pitfalls

Pitfall One: Accepting the Device Based on Total Detection Rate. If 480 out of 500 sample pieces are easy-to-detect defects, the total detection rate will naturally look good. High detection of easy-to-detect defects can dilute the missed detection of critical defects. The only correct approach is to report in layers, with critical defects judged and signed off separately.

Pitfall Two: Assuming Reliable Judgment Because of Good Repeatability. Repeatability calculated by testing a standard piece 30 times only proves that the device is stable with its own standard piece, not that it is sensitive to real defects. Stability and resolution are two different things, and the former cannot replace the latter.

Pitfall Three: Declaring Capability Based on 30 Pieces with Zero Missed Detection. If all 30 pieces are detected, the upper limit of the missed detection rate at 95% confidence is 10%, not 1%. When the sample size is insufficient, the correct statement is "no missed detections found," not "missed detection rate below a certain value."

Pitfall Four: Viewing Overkill as "Strictness, Which Is Good." The most serious consequence of overkill is not the cost but the loss of trust: if operators find that eight out of ten pieces are false alarms, they may start to ignore alarms, cover the lens with tape, or lower the sensitivity to solve the problem themselves. At that point, real missed detections will multiply, while the reports will show everything as normal.

Pitfall Five: Deploying Algorithm Upgrades Without Verification. "Just optimized the parameters" is the most common statement from suppliers and the starting point for most escape incidents. Any update that changes the judgment logic must re-run the defect sample library.

4. Self-Check List

• Each type of defect at this station has corresponding defect samples, with no fewer than 30 samples for each type (critical defects have additional downstream poka-yoke verification records)
• The acceptance report stratifies the missed detection rate, overkill rate, and judgment consistency Kappa by critical/severe/minor, and does not use a single "total detection rate"
• Challenge samples are run at the start of each shift, and the shutdown and release rules after failure are documented in the work instructions
• Re-inspection triggers (model change, optical component change, algorithm upgrade, client escape complaint) are listed, and the most recent trigger has a re-inspection record
• Overkill pieces have a separate isolation area, re-inspection signatures, and a monthly cost ledger, with threshold adjustments documented and traceable

Detection capability must be reported in layers, and the two types of errors should be balanced by calculating costs to set the threshold.

Knowledge code: 12.2.3

Version: v20261023

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.