Advancing QE Skills (24) | Design Verification (DV) and Production Validation (PV): How to Design a Valid Test Plan

A car sensor produced by an electronics manufacturing company was questioned by a customer quality engineer during a new product introduction review meeting. The first question: "The design verification report states '3 samples were tested for all 12 items, all passed, verification successful.' What is the confidence level of this conclusion?" The second question: "Do the sample parameter combinations cover the worst-case scenarios?" The third question: "Was the production validation batch produced under mass production tooling and mass production rhythm?" The meeting fell silent. Upon reviewing the report, it was found that the production validation batch consisted of 30 manually assembled units in the trial mold workshop, which were inspected and deemed qualified. However, there was no process capability index, and the sampling was not from a continuous batch. The tests were indeed conducted, and the qualified data was indeed obtained, but this data neither proved the design margin nor the production line capability—tests were done, but they did not count.

1. Key Principles: DV and PV Prove Two Different Things

DV focuses on whether the design output meets the design input, with the inference object being "the design itself." Therefore, its evidence logic is based on margin and reliability: either proving that the design has a margin under extreme conditions or using "zero failure + sample size" to establish reliability and confidence. PV focuses on whether the mass production process can stably produce conforming products, with the inference object being "the process distribution." Therefore, its evidence logic is based on process capability: whether the distribution width is sufficient relative to the specification width, whether the center is offset, and whether it remains stable over time. The two cannot be interchanged: DV data cannot answer "Is the production line stable?" and PV data cannot answer "Is the design margin sufficient?" The first step in designing a test plan is to write down what you need to prove in measurable criteria, rather than deciding how many samples to test first.

2. Five Practical Steps and Quantitative Criteria

Step One: Establish a Verification Requirements Matrix. List all design inputs (function, performance, environment, lifespan, regulations) and match each to a test item, method standard, and acceptance criterion. Criteria: Coverage must be 100%, and any non-quantifiable requirements must be broken down into measurable alternative indicators (e.g., "vibration resistance" broken down into frequency, amplitude, duration, and judgment threshold). If this is not possible, a written explanation of the reason must be provided and approved by the customer or design leader. Any items simply stating "meets customer requirements" should be returned.

Step Two: Determine DV Sample Size Using Statistical Standards. Follow the sample size specified by the customer or industry standards if they are provided; if not, use the reliability-confidence formula for "zero failure" tests:

n = ln(1 − C) / ln(R)

where C is the confidence level and R is the reliability. Common combinations: when C=90% and R=90%, n=22; when C=95% and R=90%, n=29; when C=90% and R=95%, n=45; when C=95% and R=95%, n=59. If failures are allowed, the sample size must be increased (for example, when C=90%, R=90%, and 1 failure is allowed, approximately 38 samples are needed). Criteria: For general characteristics, the minimum is C=90%, R=90%; for safety and regulatory characteristics, at least C=95%, R=95% must be achieved; it is not acceptable to issue verification conclusions based on 3-5 "empirical samples." If the cost prevents meeting the sample size, the actual confidence level and residual risk must be clearly stated in the report and confirmed by the customer.

Step Three: Design Test Conditions to Cover "Worst Case." The value of DV lies in the boundaries, not the nominal points. Parameters should be set to the upper and lower limits of specifications, the most unfavorable tolerance combinations, extreme temperature/humidity/voltage, the worst material batches, and the worst fit gaps. Criteria: Each key parameter must cover the upper and lower boundaries at least once, and the most unfavorable combinations between key parameters must be tested, with each value and its basis clearly stated in the report; the samples must use the materials and processes intended by the design. If rapid prototyping or non-mass production tooling is used, a written assessment of the impact of these differences on the conclusion must be provided, and additional verification must be arranged.

Step Four: PV Must Prove "Production Line Readiness," Judged by Capability and Continuity. PV must use mass production tooling, mass production rhythm, follow mass production work standards, and be executed by mass production personnel, with a measurement system confirmation completed beforehand. Sampling requirements: At least 25 subgroups must be taken from continuous production batches, with 4-5 pieces per subgroup (a total of 100-125 pieces), covering shifts and equipment changes; if the customer's CSR has stricter requirements, follow the CSR. Criteria: Key characteristics Ppk≥1.67, important characteristics Ppk≥1.33, general characteristics Ppk≥1.00; the measurement system %GRR must be <10% to be acceptable, 10%-30% must be explained in writing as having no substantial impact on the judgment conclusion, and >30% must be improved; for attribute inspections, a Kappa≥0.75 is the passing line. If the production rhythm is involved, a Run@Rate verification is also required, and both capacity and FTQ must meet the criteria to pass.

Step Five: Issue Conclusions in Three States and Lock in Revalidation Triggers. Test conclusions should only be one of three states: "unconditional pass," "conditional pass," or "fail." Terms like "basically qualified" or "generally good" are not allowed. Criteria: The report must include original data, method standards, sample size, batch number, tooling number, equipment calibration status, acceptance criteria, deviation records (deviations from specifications, batch deviations, condition substitutions) and their impact assessments, and approval signatures; any invalid data must not overwrite the original records but should be accompanied by a deviation explanation. Revalidation is required in the following situations: design or material changes, supplier or production location changes, significant process adjustments, continuous complaints or batch failures, and expiration of the validation conclusion's validity period.

3. Common Pitfalls

Pitfall One: Using DV Data to Answer PV Questions. The most typical example is using data from manually assembled samples or rapid prototypes to calculate process capability or fill out the initial process study in PPAP. Different sample sources result in different distributions, making the conclusions inherently invalid. A straightforward way to judge this is to look at the "batch and tooling" section of the report; if the PV row states the trial mold workshop and manual assembly, it is likely an invalid PV.

Pitfall Two: Sample Size Determined by Guesswork, Without Reporting Confidence Levels. "Test 3 pieces, all passed, so the design is fine"—the lower confidence limit for zero failures in 3 pieces is extremely low and cannot withstand any audit scrutiny. Sample size should be derived from the test plan and criteria, not added as an afterthought.

Pitfall Three: Testing Only Nominal Conditions, Not Boundaries. A large number of "passed verification, failed in the market" cases originate from this: tests are completed at 25°C, rated voltage, and nominal dimensions, but failures occur under extreme temperatures, dimensional limits, and the worst material batches. The selection of verification conditions itself is a risk assessment action and should be documented in the DFMEA detection measures.

Pitfall Four: Confusing Design Verification with Design Validation. Verification (Verification) asks whether "the produced item is correct (meets design input)," while validation (Validation) asks whether "the correct item is usable (meets actual user requirements)." Focusing only on the former can result in all indicators being compliant but not suitable for user scenarios; industries such as medical and automotive have clear distinctions between the two and they cannot be substituted for each other.

Pitfall Five: Verifying Only Once, Without Setting Revalidation Triggers. Treating verification as a one-time checkpoint and not reassessing after changes. The correct approach is to write revalidation triggers into the procedure document and include them as mandatory items in the change approval form.

Pitfall Six: Reports Contain Only Conclusions, Not Criteria. Reports stating "qualified" throughout without providing acceptance criteria or the source of the criteria have almost no evidentiary value during customer audits or recall investigations.

4. Self-Check List

• □ Is each test item traceable to a measurable design input, and is the demand coverage 100%?
• □ Is the sample size statistically justified (confidence level and reliability values clearly defined), and do safety/regulatory characteristics meet C=95%, R=95% or customer specifications?
• □ Do DV conditions cover the upper and lower boundaries and the most unfavorable combinations of parameters, and have material and process differences been assessed in writing?
• □ Does PV use mass production tooling, rhythm, personnel, and work standards, and is the sampling ≥25 subgroups, with Ppk and %GRR meeting the criteria?
• □ Are the conclusions in the report presented in three states, and do they include original data, batch numbers, calibration status, deviation records, and revalidation triggers?

Sample size must be justified, and verification conditions must cover the boundaries for the conclusions to count.

Knowledge code: 8.2.3

Version: v20261004

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.