When we last left Stat-Woman, she was wrapping up a rigorous session in the Training Grounds, where visualization techniques were sharpened, non-normal distributions were confronted honestly, and measurement systems were put under the microscope. Now, the safety net is gone. Mr. Ledger reminded her that the teams who take the Training Grounds seriously are the ones who rarely need to explain themselves later. Tonight, she will find out whether that guidance holds true with a challenge unlike anything faced before. Design Verification (a.k.a. DV) is where exploratory thinking gives way to confirmatory evidence, where every statistical decision must be deliberate, and where the consequences of a wrong answer are no longer theoretical. There is no Holo-Deck reset here like in the Training Grounds. The data is real, the acceptance criteria are locked, and The Rogue Distribution is paying close attention.

Crossing the Line: From Exploration to Confirmation

The transition from exploratory statistics to Design Verification is not just a change in methods; it is a change in mindset. During the Training Grounds, the goal was to learn techniques, explore variability, and test assumptions in a low-stakes environment where failure was information. In the field, the goal is to demonstrate. Every test is designed to answer a specific question tied to a precise requirement and the answer must hold up under regulator scrutiny.

This shift catches more teams off guard than it should. The exploratory phase rewards curiosity and flexibility where Design Verification rewards precision and pre-specification. A team that carries an exploratory mindset into Design Verification will find themselves making decisions about statistical methods after the data is already collected, adjusting acceptance criteria based on what the results look like, and reframing the question to fit the answer they already have. These are rationalizations, not analytical techniques, and they are exactly what The Rogue Distribution exploits. This is where the earlier phases earn their keep. The requirements that were mapped to statistical approaches in the Stat-Cave and stress-tested in the Training Grounds now execute exactly as written. Test protocols are locked, sample sizes are justified, and acceptance criteria are traceable to risk assessments and intended use. If the Stat-Cave was where the strategy was built and the Training Grounds were where it was rehearsed, Design Verification is where it gets proven.

Choosing Your Weapons: Acceptance Criteria That Actually Mean Something

One of the most consequential decisions in Design Verification is the selection of acceptance criteria, and that decision should be documented in the DV test matrix long before the first test unit is built. The test matrix maps every design requirement to a statistical approach, and at the heart of that mapping is a choice between four primary tools: confidence intervals, tolerance intervals, capability metrics, and reliability testing. Each answers a fundamentally different question about the same data, and selecting the wrong one can mean the difference between a meaningful conclusion and a misleading one. A confidence interval on a mean, for example, may be appropriate for characterizing a performance parameter against a specification, but it only tells you about the central tendency and says nothing about the tails of the distribution, which is exactly where failures hide.

Tolerance intervals answer a different question entirely. They estimate the range within which a specified proportion of the population falls at a given confidence level. For requirements where every unit matters and not just the average, tolerance intervals are often the right tool. Capability metrics like Cpk and Ppk serve a complementary purpose: they tie performance directly to specification limits and quantify how well centered and controlled a process is relative to those limits. A Cpk of 1.33 means the process mean sits at least four standard deviations from the nearest specification boundary, which provides a meaningful margin of safety. Where these metrics fall short is that they assume a stable, normally distributed process, so applying them to early DV data without verifying those assumptions can produce misleadingly optimistic results. Reliability testing takes a time-based view altogether, asking not just whether the product works now but whether it will continue working after hundreds or thousands of cycles under defined conditions. Weibull analysis, life data modeling, and reliability demonstration tests each address different aspects of this question, and the choice among them depends on the failure mechanisms most relevant to the product’s intended use and risk profile.

The point is that none of these four tools are interchangeable. A dimensional tolerance on a critical interface may call for a tolerance interval. A cyclic fatigue requirement may demand reliability testing with a Weibull analysis. A drug delivery rate specification on a combination product may require a confidence interval on the mean alongside a tolerance interval on individual doses. In each case, the choice is driven by what the requirement demands, what risk the failure represents, and what the clinical consequence would be if the product does not perform as intended. Every cell in the DV test matrix should be traceable to that chain of logic: a requirement, a risk, and an intended use. Anything less is a gap that a regulatory reviewer will find.

Sample Size: A Deliberate Decision

If acceptance criteria define what “success” looks like, sample size determines whether you have enough evidence to claim it. Too often, sample sizes in DV are inherited from previous projects, borrowed from industry convention, or dictated by how many units happen to be available. None of these are statistical justifications. They are conveniences and conveniences have a way of becoming vulnerabilities when someone asks “why?”

A properly justified sample size starts with the question being asked, and that question is inseparable from the risk it addresses. What level of confidence is required? What proportion of the population must be covered? What is the expected variability based on exploratory data from the Training Grounds? The answers to these questions should scale with the clinical relevance of the requirement. A non-critical dimensional characteristic may justify a smaller sample with moderate confidence levels, while a drug delivery accuracy requirement on a combination product, where under-dosing or over-dosing directly affects patient safety, demands substantially higher coverage and confidence. For attribute testing, sample sizes are often driven by reliability demonstration tables or Lot Tolerance Percent Defective-based sampling plans. For variable data, the calculation depends on the acceptance criterion selected, whether it is a confidence interval, tolerance interval, or capability metric, and the assumptions about the underlying distribution. This is where the honest assumptions about variability documented in the Stat-Cave and tested in the Training Grounds finally earn their keep. A sample size based on an assumed standard deviation that was never validated is not a plan, it is a guess. Powerless, that exhausted member of The Rogue Distribution, is always waiting for teams who underestimate how much data they actually need. Ultimately, sample size is not just a statistical calculation. It is a statement about how much evidence you believe is necessary to support a claim that affects patient safety, and that statement must be defensible.

The Outlier Problem: When to Exclude, When to Investigate, and When to Listen

Months into a Design Verification testing campaign, Stat-Woman encounters one of her most complex adversaries: The Outlier. He does not arrive with the swagger of Captain Assumption or the quiet manipulation of The Confounder. He simply appears in the data, standing apart from everything else, daring you to decide what he means.

The instinctive reaction is to remove him. An outlier throws off your mean, inflates your standard deviation, and can single-handedly turn a passing result into a failure. Exclusion feels like the clean, efficient solution, but The Outlier is rarely just noise. He may represent a hidden failure mode that the test was not designed to detect. He may be exposing a limitation in the test setup itself, such as a fixture that slipped, an environmental condition that drifted, or a measurement system operating at the edge of its range. Or, he may be the first signal of a systemic issue that has not yet manifested across the broader population. Each of these scenarios demands a different response and lumping them all under “exclude as outlier” is the statistical equivalent of ignoring the smoke detector because you do not see flames.

When it comes to deciding what to do with an outlier, statistical tests and regulatory expectations both play a role, but they answer different questions. Statistical tests like Grubbs’ test or Dixon’s Q test determine whether a data point is unusual relative to the rest of the sample. They provide a mathematical threshold, but passing that threshold does not automatically justify removal. The regulatory expectation goes further: it asks whether there is an assignable cause that explains why the point is different. A fixture that slipped, a documented environmental excursion, a measurement system operating outside its validated range are all assignable causes. Without one, exclusion is not analysis. It is selective reporting. And if you cannot explain why a data point was removed when a regulatory reviewer asks, the omission becomes a bigger problem than the outlier ever was. The statistical test opens the door. The assignable cause investigation determines whether you are justified in walking through it.

In the narrative, The Outlier is a single character (although there was that one time he cloned himself in a botched experiment), but in practice, outliers rarely arrive alone. A single unexpected data point may be the only visible manifestation, but the investigation it triggers often reveals that the underlying cause affects more than one result. Stat-Woman has learned to approach each outlier with curiosity rather than hostility. When one appears, the first step is always investigation, not exclusion. Was there a documented deviation during testing? Does the measurement system data from the Test Method Validation support that this result is within the system’s capability? Is there a physical mechanism that could explain the observation? Sometimes the investigation reveals a legitimate assignable cause and the data point is properly excluded with full documentation. Sometimes it reveals a previously unknown failure mode that needs to be addressed before testing continues. Alternatively, the outlier is simply telling you something about the natural variability of your product that you were not expecting. In those cases, the right response is not to discard the information but to listen, adjust your understanding, and let the data disclose the complete story.

Holding the Line: Final Thoughts from the Field

Design Verification is where planning meets reality. Every decision made in the Stat-Cave, every assumption tested in the Training Grounds, and every measurement system validated along the way converges here. The methods are confirmatory, the documentation is formal, and the results must stand on their own when someone asks “why?” months or years from now. There is no room for improvisation, but there is room for judgment, and that distinction matters. The best DV strategies are not the most rigid ones. They are the ones built on a foundation strong enough to support the weight of the decisions resting on them.

The next time we visit Variance City, the stakes rise. Stat-Woman will face the battle of validation and regulatory decision-making, where statistical rigor is not just expected but demanded and where Dr. Data himself will test every assumption, every method, and every conclusion Stat-Woman has built across the entire lifecycle. The acceptance criteria will be absolute, the scrutiny will be relentless, and the error margin zeros out.

AUTHOR

Alex Spivak, Principal Consultant, Suttons Creek – Alex is a highly strategic Biomedical Engineer with 13 years of MedTech industry experience, specializing in solving complex R&D challenges and ensuring strict Quality Assurance and Compliance. Alex transforms product development through rigorous statistical analysis and process optimization, consistently delivering substantial quality enhancements and guaranteeing minimal Post Market Surveillance complaints. As the lead Quality Engineer for numerous global commercial product launches, he has successfully operated in diverse environments, from large corporations to small startups, and across multiple domestic and international locations, including Japan.