What Would You Validate First?

IN THIS LESSON

Once you have made a recommendation, the next useful question may be what you would validate before investing further.

Start with the uncertainty most capable of changing the decision. Then choose the fastest credible way to reduce that uncertainty enough to decide what to do next.

The validation method follows from the uncertainty. It may be a prototype, experiment, user study, technical spike, data analysis, operational test, market test, or another form of evidence. An A/B test is one option, not the definition of validation.

What this task proves about you

Product decisions are made with incomplete information. A good PM does not respond by trying to eliminate every uncertainty before acting. They identify which unknowns are consequential and spend learning effort where it can actually change the decision.

This follow-up therefore tests judgment under uncertainty. Can you distinguish a risk that threatens the recommendation from an ordinary implementation question? Can you choose evidence that speaks directly to that risk? Can you explain what you would do differently depending on what you learn?

Start with the biggest decision-changing uncertainty

Return to the Feasibility discussion from solution prioritization and ask which handicap or assumption could make the chosen solution lose.

For the adaptive study planner, you may believe the product has strong Impact because it converts a deadline and remaining workload into a daily recommendation. But that depends on the recommendation being trustworthy enough for learners to follow. If task-time estimates are consistently wrong, the product could produce false confidence instead of useful guidance.

That is a stronger validation target than a generic question such as whether learners like the interface. The estimate quality sits inside the mechanism that justified the recommendation.

A useful test is: if this assumption turns out to be wrong, would I materially change the product, the scope, the launch plan, or the recommendation itself? If not, it is probably not the first thing to validate.

Match the method to the question

Different uncertainties require different evidence.

If you are uncertain whether customers understand or want the experience, a prototype and customer observation may be more useful than an experiment. If the risk is technical feasibility, a technical spike or offline evaluation may answer the question faster. If you need to know whether a recommendation changes behavior, a controlled experiment may be appropriate. If the product depends on a new operational process, manually delivering the service to a small number of customers may expose the real constraints before software is built.

Do not choose an A/B test merely because experiments sound rigorous. An experiment can only answer a question that the treatment, population, and observable outcome actually expose.

Test the mechanism before polishing the product

Early validation is most valuable when it examines the causal claim at the center of the product.

Suppose the parking recommendation depends on predicting the true overhead between reaching the destination area and actually arriving at the venue. Before building a rich parking interface, you could examine historical movement data for a small number of large venues and determine whether that overhead is predictable enough to produce useful guidance.

If the answer is no, adding interface polish will not rescue the mechanism. If the answer is yes, the product team has stronger reason to invest in the experience around it.

The same principle applies to marketplaces, assistants, recommendation systems, and workflow products. Validate the thing that has to be true for the product to create the value you claimed.

Define what the evidence would change

A validation plan is incomplete if every possible result leads to the same next step.

Explain what you would do if the evidence is strong, weak, or ambiguous. For the study planner, highly accurate estimates combined with customers following the recommendations might justify building a more automated version. Poor estimate quality might push you toward a product that helps learners reason about workload without pretending to prescribe an exact daily commitment. Strong estimate quality but low customer trust might make the experience or explanation the next problem to solve.

You do not need to specify precise statistical thresholds unless the case gives you a reason to. What matters is showing that the learning is connected to a decision.

Keep validation proportional

You are looking for enough evidence to make the next decision better, not permanent proof that the product will succeed.

Early evidence can be small and imperfect when it is cheap, relevant, and capable of revealing a major flaw. A manually operated pilot may be more informative than months of building. A prototype with a handful of the right customers may expose whether the interaction makes sense. Existing data may answer a question without any new experiment at all.

As confidence grows and the cost of the next decision increases, the evidence bar can rise with it.

The point

Validate the uncertainty most capable of changing what you would do. Choose the evidence source that fits that question, test the core mechanism before polishing around it, and make clear how the result would affect the next decision. Validation is useful because it updates judgment, not because every product needs an experiment.