What I Learned Building the Product Simply AI Coach

I have spent thousands of hours coaching product managers through interviews, and over time that work developed into a fairly specific methodology. I teach recurring principles, diagnose common failure modes, and have opinions about interviewing that sometimes differ meaningfully from conventional advice. As the capabilities of language models improved, it seemed natural to ask whether some of that expertise could become a useful product when I was not available to coach someone directly.

The first version of my AI coach was built around a simple premise. I already had a course, a substantial body of supporting material, and a clear sense of how I wanted candidates to approach interviews. If I gave a capable model access to that material and explained how I coached, I expected it to produce something recognizably different from a general-purpose chatbot.

The initial results were encouraging enough to conceal the real problem. The coach usually produced coherent, useful answers, and it often referenced the ideas I had given it. But it was also drawing freely on everything the underlying model already knew about product management, interviewing, leadership, communication, and technology companies. When my own material left something unspecified, the model filled in the gap. When my advice differed from common interview guidance, the model could drift back toward the conventional answer. It could also extrapolate from my ideas in ways that sounded perfectly reasonable but that I would never have taught a customer myself.

Those failures were difficult precisely because they were usually plausible. I was not dealing primarily with obvious hallucinations or nonsensical responses. I was dealing with answers that another experienced PM might read and consider perfectly sensible, while I would read the same answer and recognize that it contradicted the methodology the product was supposed to represent.

At that point, the value proposition became difficult to defend. If a customer using my AI coach was ultimately receiving whatever a strong general-purpose model happened to believe about PM interviewing, the customer could simply use ChatGPT instead.

The problem was control, not access

That experience changed the question I was trying to answer. I had started by asking how to give the model enough of my knowledge. I increasingly found myself asking which knowledge it should be allowed to use, what it should infer from that knowledge, and how it should behave when my methodology did not contain an answer.

That distinction matters for any AI product whose value depends on a particular body of expertise or institutional judgment. A capable foundation model arrives with enormous world knowledge, and much of that knowledge is useful. The same capability also creates a source of product risk because the model can generate reasonable conclusions that are inconsistent with the behavior the product is intended to provide.

For my coach, that meant I could not define quality simply as whether an answer was intelligent, fluent, or even useful in the abstract. A high-quality answer had to be faithful to the coaching methodology as well. When the underlying model's general knowledge conflicted with that methodology, the product needed a way to prefer the product's point of view.

This is one of the more important changes in how I now think about AI product development. The objective is not always to expose as much model intelligence as possible. In many products, the harder and more valuable work is deciding where that intelligence is allowed to come from and how much freedom the system should have to extend beyond the domain knowledge it has been given.

A course is not an AI knowledge architecture

Solving that problem also forced me to reconsider what it meant to give the model access to my expertise. I already had a course designed to teach humans, so I initially treated it as a ready-made knowledge base. In practice, content that works well for a human learner is not necessarily organized in a way that lets an LLM use it reliably.

Humans consume a course sequentially and accumulate context along the way. They can usually tell that an example illustrates a principle without assuming every incidental detail of the example is itself a rule. They can recognize rhetorical emphasis, reconcile ideas that appear to be in tension, and understand that guidance presented in one section may depend on assumptions established somewhere else.

A model retrieving isolated pieces of that material has a different problem. It needs to distinguish principles from examples, understand which guidance is general and which is conditional, know how to resolve apparent conflicts, and recognize when the source material does not justify an inference. Those relationships were often obvious to me because I had created the material, but they were not necessarily explicit in the material itself.

The knowledge base therefore had to be rearchitected rather than merely indexed. I began separating concepts that had previously been intertwined, making conditions and exceptions more explicit, establishing clearer relationships between principles and examples, and adding information whose primary purpose was to help the system understand how the methodology fit together.

I also needed controls above the knowledge itself. Some governed what information the coach could rely on. Others governed how it should reason from that information, how much extrapolation was acceptable, and how the resulting guidance should be expressed. Safety and behavioral constraints needed their own treatment rather than being left implicit in the course material.

That work changed what I meant when I talked about the coach having a distinct voice. The superficial version of voice is stylistic: vocabulary, sentence structure, tone, or verbal mannerisms. That was not especially valuable to me. A system could imitate the way I write while recommending something I fundamentally disagree with.

The more useful definition of voice includes judgment. I wanted the coach to reach conclusions in a way that was recognizably consistent with the methodology behind the product, while still allowing enough flexibility to respond naturally to different customers and situations. The result is not deterministic in the software-engineering sense, but the behavioral space is intentionally much narrower than the underlying model's. The goal is a probabilistic system whose variation occurs inside boundaries I have chosen.

Customer feedback changed the job of the product

The technical changes were only part of the work because customer feedback was also changing my assumptions about what the coach should do.

AI makes it unusually easy to confuse capability with product value. Once a model can discuss a topic competently, there is a temptation to expose that capability broadly and allow users to ask it almost anything. My earliest conception of the coach was influenced by that possibility.

Actual customers were more specific. They arrived with particular problems they were trying to solve, and some of the uses I had imagined mattered less than I expected. Feedback also exposed places where a narrower, more opinionated intervention was more useful than an open-ended conversation.

I began revising the use cases themselves, defining more clearly what job the coach should perform in each situation and what a useful outcome looked like. This improved the product in ways that additional prompting alone could not have done because some behavior problems were really product-definition problems. If I could not state what the system should accomplish in a particular situation, I had little basis for deciding whether its answer was good.

The target therefore became narrower over time. I was no longer trying to build an AI that could talk intelligently about PM interviewing, because general-purpose models were already capable of doing that. I was trying to build a product that performed a defined set of coaching jobs according to a specific methodology and did so consistently enough that customers could rely on the distinction.

Defining the behavior made measurement possible

Once I had a clearer definition of intended behavior, informal review became increasingly unsatisfactory. Reading a handful of answers and deciding that they seemed good could tell me whether an obvious change had worked, but it could not tell me whether improving one part of the coach had quietly degraded another.

I began building an evaluation harness around the behaviors I cared about. Some measures were ordinary operational metrics, including reliability, error rates and latency. Others had to be specific to the product: whether the coach applied the right principle, whether it stayed within the intended use case, whether it introduced unsupported guidance, whether it distinguished my explicit methodology from its own inference, and whether its recommendation reflected the judgment the product was designed to provide.

Turning those expectations into testable cases made the development process substantially more disciplined. Prompt changes, knowledge changes, retrieval changes and model changes could all be evaluated against a relatively stable set of situations instead of being judged entirely by feel.

The system is not perfectly objective, nor do I think language-model evaluation can be reduced to a single score that settles every product decision. Some qualities still require judgment, and an evaluation set is only as useful as the behaviors it represents. The important change was that I now had an explicit definition of product quality that could be examined repeatedly rather than an implicit standard that existed mostly in my head.

That distinction also makes customer feedback more useful. A customer reporting an unsatisfactory interaction is not merely providing an anecdote to patch. The failure can be classified against the behavior the system was supposed to exhibit, incorporated into the evaluation set when appropriate, and used to determine whether a proposed improvement solves the underlying problem without creating another one elsewhere.

Better foundation models create a new reason to evaluate

The evaluation system also changed how I think about model upgrades.

It is easy to assume that replacing an application's foundation model with a more capable model will improve the application. That may be true on average, but an application does not operate on average. It depends on a particular set of behaviors, constraints and customer expectations.

A new model may reason better, follow instructions more reliably, or make fewer obvious mistakes. It may also bring more world knowledge, make different inferences, respond differently to ambiguity, or interpret instructions in ways that alter behavior I had deliberately constrained. For my coach, some of those changes could improve the product while others could make it less faithful to the methodology even if the model itself was objectively more capable.

Without an application-specific evaluation layer, a model cutover therefore introduces unmeasured drift risk. I can experiment with a new model, read a selection of answers and conclude that it appears stronger, but I do not know how its behavior changed across the full set of situations customers depend on.

The harness gives me a better process. I can run a candidate model against the same cases as the current production model, inspect where behavior improves or regresses, and decide whether the tradeoff is acceptable before the change reaches customers. If the model is substantially better but creates a particular regression, I can often modify the surrounding system and test again before making the cutover.

I expect this layer to become more important as foundation models become more capable. Greater general intelligence increases the value of the underlying technology, but it does not eliminate the need for a product owner to define what better means for a specific customer experience. I want improvements in the foundation model to become improvements in my product deliberately, rather than allowing the behavior of the product to drift whenever one of its dependencies changes.

The improvement loop has to be easy to operate

Building evaluation infrastructure solved another problem while creating a new one. An eval harness is useful when I am actively working on the system, but the coach produces evidence continuously. Customers encounter situations I did not anticipate, product metrics change, new failure modes appear, and my own coaching methodology continues to develop.

For that evidence to matter, it needs a path back into the product. I have been building a continuous-improvement process that can identify a potential issue, connect it to the relevant product behavior, propose a change, evaluate the resulting system, and put the decision back in front of me before anything reaches customers.

The technical sophistication of that process matters less to me than the cost of operating it. If participating in the improvement loop requires opening a development environment, finding the relevant files, running tests manually and managing a release every time the coach needs my judgment, I will inevitably participate less often.

I therefore designed the human interface to the loop around something much simpler: email. An issue can reach me with the relevant context and proposed action, and I can steer the next step from my phone with a minimal interaction. The machinery behind that decision can remain comparatively sophisticated, but the burden placed on the person whose judgment is required should be small.

This has become another general lesson from the project. Continuous improvement is partly an evaluation and automation problem, but it is also an operating-model problem. The people who understand the product need to be able to participate in the loop frequently enough that the system continues to reflect their judgment. Reducing the friction of that participation can matter as much as improving the machinery underneath it.

The AI coach changed the course too

One of the more useful consequences of this work was not originally an AI feature at all. Reorganizing my methodology for the coach forced me to examine the methodology more critically, and the same weaknesses that made information difficult for the model to use sometimes exposed opportunities to make the material clearer for humans.

Ideas that seemed obvious to me had to become explicit. Advice that worked in context needed clearer boundaries. Relationships between principles became easier to see once I had to formalize them, and defining evaluation criteria forced me to ask more rigorously what outcome each piece of guidance was supposed to produce.

Customer feedback added another source of evidence. As I became clearer about which problems people were trying to solve and where they struggled, I could look back at the self-directed course and ask whether it was organized around those outcomes as effectively as it could be.

The course is now undergoing a substantial overhaul as a result. That matters because one-on-one coaching is not the right product for every customer, and it is particularly difficult to justify for some more junior candidates who do not have the budget for an intensive coaching experience. The self-directed product therefore needs to be capable of producing useful outcomes on its own, rather than existing primarily as supporting material for customers who can also work with me directly.

In that sense, the development loop has begun to run in both directions. The course provided much of the original intellectual foundation for the AI coach, while the work required to make the AI coach reliable has given me a more rigorous way to examine and improve the course.

What I would do differently now

When I started, I thought the core challenge was transferring my expertise into an AI product. I now think that description gives too much weight to the transfer and too little to the product system surrounding it.

Foundation-model capability was necessary, but access to a capable model and a collection of domain content did not create a differentiated coach. The differentiation began to emerge only when I defined the jobs the product should perform, reorganized the knowledge around those jobs, constrained how the model could use that knowledge, made the intended behavior explicit, and built a way to measure whether the system was actually behaving as intended.

The evaluation and improvement layers changed the nature of the product again. They gave me a way to learn from failures instead of simply patching them, to make changes without relying entirely on intuition, and to evaluate future foundation models against the customer experience I am responsible for before putting those models into production.

The project has also made me less interested in whether an AI product has access to impressive underlying intelligence. That capability is becoming broadly available. I am much more interested in the systems that determine how that intelligence is applied: what the product knows, which judgments it is supposed to make, where its freedom ends, how quality is measured, and how the people responsible for the product continue to steer it as both customer needs and foundation models change.

Those systems are what turned a general-purpose model that knew a great deal about interviewing into something that could begin to behave like a Product Simply coach. Building them has also forced me to understand my own coaching product more precisely, which may prove to be the more durable lesson.