Stumped by Suction: Measuring Success for a Robot Vacuum

Even after completing thousands of coaching sessions, I'm consistently amazed at the fresh challenges candidates bring. Recently, a high-level Meta candidate presented a success measurement prompt that had both of us deep in thought. The scenario:

Imagine you work for a big tech firm, and they decided to build a robot vacuum. How would you measure its success?

This isn't as simple as it sounds. We batted around initial ideas, each with its own flaws:

• Number of times the vacuum started? What if sessions are negligible, or the vacuum fails mid-clean? That doesn't feel like success.
• How much dirt was collected? Intriguing, but incredibly hard to measure reliably. Plus, a small apartment might be impeccably clean, while a large, messy house might generate more 'success' without actually being a better product experience.
• Time spent vacuuming? This one seemed promising, but then we hit a snag: house size. Is a vacuum more successful just because its owner has a bigger house, leading to more vacuuming time? Not necessarily. We need to disentangle extraneous variables.
• Number of sessions without getting stuck? This is a crucial quality metric, but it’s too granular and complex to serve as the overarching North Star. We need something simpler, something that captures the core value.

The discussion highlighted a fundamental truth in product management: defining success isn't about picking the first plausible metric. It's about a deep, empathetic dive into the product's purpose and the user's true needs.

What's the Robot Vacuum Really For?

Before we measure anything, we must clarify the product's fundamental purpose. What problem is a robot vacuum solving? At its heart, a robot vacuum exists to clean floors autonomously, freeing up a user's time and effort. It's about convenience and cleanliness. The user (the 'demand side') wants cleaner floors without lifting a finger.

Now, how do we translate that core purpose into a measurable North Star?

Let's revisit the options, informed by a deeper understanding of product purpose:

• Number of starts: Still problematic. A start doesn't guarantee a clean floor. It's an input, not an outcome.
• Dirt collected: Too hard to measure accurately and consistently across diverse environments. How do you normalize for different levels of pre-existing dirt?
• Number of sessions without getting stuck: This speaks to quality and reliability, which are critical, but not the ultimate purpose of the product. It's a vital counter-metric or guardrail, but not the North Star itself.

This brought us back to time spent vacuuming. This measure scales with customer usage, where more = better. That's a key factor for the NSM. Although different customers have different size houses, the average house size will likely be stable and not be a major factor in the metric.

Overall, be sure to remember what your NSM needs to be: simple, aligned to the goal of the product, and goes up and to the right.