Accuracy doesn’t help you if your question doesn’t matter

Let me ask you a question. Does your business have money to burn?

Most don’t, and yet I see a common behavior that seems to suggest otherwise.  The pursuit of high accuracy about questions that don’t matter.

Let me explain what I mean.

In your business, there are probably many things that will affect the success of your product.  If you are starting a new product, you should be thinking about:

  • Does your product solve a problem that people actually have?
  • Do those people have money?
  • How do those people find out about your product?


If your product is already well established, you should be thinking about:

  • Are your customers happy?
  • Are they having a good experience with your company/product?
  • What other needs do your customers have that you might be able to help them with?
  • How can you attract more people to your product?

And of course, there are always these big issues to think about:

  • Competitive landscape
  • Economic environment
  • Quality and capabilities of your team

There are many more but this is a good starting list that would be uncontroversial with most business owners.

And yet, I see many research projects that try to answer questions such as:

  • Do people prefer the word “Health Clinic” or “Health Center”?
  • How likely are people to bring mobile devices into the bathroom?
  • How do people like this logo vs. that logo?

(I am not making these up.  I have seen companies sink tens of thousands of dollars into this type of research)

Typically, the bigger the company, the smaller the questions they try to answer.  And here is the kicker.  Because they want to have “statistical significance”, they pull from a very large sample, which dramatically increases the costs of the research.  The large sample size does indeed give them more accuracy, but accuracy doesn’t help you if your question doesn’t matter.

The reason that companies spend comparatively little time on the bigger questions is because, well frankly, those are a lot harder.  If your product serves a very specialized niche, or if it is Business-to-Business rather than to Consumers, it can be very difficult to talk to a large sample size.  Prohibitive even.  As a result, people sometimes don’t bother doing it at all because it won’t be statistically significant.

People often confuse statistical significance with accuracy.  First of all, they are not the same thing.  Second of all, there is no magic number for the accuracy you need for a business decision.  And, more often than not, you don’t need a high level of accuracy to inform a decision.

Let’s say you want to find out if people think your product solves a problem that they would be willing to pay for.  Does it really matter whether that number is 90% of your target market or 80%?  In practical terms, what would you do differently if that was the case?  Probably not much.

For early product ideation, you are usually trying to make a “proceed, adjust, or kill” decision”.  For that kind of decision, you can work with some pretty wide margins of error.  You can easily work with only three buckets: “low”, “medium” and “high”.

  • If the percentage of people interested in your product is low (less than 30%), people are not really interested, and you should kill the project.
  • If the percentage of people interested in your product is moderate (between 40%-70%), you should probably make some adjustments and check again.
  • If the percentage of people interested in your product is high (greater than 70%), you should proceed to implementation.

Notice that these are pretty wide intervals.  You just need to know low, medium or high.  You don’t need to know whether it is precisely 5% or 20%.  Either way, you should kill the project.   Data with a wide confidence interval can help you make go or kill decisions pretty quickly, at relatively little cost, and the benefit to your business could be huge.  It is also a good way to iterate on your product idea quickly without running up a huge research bill.

Consider on the other hand, a typical market research project that might ask “Should we call the product Awesomeness on Toast or Magic on Bread?” and then proceed to survey 2000 people about this.  Certainly, they will get accurate results with this approach.  However, if no one wants the product in the first place, it won’t make much impact on its success or failure.

Here is the key point.  It is far better to have less accurate answers to questions that matter to your bottom line, than to have accurate answers to questions that don’t.

Don’t burn your money.  Spend it on what matters.