Researchers like their statistics. There’s something about having a number that’s comforting. It’s a specific number, right there on paper – or screen. We even get to talk about the margin of error and the level of certainty that goes with it.
All this makes it really easy to forget a very important point – accuracy and precision are two different things.
Accuracy often becomes an unintentional casualty in our quest for precision by using bigger scales and more detailed questions.
How can that be?
When I work with a client on a survey, often the conversation about using a 10-point or a 7-point scale instead of a 5-point scale comes up. The usual line of thinking is that more points on the scale allow the respondent to be more precise with their answer – so our research is more precise. This statement is true, but it ignores a more important point. More choices – like when there are more points on the scale – make it more difficult for the respondent to answer the question. Honestly, could you explain the difference between rating one brand a 6 and another brand a 5 on a 7-point scale?
Probably not. Certainly not in any consistent way.
The same is true when we try to drill down to a very micro level about someone’s attitudes and behaviors. There is a ton of research – particularly when it comes to behavior – that tells us that people’s self-reporting is often wrong. Yet we persist in asking very detailed questions. We would be much better off sacrificing some of the alleged precision we think we’re getting for a question that is easier to answer, and therefore more accurate.
It all comes down to paying close attention to the questions you are asking.
I’ve written about this before – ask questions that people can answer! That will often mean sacrificing “precision.” What you get in return, though, is accuracy. That’s a trade off I’m willing to make anytime.