When planning a research project, many people just start writing a questionnaire and organize their thinking on the fly. Perhaps there is some vague objective, like “measure the effectiveness of our ad.”
For those thinking “What’s wrong with that objective?!” I’ll have more to say about setting objectives in a future tip of the week.
Using this approach to planning a survey is like jumping in the car and driving away without knowing where it is you are trying to go.
So, what do you do? Start from the end.
By end, I mean start from a well defined objective. Think about what calculation you have to make or what piece of information you need to achieve the objective. Next, figure out what inputs you need to do that, and what inputs you need to get these inputs. Continue this until you work your way back to a single, specific data point.
An example: The objective is to see if your ad changed people’s opinion about the quality of your product. To do this, you need to compare the opinion of quality held by those who saw your ad and those who didn’t. To make that comparison, you need two data points:
- Did the person see the ad?
- What do they think about your product’s quality?
Depending on how you advertised, knowing who did and didn’t see the ad may be quite easy (email) or not so easy (broadcast, print, etc). In the case of email, if you tracked the campaign, you already have the answer to whether or not someone saw the ad.
Asking about an opinion of product quality is a fairly straightforward thing to do.
By starting from the end, we were able to do several things to get better research:
- We identified who we needed to talk to
- We identified exactly what pieces of data we needed from them
- We determined whether we had to ask a question or we could get the data from existing sources
- You are sure of getting exactly the information you need and your survey respondents are spared wasting time answering questions that you can’t use or don’t need.
Win – win.
While meeting with a potential client the other day, I was reminded yet again of what a small world we live in .
Our conversation, a mix of business opportunities and war stories, stumbled upon a number of familiar names. Now, this is someone whom I had only met once before this meeting. Yet, here we are finding person after person that we had each worked with, worked for, or personally knew in some context. In all, a 30 minute conversation turned up no less than a half dozen common links, and we weren’t even trying!
This certainly wasn’t the first time I’ve had this experience. Old clients from different companies who have moved to new jobs have landed in the same departments at new companies several times. A former employee, from several years back, is now a client. Two other former clients now work together at another research firm; one works for the other. A colleague and I have been crossing paths in various capacities for 10 years. First as competitors in a very small niche market, then as colleagues at a research firm, now as collaborators each running our own firm. And on and on.
The lesson is that this business, really all business, is about relationships. If you plan to be in a profession for any length of time, the relationships and reputation you develop (or don’t) will make or break you in the end.
Is this an earth-shattering revelation? For most people, no.
Is it something that most of us need to pay more attention to? Yes.
A quick trip to Amazon or any bookstore will confirm that one could fill a library with books about how to network and how to build better relationships. In my opinion, one of the best reads in this genre, both in style and substance, is Never Eat Alone by Keith Ferrazzi.
Haven’t read it yet? Get it and read it.
One of the basic statistical concepts that we in research deal with every day is the notion of “statistically significant” differences. This – in my experience – is also one of the most misunderstood concepts among my clients. So, here is a plain English explanation of everything you need to know about “statistically significant.”
What I most often hear from clients is that they want a “statistically significant sample” or “how many people do I need to be statistically significant?” This is the wrong question, which comes from confusing a few different ideas with one another.
- “Statistically valid”: A sample of at least 30 records (people, households, buildings, companies, whatever your sampling unit is) that was selected from the population using some appropriate sampling procedure.
- “Statistically significant”: There is a very high probability that the difference between two measures, or between a measure and a benchmark value is not the result of random variation, but of real differences between the items compared. The idea of statistical significance exists only when comparing two or more values.
- “Significance level”: The threshold probability to be considered “statistically significant.” Usually expressed as a percent. Commonly used significance levels in market research are 80%, 90%, and 95%.
- “Margin of error”: The margin of error defines the range of values that contains the “true” value. The margin of error is related to the significance level, the sample size, the variance, and the measured value itself (for proportions). When you see a mention of a survey being “+/- 3%,” you are seeing the margin of error. ‘So, when a research result such as the following is reported:
- “We tested the difference in our measurement of product interest for Group A and Group B using a significance level of 95% and found the differences to be statistically significant.”
What we mean is that there is at least 95% probability that the measured values between Group A and Group B are different because of real differences between the groups, not random variation.
When we say: “The purchase intent for the new product is 62%, with a margin of error of plus or minus 3%.”
We mean that the actual purchase intent is somewhere between 59% and 65%. The significance level also comes into play here. If the significance level was 95%, then we’re saying that there is a 95% probability that the actual value is between 59% and 65%. For any given measurement, the higher the significance level (i.e. closer to 100%), the bigger the margin of error. This is simply saying that we can be more confident that a value falls within a larger range.
Returning to our original question “How many people do I need to be statistically significant?” should be “How many people do I need to be statistically valid?” This question always has the same answer.
I hope this has helped shed some light on these important ideas in market research. For further reading, the best introduction to basic statistics I have ever read is:
The Cartoon Guide to Statistics: http://www.assoc-amazon.com/e/ir?t=wwwericksonmr-20&l=ur2&o=1 by Gonick & Smith. As the name implies, its a comic book. It does an incredible job at explaining many basic statistical concepts in a very easy to understand way.