A survey can be carefully analysed and still give you an unreliable answer. Sometimes the problem was built into the questions before anyone responded.
The starting point is what you can reasonably expect a person to understand, remember and tell you. These examples are illustrative.
Begin with a real occasion
“How do you usually shop?” asks someone to summarise many different trips. They might picture a weekly shop, yesterday’s visit or the version of themselves they prefer to describe.
“Think about your most recent visit. What did you go in for?” gives the conversation a clearer reference point. You can then explore what happened, what changed and how the trip differed from other occasions.
A recent episode is not always representative. The choice depends on the question you need to answer, but be deliberate about the time period and check how memory could affect the response.
Explore behaviour before explanations
A useful starting order is to establish what happened before asking why someone thinks it happened. Otherwise a question about attitudes can steer how they describe their behaviour.
For example, establish what someone bought, what else they considered and where the choice was made before asking how important sustainability was to them.
Treat this as a design principle, not an inflexible rule. Some research needs a different sequence. The point is to consider what an earlier question might prompt a later answer to become.
Leave room for an unexpected answer
Closed options make analysis easier, but they also define what people are allowed to tell you. If your list misses the real reason, forcing a selection produces a tidy dataset with a misleading interpretation.
Where appropriate, explore the issue openly before supplying a list. Make room for “none”, “not applicable”, “I don’t know” or another answer when those are genuine possibilities.
Avoid making people evaluate a brand they have never encountered. Lack of familiarity is evidence; it should not be converted into an invented opinion.
Spend respondents’ attention carefully
A long sequence of similar ratings can reward speed and habit rather than thought. Every additional question also asks people to spend more time on something that may matter less to them than it does to you.
For each question, identify the decision it informs. Remove questions whose answers will not change anything. Pilot the wording and the flow with people who resemble the intended audience, and ask where they became confused or tired.
Check who is answering
Even excellent questions cannot correct for hearing only from the wrong people. A survey of loyal customers may be useful for understanding loyalty, but a poor basis for explaining why others stopped buying.
Consider who can enter the sample, who chooses to respond and who is missing. A larger sample can make a biased view look more precise without making it more representative.
Try it with your team
Choose five questions from a questionnaire or customer interview guide. For each one, ask:
- Could someone understand it without knowing our internal language?
- Could they realistically remember the answer?
- Are we asking one thing, or several at once?
- Can they give an honest answer we did not anticipate?
- What decision will this help us make?
Rewrite the weakest question, then test it in conversation before adding it to a larger programme.
Previous: Why more data does not automatically mean insight. Next: Brief the decision, not the method.
Have a perspective or a related question?
Discuss this piece