The quest for the holy grail: ask the right questions
Most survey failures are written into the questionnaire before a single interview takes place. What forty years of question-wording research actually says.
Survey questions look like the easy part. They are prose, everybody can write prose, and the temptation is to draft the questionnaire last, in an afternoon, once the interesting decisions about sample and timing have been settled. This is backwards. Sampling error is visible and can be quoted as a number. Measurement error is invisible and gets reported as fact.
Questionnaire design is not a matter of taste
For a long time questionnaire design was treated as a craft learned by apprenticeship. Substantial methodological work over the past forty years has established that much of it is testable science: particular wordings produce particular distortions, reliably, across topics and populations. That is useful news, because it means the common failures have names and can be designed out rather than argued about.
The failures worth designing out
Double-barrelled questions. Ask one thing at a time. A question that asks a respondent to evaluate two concepts at once — the price and the service, the product and the support — produces answers nobody can interpret, because you cannot tell which half was being answered.
Acquiescence. The agree/disagree format is among the most popular question forms in business research and one of the most treacherous. Less informed and less educated respondents show a measurably greater tendency to agree with whatever statement is put in front of them. If a finding rests on an agree/disagree battery, part of what has been measured is the format.
Open versus closed. Closed questions are cheaper to analyse and easier to compare over time; they also constrain the answer to the options the writer thought of. When the point of the study is to find out what you did not already know, a closed list is the wrong instrument.
Order. Questions asked earlier change how later ones are answered. This is not a defect to be eliminated — it cannot be — but it is a decision to be taken deliberately and then held constant, because changing question order between waves silently breaks a tracking series.
Iteration beats inspiration
Serious survey organisations do not write a questionnaire once. Drafts circulate, get argued over, get tested on real respondents and come back changed. That process is where the double-barrelled questions get caught. The most common reason a business survey disappoints is that this stage was skipped to get into field a fortnight earlier.
The practical test
Before a questionnaire goes out, take each question and ask what decision changes depending on the answer. If nothing changes, the question is costing response rate and buying nothing. Questionnaire length is a budget, and every question spent on curiosity is a question not available for the thing you actually needed to know.
