Online business panels: extending the capability of online research
Panels made online business research affordable. The accuracy evidence explains what you are trading away, and where the errors actually come from.
A standing panel turns a research project into a purchase. Instead of building a sample from nothing for every study, you draw from a pool of people who have already agreed to take part and already told you who they are. For low-incidence business audiences — people who specify a particular class of equipment, people who hold a particular budget — this is often the only affordable route.
It is also a methodological compromise, and it is worth being precise about which one.
Opt-in is not a probability sample
Panellists select themselves. That means the sample has no known selection probability, and the inference is model-based rather than design-based: the numbers depend on the weighting scheme being right, not on the mathematics of random selection.
The comparative evidence is unambiguous about the size of the effect. In a large study comparing probability-based online panels with opt-in online samples, the average absolute error on the opt-in samples was about twice that of the probability-based panels.
The surprise in the error
The interesting finding is where the error came from. The intuitive assumption is that opt-in samples go wrong because of who joins. The evidence points somewhere else: a large share of the bias traced to measurement error from bogus respondents — people making no genuine effort to answer truthfully.
The practical consequence is that the errors are worst exactly where researchers most want detail. Subgroup estimates suffer disproportionately, because a fixed number of fraudulent respondents is a much larger proportion of a small cell than of the total.
What this implies for business panels
Verify the respondent, not just the profile. In business research the claim being made — that this person really does authorise that spend — is the whole value of the sample. Screeners that can be guessed will be guessed.
Instrument for fraud. Attention checks, timing distributions, straight-lining detection and duplicate device checks are not an insult to respondents; they are the only defence available in a mode with no interviewer.
Be careful with small cells. A panel study that reads confidently at the total level may be noise at the level of the four sub-segments the presentation is actually about.
Say what the sample is. An opt-in panel study reported as though it were a probability sample is a reporting failure, not a sampling one. Describe the source, describe the weighting, and let the reader discount appropriately.
