Rememberthe song from school about the man who built his house on sand? I must have sung it with real conviction, because it still comes back to me whenever asurvey goes wrong in field. The house didn't fall because of the storm. It fell because of what was underneath it.
Poor survey data works the same way. Whether results are inaccurate, unusable, hard to interpret or simply short on completes, the cause usually sits further back than fieldwork: in decisions made about the sample, the questionnaire and how the data is collected.
Below are ten pitfalls we see regularly across the survey lifecycle, and how to avoid them. Most are cheap to fix before launch and expensive to fix once fieldwork is under way.
Let's start with the foundations.
Phase 1: Sampling and methodology
1. Building the sample around legacy assumptions
A large body of previous research should make sampling easier. Sometimes it does the opposite. Criteria build up from past studies until the target respondent is a composite who barely exists in the real world.
In field, response rates drop and costs rise, because every extra screening question removes more of the available population. It also leaves no room to discover anything new about the market.
Keep it real: challenge every screening criteria. If it won't change the analysis, it shouldn't narrow the sample.
2. Skipping the feasibility check
The brief asks for builders who use product X. Halfway through fieldwork it becomes clear that product X usage can't be profiled or sourced at the volume needed. The incidence was assumed, never tested.
The problem spreads to the questionnaire too. Questions written for regular users don't work for respondents with only passing familiarity, and without alternative routing they're forced to answer anyway.
Keep it real: check incidence with your fieldwork partner before the questionnaire is signed off, and agree a fallback in advance, whether that's a wider definition, adjusted quotas or alternative routing.
3. A sampling frame that leaves people out
A sample can meet every client requirement and still be biased if the sampling frame excludes part of the population. A digital-only approach, for example, will under-represent older people, those with limited internet access and many vulnerable customers, who are often the people whose views matter most.
If the frame is wrong, weighting can't fully repair it. Consider whether a mixed-mode approach, such as adding telephone interviewing, is needed to reach the whole audience.
Phase 2: Survey design
4. Designing without an analysis plan
This is the one that causes most of the others. If you don't know what you'll do withthe numbers once they arrive, you'll collect data you can't use and miss the insight you actually needed.
Before writing a single question, agree the outputs: the key measures, the cuts by audience, and the decisions the research has to inform. Every question should then earn its place by feeding one of them.
5. Asking two questions in one
Double-barrelled questions are everywhere: "How satisfied are you with the location and ease of access?" A respondent may love the location and struggle with access, so whatever they answer, you can't tell which part they meant.
In an online survey this produces ambiguous data. On CATI or CAPI it's worse. Respondents ask for the question to be repeated, hesitate while weighing two different things, and the interview slows down. Ask about one thing at a time.
6. Length, fatigue and the curse of "good to know"
I have a rule in my house: the phrase "good to know" is banned (I still use it occasionally). In survey design, most of what gets labelled "good to know" is dropped the moment reporting starts. If a question doesn't feed the analysis plan, delete it.
Long, repetitive questionnaires cause fatigue, and tired respondents give weaker answers. There's a cost point too: on telephone surveys, every extra minute of interview length adds interviewer time to every complete. A shorter questionnaire usually means better data and a lower cost per interview.
Get someone outside the project to complete it. For telephone work, read it aloud and time it. If they're bored, your data is already compromised.
7. Biased scales and dead-end routing
Two related problems push respondents towards answers that aren't true.
The first is unbalanced scales, such as three positive options, one negative and no genuine neutral point. The second is routing with no way out. We've seen postgraduate students asked which year of their undergraduate degree they're in, with no option that fits, so they have to miscode just to move on.
Make sure every respondent has a valid answer to every question they see. That might mean "Other", "Don't know" or "Not applicable", or it might mean better routing. Test each path through the survey for each type of respondent before launch.
Phase 3: Fieldwork and data collection
8. Launching without a soft launch
Going straight to full fieldwork means problems surface at 200 completes instead of 20. A soft launch, followed by a proper check of the early data, shows whether the routing works, whether the interview length matches the quote, whether the screen-out rate matches the incidence assumed, and whether any questions are being misunderstood.
It usually costs a day. It can save the project.
9. Not designing for the mode
A questionnaire written for online and then used unchanged on the telephone rarely works. Long grids, fifteen-item lists and visual scales are hard to follow when read aloud, because respondents can't hold every option in their head. They tend to pick the first or last thing they heard.
The reverse applies too. Open questions that produce rich answers with a skilled interviewer often produce one-word answers online. Adapt the questionnaire to the mode: break up grids, shorten lists and rethink open questions.
10. Weak quality controls on online sample
Bots, speeders, straight-liners, duplicate respondents and AI-generated open-ended answers are a growing problem in online research. Basic automated checks catch some of them, but not all.
Agree quality checks and data-cleaning rules before fieldwork starts, review open-ended answers by hand, and be prepared to remove poor-quality completes even when it hurts the timeline. Our team explains how we approach online screening in more detail.
Before you launch: a quick checklist
- Is the analysis plan agreed, and does every question feed it?
- Does every screening criterion change the analysis?
- Has incidence been checked with your fieldwork team?
- Does the sampling frame reach the whole target audience?
- Has someone outside the project completed the questionnaire (or, for telephone, read it aloud and timed it)?
- Does every respondent have a valid route through every question?
- Has the questionnaire been adapted for the mode?
- Is there a soft launch, with a data check before full launch?
- Are quality checks and cleaning rules agreed before fieldwork starts?
How Field Locker can help
Most of these problems are far cheaper to fix before fieldwork starts than during it. Our project and commercial teams check sample feasibility, questionnaire length, routing and mode suitability on every project we quote for, so issues are flagged before they cost you time or data.
Planning a survey? Send us the brief or draft questionnaire and we'll tell you where we'd expect it to struggle in field.
Get in touch
.svg)
