Survey Designer

---
name: survey-designer
description: Design effective surveys — question writing, response scales, sampling, bias reduction, and pilot testing. Use when collecting structured data from people and wanting answers you can trust.
category: ai-research
---

# Survey Designer

A survey is a measurement instrument. Like any instrument, it needs calibration: questions people 
understand the same way, scales that capture real variation, samples that represent the population, 
and testing before launch.

## Overview

Design in order: define what you're measuring (constructs, not just topics), write questions that 
operationalize them, choose response formats that fit, structure the flow to minimize fatigue and 
bias, sample properly, and pilot everything. Most survey failures are question-writing failures — 
ambiguous, leading, or double-barreled questions — and they're all preventable before launch.

## When to use

- Measuring attitudes, satisfaction, needs, or behaviors in a population.
- Evaluating a product, program, or intervention via self-report.
- Market or user research requiring quantifiable responses.
- Any "let's just send a quick survey" request — quick surveys need design most.

## Core concepts

- **Constructs**: what you're actually measuring (e.g., "trust," "usability") defined before any 
question is written. Each construct gets multiple items; single items are noisy.
- **Question writing**: one idea per question, plain language, no jargon, no leading ("How 
excellent was…?"), no double-barreled ("fast and reliable"), balanced response options.
- **Response scales**: matched to the question — agreement (Likert), frequency, satisfaction. 
5–7 points; label all points or at least endpoints; keep direction consistent or flag reversals 
clearly.
- **Flow design**: easy first, sensitive last; group by topic; progress indicators; respect time 
— every extra minute costs completions.
- **Sampling**: define the population, choose the frame, and sample to represent it. Convenience 
samples answer "these people," not "people." Weight when needed.
- **Bias control**: order effects (rotate), social desirability (anonymous, neutral wording), 
satisficing (attention checks used sparingly), nonresponse (track who doesn't answer).

## Practical workflow

1. Define constructs and the decisions the survey will inform. Write the analysis plan first — it 
dictates the questions.
2. Draft items: 3–5 per construct, plain language, one idea each. Borrow validated scales where 
they exist.
3. Choose formats and flow: group by topic, easy → sensitive, estimate completion time honestly.
4. Cognitive pretest: have 5 people think aloud as they answer. Fix every confusion they reveal.
5. Pilot with 30–50 real respondents: check completion rates, timing, missingness, and scale 
reliability.
6. Launch with sampling plan and response tracking; monitor for bias (who's not responding) during 
fielding.

```text
Pre-launch checklist:
[ ] Every question maps to a construct in the analysis plan
[ ] No double-barreled, leading, or jargon questions
[ ] Scales labeled; direction consistent
[ ] Flow tested for fatigue (time it yourself)
[ ] Cognitive pretest done; issues fixed
[ ] Pilot data checked: reliability, missingness, timing
[ ] Sampling frame defined; nonresponse tracked
```

## Common pitfalls

- **No analysis plan**: collecting data you can't analyze. Write the analysis before the questions.
- **Leading questions**: "How satisfied were you?" assumes satisfaction. Use neutral wording.
- **Double-barreled items**: "The support was fast and helpful" — which one am I rating? Split 
them.
- **Skipping the pretest**: launching untested questions. Five think-alouds catch most disasters.
- **Convenience sample generalization**: surveying your Twitter followers about "users." Know your 
frame's limits.
- **Survey too long**: fatigue produces garbage data at the end. Shorter beats comprehensive.