references/survey_methodology.md
Survey Methodology
Reference for survey design and sampling. Pairs with sample_size_planner.py.
Sample size from first principles
For estimating a proportion, the required sample is n₀ = z²·p·(1−p)/e², where z is the critical value for the confidence level, p the expected proportion, and e the margin of error. The maximum-variance choice p = 0.5 is the conservative default. When the sample is a meaningful fraction of a finite population N, apply the finite-population correction: n = n₀ / (1 + (n₀−1)/N). The planner does both.
The most common analyst error: powering the overall sample but then reporting per-segment results that the sample cannot support. If you will report three segments at ±8%, each segment needs ~150 respondents — so the survey must be sized to the segment floors, not the aggregate. The planner computes per-segment minimums explicitly.
The total survey error framework
Sample size only addresses sampling error. Groves' total-survey-error framework names the others, often larger:
- Coverage error — the sampling frame omits part of the population (e.g., an email panel misses non-users).
- Non-response error — those who respond differ systematically from those who don't.
- Measurement error — the instrument itself biases answers (question wording, order, scale).
A tight margin of error on a biased frame is precision without accuracy.
Question design
- Avoid leading questions that imply a preferred answer.
- Avoid double-barreled questions that ask two things at once ("Is the product fast and reliable?").
- Watch scale and order effects — response options and question sequence shift answers.
- Pre-test every instrument with a small cognitive-interview pass before fielding.
Standards
AAPOR (American Association for Public Opinion Research) publishes disclosure standards and the standard definitions for response-rate calculation. Reputable market research follows them so results are comparable and auditable.
Sources
- Cochran, W.G., Sampling Techniques, 3rd ed. (Wiley, 1977).
- Dillman, Smyth & Christian, Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th ed. (2014).
- Groves et al., Survey Methodology, 2nd ed. (Wiley, 2009) — total survey error.
- Schuman, H., & Presser, S., Questions and Answers in Attitude Surveys (1981) — wording/order effects.
- AAPOR, Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.
- Tourangeau, Rips & Rasinski, The Psychology of Survey Response (Cambridge, 2000).