​Survey Design

Steps in survey design
0. Make sure your results are generalizable to an appropriate population
We survey 1000, 2000, 5000 people. But we need to ensure that what we’re capturing here is generalizable to our entire customer base
Things to note:

• Population must be defined correctly (e.g. 25-30 year old males)
• Sample must be representative of the population
• Respondents selected to be interviewed are able and willing to cooperate
• Respondents understand the question and have knowledge
• Motivate them to provide information (incentive)

Problem: Response rate – in general, 5%
How to handle low response rates?

• Collect data on non-respondents

• Test the differences between those who respond and those who do not – the two should not be systematically different since the generalization is not good then.

For example: If you're interested in capturing people's responses of prices that you're charging for your product, and let's say there's a systematic difference between people who respond and people who don't respond in terms of their income.
That's clearly not very good because then what you're capturing is not a representation of the true target market.

• Follow-up non-respondents

• Try to convert them into responding
• Could be a problem if they give bad data after repeated requests – compare waves of respondents to see their responses
• Compare each wave to make sure they are similar

 
1. Develop detailed listing of ‘bits’ of information
Ensure that you translate research objectives into information requirements.
Check for relevancy à What would you do with the answer if you knew it?
2. Determine appropriate data collection method for each question
Formatting the question can be done in two ways:

• Open-response questions à Why do you shop at Genuardi’s?

Advantages:

• Respondents can give general reactions to questions such as “Why do you say that brand X is better?”
• Response given in “real world” terminology, i.e., consumer’s own language
• Can help interpret closed ended data à e.g. Why was color the most important product attribute?
• May suggest additional alternatives to be used in closed-ended questions.

Disadvantages:

• Often not good for self-administered surveys
• Answers depend on respondent’s ability to articulate
• Post-coding is tedious

• Closed-response questions à Respondents are provided with pre-determined descriptions and selects one or more of them. e.g. How often do you shop at Genuardi’s? (Scale: Very often to Not at all)

Advantages:

• Easy to use in field
• Less threatening for respondent
• Simple to code and enter data
• Cheaper to administer

Disadvantages:

• Usually requires pre-testing
• Presumes the list of responses is complete

Best practice: Use open-ended questions for exploratory research and use closed-ended research for quantitative research
 
3. Write draft questions

• Use simple, conventional language

Example:
Did you notice any malfunction at the time of purchase?
OR
Did you notice anything wrong with it when you bought it? (easier for the general audience to understand)

• Avoid leading and loaded questions
• Avoid ambiguity: Be as specific as possible
• No long questions (not more than 20 words)
• Start broad and then narrow down

4. Design Flow and Layout
Layout Guidelines

• Open the survey with an easy and non-threatening question
• Smooth and logical flow
• From general to specific
• Sensitive or difficult questions should not be placed at the beginning of the questionnaire

Order Bias (How we order the questions, where is it being asked in the overall survey is extremely important)

• Common bias: fatigue
• Randomize ordering of your questions

 


​










​5. Evaluate/Pilot test/ Redraft

• The idea here is to make sure that before we implement the survey with your final target sample, just clean out the survey.
• Make sure all the questions that you do are meaningful.
• Ensure that people who are interested in answering the surveys, they understand what the questions are.
• You have designed the questions with the respondent in mind. What that means is, you make sure that you're using verbiage or language, that the respondent would be familiar with.
• Ensure that you're asking questions which go from more general to more specific.
• Always lead off with questions which are non-threatening so that consumers or respondents who are answering the survey can easily get into the survey.

6. Get approval from all parties
If you're working within your company or if you're a consultant working for another company, get approval from all the parties. Extremely important, of course, if you're to implement the survey with customers
7. Pilot test/ Redraft
Similar points as in Step 5.
8. Final copy/ implementation
 

---