Pre-Testing

• Question development

• This makes sure that the questions can be clearly understood and have an adequate range of responses.

• This step eliminates possible errors made by people incorrectly interpreting the meaning of questions as well as ensuring that there is enough variation to actually analyze the data.

• In this stage, the questions are given to pre-testers who are aware that the questions are still being developed, and who are encouraged to give feedback on the questions.

• Key items in this stage are:
  • Variation - Check to see that people are actually giving different answers to questions. If they are not, perhaps there needs to be a more specific set of answers similar to the one which most people are checking. Without variation in a question no analysis can be done later.
  • Meaning - Make sure that the respondents understand each question and each answer to the question. You also must ensure that you can understand the responses.
  • Redundancy - Two question which measure the exact same thing should not be in the final questionnaire. For the pre-test, it is ok to have similar questions to see which works best, however redundancy in the final questionnaire is bad.
  • Non-response - Ensure that each question is answered by most of the respondents. If many people refuse to answer a particular question it will hurt your sample size. Questions which are confusing, long, extremely personal, provide inadequate responses, similar to previous questions, or have nothing to do with the main topic of the questionnaire can lead to reduced answers to questions.
  • Acquiescent response - Acquiescent response is the tendency people have to agree with something rather than negate it. You must ensure that if you are using likert scales or similar agreement measures that people do not simply agree with each statement. This can be done by taking two contradictory questions and seeing if people agree with both.

• Questionnaire development
• This allows you to see how people respond to the questionnaire as a whole.

• Important things to look for are:
  • Flow - The transitions between sections of the survey should be smooth and make sense. The questions should fit together in those sections. Contingency questions should easy to follow.
  • Question skips - People should not skip questions that they were supposed to answer and also should not complete questions which they were not supposed to answer. The spacing should be adequate and contingency questions should be obvious and have clear instructions.
  • Time - The time it takes to complete the survey is important to give a good estimate to respondents to the final survey as well as to personally budget your time if administering the questionnaires in person. Time is also important to make sure that the questionnaire is short enough so that people will actually agree to complete it.
  • Interest and Attention - You should determine if people are not responding well to certain parts because of length, monotony, or simply bad questions and layout.