Good Survey Questions

  • There are 5 primary types of questions:
    • Behavior - What people do... their actions (What sports do you watch on TV?)
    • Beliefs - What people think... what is true and false for them (Estimate the percentage of minority owners in major sports.)
    • Knowledge - The accuracy of people's beliefs (How many minority owners are there in major league baseball?)
    • Attitudes - what people think is desirable as opposed to true or false. (Do you think that major league baseball should promote minority ownership?)
    • Attributes - Characteristics (age, sex, etc.)
  • Open ended questions leave room for respondents to answer how they wish without any suggested answers.
    • These questions allow the respondent to formulate their own answer to the question, which is good if:
      • you do not know all the possible answers.
      • you want more in-depth responses.
    • But can be bad because:
      • they can be very hard to code.
      • they hurt responses from less talkative people
  • Closed questions have set options for the respondent to choose between.
    • These offer much easier coding.
    • But care must be taken to formulate good sets of answers.
  • All types of questions are important in most research
  • It is important to properly word questions and make sure that the questions accurately reflect the concepts that you want to study. (see validity)

Examples of bad questions and how to fix them:

Bad question: "Do you think arbitration in MLB exacerbates the current epoch of turmoil between the union and the owners?"

Problem: Using big words, technical terms, and/or abbreviations can confuse and discourage respondents.

Better question: "Do you think the baseball players' right to have a judge determine salary hurts the already stressed relationship between the players and owners?"

 

Bad question: "Given the current trend of more hits, more home runs, longer games in general, and more injuries in baseball today, do you think that steroid use should continue to be banned even though it is not enforced?"

Problem: Long questions can be vague and confusing.

Better question: Steroid use has both positive and negative effects on baseball. Do you think that steroid use should be banned?"

 

Bad question: "Do you watch baseball regularly?"

Problem: Vague questions are difficult to answer (what is the meaning of "regularly"?, watch baseball it what form?)

Better question: "How often do you watch baseball on TV?"

 

Bad question: "Are the Cubs and Braves good baseball teams?"

Problem: Double-barreled questions ask two separate things (perhaps the Cubs are bad and the Braves are good)

Better question: "Are the Cubs a good baseball team?"

 

Bad question: "Even though steroid use makes baseball more exciting, it is still banned. Do you think that it should continue to be banned?"

Problem: Leading questions set the respondent to answer one way (rather than what they really think).

Better question: "Should steroid use be banned?"

 

Bad question: "Barry Bonds agrees with most other professional players when he says that the umpires need to speed up games. Do you think that umpires should take this initiative?"

Problem: Prestige bias skews the data because people have the tendency to agree with people who have prestigious jobs or titles.

Better question: Do you think that umpires should take the initiative to shorten baseball games?"

 

Bad question: "Young children are killing themselves by trying chewing tobacco after they see professional baseball players do it. Should chewing tobacco be banned in all levels of baseball?"

Problem: Emotional language causes people to respond to an emotionally-charged question differently than they would to a neutral one.

Better question: "Based on its effect on younger players, should chewing tobacco be banned in all levels of baseball?"