Survey Design Guide

© Dr. Karl Albrecht. All Rights Reserved.

Users of Dr. Karl Albrecht's SurveyMaker software product can create their own survey questionnaires quickly, easily, and conveniently. Here are some practical guidelines for designing questionnaires that will get you the data you need and help you achieve your business objectives.

Rule #1: Keep the end user of the data in mind when designing your questionnaire.

The person answering the survey only sees it once, but the people who read the results will constantly benefit from (or be victimized by) the design thinking that goes into the questionnaire. If it is unnecessarily long, repetitive, poorly sequenced, or has poorly worded questions or multiple-choice scales, your clients will be distracted from interpreting the reports based on the results. Design the questionnaire for the end user of the data, while keeping the respondent in mind.

Rule #2: Keep your survey questionnaire as short as possible.

Some surveys should be no longer than five or ten questions, especially when the respondents are busy, in a hurry, or otherwise not highly motivated to answer, such as in an airport or shopping mall. For other surveys, 20-25 questions may be the limit. Very seldom should you use more than 50-60 questions on a survey, unless you are confident the respondents will take the time to answer truthfully and thoughtfully. It is rarely advisable to use more than 100 questions on a survey. If your questionnaire gets to be very lengthy, you may be trying to accomplish too many things with one survey project. You may actually be trying to do two projects, or even more, in one survey.

Rule #3: If you're working with a group or task force, don't let a committee write the questions.

Have one literate person compose a draft and let others comment. Know when to stop nit-picking and get on with it. Get agreement on the objective of each question, but not necessarily on the exact wording. In writing the actual questions, it is very important to phrase your questions skillfully to make sure you get reliable answers. If your respondents misunderstand your questions, you will misunderstand their answers. Here are some guidelines to keep in mind as you compose your survey questions.

• Keep each question as short as possible.
  

• Use simple, concrete terminology. Avoid terms the reader might not know. You'll be amazed at the number of ways people can misread or misinterpret questions.
  
• Ask only one thing with each question. Avoid "compound" questions, such as: "How do you like the quality and selection of our merchandise?" The customer may think your quality is high but your selection is poor, or vice versa; he or she won't know how to respond to the question. In some cases, you may want to combine two factors together, but be careful not to combine unrelated factors that will confuse the respondent.
  
• Use a simple and consistent pattern of presentation in the wording of your questions.
  
• Use "you" when possible; make it personal. You want the respondent to answer from his or her own point of view. Or, you might want to phrase your questions in the form of "I-statements," e.g. "I have opportunities to get ahead in this organization."
  
• Avoid "loaded" questions that imply certain positive or negative evaluations are appropriate. Don't "shop" for answers.
  
• Minimize mental gymnastics in answering the questions; don't make the respondent do calculations or work out logical conclusions.

Rule #4: Use the right type of question to get the right result.

To create your questions with SurveyMaker, you have five preformatted choices. You can use multiple-choice questions, numeric questions, list questions, ranked questions, or comment questions.

• Use a multiple choice question when you can offer the person a short list of pre-established answers that will tell you what you want to know, i.e. range of opinions, male/female, education levels, or degrees of satisfaction.
  
• Use numeric questions for continuous variables like age, number of years at current residence, or the number of people in the family. You can also use numbers as category identifiers if there are many different subgroups in your population.
  
• Use a list question when you want the respondent to select from a group of choices, in the case where any or all of them might apply. For example, you might ask "Which of these best-selling books have you read in the past 12 months?"
  
• Use a ranked question to ask respondents to evaluate a group of choices against one another, i.e. number them in order of value, importance, or priority.
  
• Use a comment question when there is no way to predict the nature of the answer. Comment questions allow the person responding to express the answer in his or her own words.

For some question items, you need to think carefully in choosing between the various formats in deciding which type to use. With the question of age, for example, you can either ask for a specific number, or you can divide the range of expected ages into bands and assign each band to a multiple-choice option.

If you have any doubt about which format to use, think about how you will actually be using the information. Numeric questions allow you to make finer distinctions in the population, because they are continuous variables. Multiple-choice formats, however, offer simplicity and convenience in processing the data.

Rule #5: Don't misuse multiple choice questions.

For most opinion surveys, it is customary to use multiple choice questions, or scalar questions, as the primary means for asking people about their opinions. The multiple-choice form is familiar, easy to read, easy to answer, and easy to analyze. But many survey builders misunderstand the principles of using scalar questions. It is important to use scalar questions consistently, to make the survey reports easy to read and interpret.

Decide on the number of options for your multiple-choice questions. The most common scale for multiple-choice questions in opinion surveys is the five-point Likert scale. Dr. Rensis Likert, of the University of Michigan, developed this scale many years ago for behavioral sciences research. It is widely accepted because it offers a convenient range of choices that meets the needs of most situations.

Some people like to use other kinds of multiple-choice scales, such as the seven point scale or even a ten point scale. Advocates of the seven point scale claim it gives them the ability to make finer distinctions in the measurement of opinion. Advocates of the five-point scale claim the aspect of finer distinctions is an illusion, and that the average of the scale values provides the same differentiation for measuring opinions which are, after all, subjective variables at best.

In some cases, it is possible to "anchor" the various points on a rating scale to some verifiable criteria, such as monetary amounts, objective degree of illness, medical diagnosis, or observable behaviors. In those cases, a wider rating scale might be appropriate. Survey experts call these anchored rating scales.

Bear in mind, however, that long scales result in reports that are more difficult and tedious to read. When you are laying out your report in tabular format, i.e. with each question taking up one row and the various options arranged in columns, you may have to reduce the font size to make seven or ten columns fit across the page. Or, you might need to change the layout to "landscape" style to fit all the columns on the page. In addition, forcing the reader to study ten columns of frequency values can make it more difficult than necessary to interpret the "spread" of data, especially when the wider scale does not convey any additional precision.

Rule #6: Don't "Lead The Witness."

The Likert scale presents a person with five options, ranging from "least to most" or "most to least." There is some debate among the experts as to which sequence to present. Some people like to put the most "negative" or critical option first on the list and finish with the most positive option. Some people prefer to put the most positive option first and have the others progress "downhill." They may feel that one sequence or the other will create a mental bias in the respondent, causing him or her to select a more positive or more negative answer.

It is best to avoid such psychological hair-splitting. Most people are accustomed to associating high numerical ratings, such as 4 or 5, with positive evaluations. Using 1 for a high score and 5 for a low score can be confusing to them. Consider that the people reading your report will tend to think of a higher mean value as "better" than a lower one. When in doubt, stay with the old rule, "big number means good score."

Whichever form you choose, use it consistently throughout your questionnaire. Don't change back and forth from using 1 as the highest and 5 as the lowest and then to 1 as the lowest and 5 as the highest. That will confuse your respondents, your data entry person, and probably yourself. It will certainly confuse the people who will read your survey report.

Many survey takers like to make the multiple-choice question a declarative statement with which the respondent can agree or disagree using a scale from 1 through 5. An example might be:

"Product X gives good value for the price."

1 = strongly disagree
2 = disagree
3 = neutral; no strong feeling
4 = agree
5 = strongly agree

Rule #7: use sequence and idea flow to your advantage.

Think about the logical flow of ideas implied by your questions as the respondent reads them one at a time. Also, think about how the questions relate to overall themes as your clients read the report. Consider placing the more personal questions, e.g. age, gender, marital status, etc. at the end of the survey. This has two advantages. First, people responding to the survey may feel more comfortable with these types of questions once they have provided answers to the main body of the questionnaire. Seeing these questions at the very top of the questionnaire can cause a person to feel his or her personal identity is being encroached upon. Second, if the questionnaire has many questions, people may experience "survey fatigue," and feel less motivated to answer carefully and thoughtfully as they proceed. They may find these types of demographic questions easier to answer and more palatable at the end.