The Use and Misuse of Surveys

© Dr. Karl Albrecht, all rights reserved.

The fictional detective Sherlock Holmes remarked, in one of his adventures, "It is a capital mistake to theorise in advance of the facts." This is why surveys, of many kinds, can be so important in making critical business decisions.

However, the customer survey is one of the most misunderstood and misused of all tools for gathering business information. The main reason for failing to get useful information from surveys, or - worse yet - getting misleading information, is naively believing that one understands how to do surveys. To quote another philosopher, who actually lived, the German intellectual Goethe said "There is nothing more frightening than ignorance in action."

After all, it seems like such a straightforward thing to do. Just write up a questionnaire, send it out to your customers (or employees or students or voters or whomever), get the answers back and calculate the results, right? Not necessarily.

Explaining just the most common survey blunders would take more space than available here, but we can target a few of the most important ones, to build a perspective on the overall process. Here are four of the most basic blunders committed by would-be survey takers.

Mistake # 1. Doing a Survey

In some cases a survey is one of the least effective ways of learning what you need to know. For example, if you are a service firm doing business regularly with a limited set of customers, say 30 or 40 of them, using a questionnaire is usually a case of overkill. It makes much more sense to go to your clients on an individual basis and explore their perceptions in depth. Sending all of them the same survey every six months will quickly create a case of survey burnout; they get tired of being bombarded with surveys so often.

Surveys are also likely to be ineffective when the measurement model is not clear. If you're not sure what you're trying to find out, it makes little sense to use a survey. The first order of business is to use a skillful discovery process such as focus group research or open-ended interviews to find out what you should be measuring. There is little point in just throwing questions at people in hopes of getting useful information. To paraphrase the old saying about data processing: gibberish in, gibberish out.

Have you ever been asked to fill out a survey while riding on an airline flight? If so, have you noticed how well-behaved the cabin crew tends to be for at least an hour or so before they hand out the questionnaire? They may be bored, slack, dull, and disinterested most of the time, but on the survey flight you'd think they were all fresh from the customer service training camp at Disneyland. Having the flight crew administer the survey is a sure way to guarantee results that are worse than useless. They will be misleading for sure.

Mistake # 2. Confusing Measurement With Research

Opinion measurement and research are two different processes. In most cases a survey is a measurement tool, not a research tool, for one very important reason. It has little or no chance of discovering anything unexpected about the customer's thinking process over and above the topics it asks about.

Very commonly a firm's marketing people will dream up a set of questions, put them into a customer survey, and send it out in hopes of getting some information that might suggest a strategy for competitive advantage. The problem is that their own thinking process contaminates the information coming from the customer, who typically only responds to the specific questions offered.

More than once our firm has been asked to review "customer research" data generated this way that turned out to be virtually worthless. It is often easier to start again in these cases than to try to extract some meager sense of meaning from a misguided survey.

Mistake # 3. Asking Useless Questions

Hotels very commonly use survey cards in guest rooms that are nearly useless for gathering any worthwhile customer information. They typically have very impoverished research models, i.e. the factors or attributes presented to the customer for evaluation are not well developed. They usually include a few simple items like staff courtesy, food quality in the restaurant, and the condition of the guest room. These surveys provide so little information that most hotels don't even use the results.

To compound the problem, customers typically realize they're worthless and unlikely to have any real impact. This probably explains why most hotel surveys get a return rate no higher than one to three per cent. Customers may be thinking about many quality factors other than the standard items on the survey, but they get no chance to respond with what they really want to say.

As a frequent business traveler, I depend on hotels to get messages and faxes to me reliably, accurately, and quickly. I've never seen a factor on a hotel survey dealing with message performance, yet it's more important to me than the food in the restaurant.

Actually, the most important question of all for the hotel survey is one very few of them ever think to ask, which is "Would you be inclined to stay here again if the occasion arose?" Repurchase intention is one of the most valuable and yet most overlooked of all survey variables. Why is it so often overlooked? Because the marketing people are still product oriented in their thinking, not customer oriented.

The purpose of the survey is to understand the customer, not the product.

Mistake # 4. Ineffective Questionnaire Design

People who should know better often do a remarkably shoddy job of putting together questionnaires. When United Airlines was launching its new shuttle service, "Shuttle by United," I happened to be in one of the company's local sales offices. On the counter I saw a customer survey dealing with the shuttle service, which the agents were trying to get customers to fill out.

The first question caught my eye. It said "How frequently do you or the people in your company fly in each of the following markets?" Below the question there appeared about 30 choices of what airline people call "city pairs," i.e. pairs of three-letter airport codes representing various flights you could take. This one item managed to demonstrate almost all of the key blunders one can make in writing questions.

First, it used industry trade jargon - "markets" - rather than customer language, i.e. "flights" or "trips". Very few air travelers would refer to the trip between Los Angeles and San Diego as a "market." Second, very few people except travel agents would be likely to understand all of the city-pair abbreviations used. How many people know that "ORD-LGA" stands for a flight between Chicago's O'Hare Field and New York's LaGuardia airport? Yet these were the descriptors presented to the customers on the survey.

Worse, it's a compound question. The phrase "you or the people in your company" gives the respondent a dilemma. If I fly in one of these "markets" frequently but nobody else in my company does, how shall I answer? If I don't but one other person does, how shall I answer? How about if a lot of us do? When they get the results of this question back, they won't be able to make much sense out of them.

Even worse yet is the fact that the question and the list of city pairs didn't cue the respondent about what kind of answer to provide. Was I supposed to put a check-mark beside the markets I fly in? What about the ones I don't fly in? Leave them blank? How do I answer for the city-pairs I don't understand?

The questionnaire had several other, equally incomprehensible questions like the first one. One was multiple-choice, while another was a free-form narrative. There didn't seem to be any consistent format from one question to another, which meant the results couldn't easily be tabulated or organized statistically. Each question would have to be interpreted in its own way by someone paging through all the surveys, interpreting them, and writing up a report on that item. This makes the results difficult to explain and interpret to managers and others who will read the report.

As mentioned, these are just some of the most common research blunders. These don't even include serious errors like surveying the wrong people, not getting a statistically valid response, and not knowing whether the people who answer the survey are a representative cross-section of the target population.

Other mistakes include putting too many questions on one survey, which actually means trying to do two or more research projects at once, and using 10-point multiple-choice scales for opinion surveys, when respondents can't distinguish that many differences on the scale.

Surveying is rapidly gaining in popularity as a management tool. It's time we brought some skill and discipline to the process, stopped wasting time and money, and stopped confusing ourselves and other people. No one should be allowed to send out surveys to customers, clients, prospects, employees, or anyone else whose views are important to the success of the organization, without having at least some basic training in survey methods. It doesn't take a statistical genius, but it does take some practical knowledge and common sense.

Surveys can be tremendously valuable, but only if they measure something worth measuring.