Writing Survey Questions: More Then Meets the Eye...





I recently reviewed three online surveys - one from a government office, another from a radio station, and a third from a university. Each was a well-intentioned effort to learn more about their respective audiences' opinions. Unfortunately, each was also critically flawed by the same fundamental problem: Question construction errors. Before a single response was received, the usefulness of each survey was in doubt.

What's most disconcerting is the likelihood that decisions will be made, and actions taken, based on poor information. Each organization would have been better served by avoiding the rush to push their survey out, slowing down, and taking a step back to carefully consider how they were asking the questions.

Let's look at an example of how a question can be revised to provide better information.

Example 1

Do you think our local news coverage has any "slant" or "bias?"

• Very liberal
• Somewhat liberal
• Fair and balanced
• Somewhat conservative
• Very conservative

Question analysis

The question seems straight-forward enough, and the response categories appear balanced. Yet confusion exists between the question and the response categories. The question prepares the respondent that news coverage "bias" is going to be measured, but the response categories introduce the terms liberal, fair, balanced and conservative.

Furthermore, an assumption is made that the question and response categories are referring to degree of bias (liberal or conservative) in a political sense.

The discontinuity between the question and response categories tax the respondent, increasing the likelihood of inaccurate measurement as well as survey abandonment.

In addition, and to my knowledge, the audience to which the survey invitation was sent consists largely of financial contributors. In which response category do you think most contributors would place themselves? Fair and balanced, perhaps?

Question revision

A better approach is to ask two questions, using question skipping logic, to obtain better information:

Do you think our local news coverage is politically biased? (Select one)

• Yes
• No

Skipping logic built into the survey would then present a follow-up question to the respondents selecting "Yes" to the bias question.


To what degree do you feel our local news coverage is politically liberal or conservative? (Select one)

• Extremely liberal
• Very liberal
• Somewhat liberal
• Somewhat conservative
• Very conservative
• Extremely conservative

In this case, a six-point scale is used because there's no need for a neutral response category. By the very nature of the first question, we know that a respondent answering "Yes" is going to pick a side (liberal or conservative) in the follow-up question. The follow-up question is simply measuring the degree of liberal or conservative.


Example 2

What are your favorite sources of local news?







Question analysis

There are at least four problems with this question and its associated response categories.

First, it's implied, but never stated, that the respondent is supposed to select a response for each news source presented. Inadequate instructions may result in the unintentional skipping of response categories, thereby reducing the amount of useable data.

Second, the scale items are incongruent. Often, Sometimes and Never are measures of frequency, but the item "One of my favorites" is not. Clearly, this poses a problem when interpreting the results.

Third, it's implied that the respondent will be able to correctly select the local NBC, ABC and CBS news programs without the aid of the much promoted local affiliate call signs.

Fourth, the "Other" response category is fine in itself, but the lack of a way for the respondent to then identify the news source is unfortunate.


Question revision

The revised question frames the frequency items to provide more useful data and provides instructions to increase complete responses. In addition, call signs were added to the response categories to help the respondent correctly identify local news on network affiliates and, finally, an option is presented to allow a respondent to identify a local news source used but not included on the list.


In an average week, how often do you use each of these sources for local news? (Select a response for each news source)







If you selected an "Other" source of local news, please provide the name of the source here:




These are just two examples demonstrating the importance of properly constructed questions and response categories. The key to gathering good survey intelligence begins with taking time to compose the best question list possible. I encourage you to invite a survey research specialist to help review and revised your question list the next time you begin a survey project. A small investment in professional assistance will pay huge dividends in the value of the intelligence you gain.

Thanks for reading...

Todd Chavez
Chavez Survey Research, Inc.
www.chavezsurveyresearch.com
(208) 429-8945

Surveying: Do it yourself or use a professional?...

When it comes to knowing what you need to know about your customers, employees or your membership, survey research is the most efficient way shed new light on the landscape in which you work. How well you perform survey research will determine just how brightly the light will shine.

Today, the Internet provides an exciting alternative in conducting survey research in your hands. New online tools are readily available and make surveying a snap! What could be better?

Actually, a lot could be better. An online survey tool is like any other tool - In the wrong hands, even with the best of intentions, it can damage. Why is this the case? Performed well, survey research is a skill that delivers accurate and reliable information. Performed poorly, it delivers inaccurate and unreliable information. When the survey report hits your desk, how do you, as the decision maker, know what you're getting? Making decisions based on good information leads to success.

The best way to avoid the "do it yourself surveying" pitfall is to engage research professionals to guide you through the surveying process - The results will be worth the effort. And the cost? Probably not as much as you think, and certainly not as much as the cost of making decisions based on bad information.

So when can you safely perform your own surveys?

Ask yourself these questions:

1. Am I getting a quick "pulse" about a single topic by asking just a few questions?
2. Is my survey target audience small? (A population of 50 or less)
3. Is my audience likely to respond to a survey coming from me?
4. Am I using the survey results simply to verify a decision I'm making based on reliable information I have from other sources?
5. Am I comfortable interpreting the results? (Typically frequency tables and free text comments)

If you can answer, "Yes" to each of these questions, you're probably OK to proceed without help. If your answer is "No" to any of the questions, you should consider taking advantage of a professional's experience.

A professional's experience is very beneficial when your survey project is characterized by any one of these considerations:

• You seek information about more than one topic
• You will ask more than, about, five questions
• Your survey population is larger than 50 people
• Your survey population is often thought of as being segmented by geography, demographics, or organizational structure
  
• You need to ask different questions to subsets of the population based on how they respond to other questions
• You need to be absolutely sure that the questions and their response categories are worded and sequenced perfectly
• Your using the survey results as a primary source of information on which to base key decisions
• You need to be able to compare responses from subsets of the population with one another
• You need to identify the results that are the most important—the statistically significant "gems" of information
• You need to identify trends over time
• You need to be able to interpret free text comments in a defendable, unbiased manner

The bottom line?

In a few instances, performing your own survey research is going to be fine. Yet I think you'll find the quality and credibility you gain from using a professional will deliver much more.

As a parting note, there is one item for which I always recommend using a survey research professional - Questionnaire Composition and Design! The process of creating a series of perfect questions, whether it's five or 50, is far more complicated than generally thought. Question wording and the assigning of appropriate response categories is a science in itself. Placing them in the right order and creating a layout that aids in navigating the survey is equally as challenging. S adly, I review too many surveys that are dysfunctional from the very beginning due to poorly worded questions.

Thanks for reading...

Best regards,

Todd Chavez
Chavez Survey Research, Inc.
www.chavezsurveyresearch.com
(208) 429-8945