The availability of user-friendly survey platforms, such as Survey Monkey and Survey Gizmo, implies that anyone can design and execute a survey to obtain useful information from clients, customers, employees or other target audiences. However, it’s important to distinguish the availability of a data collection platform from the research know-how that is needed to create meaningful survey questions. Over the years, we’ve had conversations with dozens of clients that spent a great deal of staff time and energy creating surveys that yielded very few illuminating findings. And we’ve seen consistent themes for why this happens time and time again. While it is relatively easy to create and distribute surveys using one of these tools, organizations without staff expertise in survey design and analysis should proceed with caution to avoid some of the common pitfalls we have seen over the years.
The purpose of the survey is unclear. A survey can yield invaluable information on respondent attitudes, opinions, and behaviors when done correctly, but it is important to identify clear objectives before developing specific questions. Sometimes you may simply be gathering feedback and in other instances you may create a survey to gather data to make a specific decision. You won’t be able solve all your organization’s problems with one survey. Eliminate questions that would be “nice to know” and prioritize those that focus on the issues that are most critical to your goals or mission. Otherwise, surveys can become too long and unwieldly, which lowers the quality of responses and the response rate.
Survey questions are unclear, confusing or biased. Staff who are inexperienced in survey design may have difficulty writing effective questions because they are unaware of the fundamental rules. In general, questions should be as simple and concrete as possible to avoid confusion and increase respondent accuracy. Survey questions should also include neutral and unbiased language to avoid “leading” a respondent to answer in a specific way. Double-barreled questions, which touch on more than one issue yet allow only one answer, must be avoided as well. These mistakes can be avoided by having a third-party review the survey to identify any questions that may be problematic.
There are too many open-ended questions. Open-ended questions require respondents to write their own responses in a narrative format, while closed-ended questions allow respondents to select a response from a list of options. Novice survey writers often rely too heavily on open-ended survey questions because they believe they will get “better” responses. However, survey takers are more likely to skip open-ended questions because they require more effort to answer. The analysis of open-ended questions is also more time-consuming because each narrative response should be reviewed and coded to identify and summarize recurring themes. Because close-ended questions are typically more efficient, for both the respondent and the data analyst, it is best to use them as much as possible.
Response rates are too low to produce meaningful information. Most respondents will only take the time to complete a survey if they are invested in the organization and want to help them or if they believe they will receive a direct benefit. Organizations can increase response rates through introductory messages that explain the purpose of the survey and how the information will be used. Modest participation incentives can also be an effective way to increase survey participation rates, but these should be used with caution because overly generous incentives can produce a biased sample that may not be representative of your target population.
Well-designed surveys can provide unique and meaningful insights to organizations, but poorly designed surveys may yield results that are not particularly helpful or accurate. If your organization is interested in conducting a survey but lacks the requisite in-house expertise or capacity, Knowledge Advisory Group can assist with any aspect of the survey administration process, including survey design, administration, and analysis.
