Discover the essential steps for designing a respondent-friendly survey questionnaire that's easy to analyze. Learn best practices, tips, and tricks that will empower you to create an engaging survey experience that yields a quality data set you can draw valuable insights from.
Introduction
In this article, we will guide you through the necessary steps to ensure your survey gathers reliable data, setting you up for a smooth analysis. In addition, you will learn how to effectively engage your respondents, prevent conflicting outcomes, and structure your questionnaire to generate easy-to-understand outputs that seamlessly integrate with your preferred analysis methods.
When creating a survey questionnaire, it's crucial to consider the data analysis process from the start. Understanding how the data will ultimately be presented informs the survey design process.
Once you understand the fundamentals of your reporting options, it's equally as important to know how the format of your questions and the response choices you provide (or don't provide) will impact your respondent engagement, the quality of your final data set, as well as the overall ease of analysis.
This article will show you how to design a quantitative survey questionnaire that lends itself to an efficient and accurate analysis.
Structured questions, structured results
Quantitative research focuses on gathering and evaluating survey questions with closed-ended answers.
Closed-ended question types are efficient for data collection. These questions present limited response options, allowing participants to choose from predetermined answers, yielding structured results.
Quantitative research is particularly valuable when seeking to derive measurable comparisons and conclusions, as it focuses on closed-ended responses that streamline the analysis process and provide well-defined outcomes.
Reporting options
Now let's talk about how you can review your results. The primary options are listed below.
- Exports: Excel, CSV, and SPSS exports allow you to easily import data into third-party software for further analysis.
- Web reports: Simplify large data sets into visual overviews to allow you to quickly identify and compare high-level data points. These visuals are often most useful when looking for a quick takeaway.
Filtering your results
In modern survey software, you can filter any web report by a variety of attributes. The most common are listed below.
- Responses to questions: Filter by a single answer choice or a combination of several answer choices.
- Collectors: Easily segment different respondent sources using different collector links.
- Start time / end time: Compare different timing cohorts of responses.
- Duration: This is reported in seconds.
- Variables: This includes any third-party data you may append to your collector link.
Want to really dig into reporting options and best practices? Check out our article Data Analysis Demystified — 5 Key Reporting Methods.
Statistical significance
To bolster confidence in your findings, it's crucial that you obtain a sample size that yields statistical significance. Try our margin of error calculator, where you can enter project specific details to understand the optimal number of responses needed to achieve a low margin of error.
Survey design best practices
Now that we understand the types of reports we can create and the options to filter those reports, we're prepared to dig into the main course. Let's explore useful tips, tricks, and best practices that will make designing and analyzing your next survey a breeze!
Respondent engagement
When designing a survey questionnaire, it's essential to consider the respondent's point of view. Individuals who answer your survey spend valuable time providing the researcher with opinions and feedback.
Respondents generally elect to participate in surveys in their free time. For example, they might be on the train during their commute, taking a quick survey on a lunch break, or even relaxing with a glass of wine after a long day.
Survey respondents that are engaged with the survey content and clearly understand the researcher's expectations consistently provide the highest quality data. Therefore, one of your primary goals as a researcher is to make that experience as simple, straightforward, and engaging as possible.
Be direct and concise
Shorter, concise surveys help prevent respondents from becoming fatigued or losing interest, resulting in lower engagement and response quality.
When utilizing online survey panels for data collection, we've found 25-45 questions / 5-9 minutes is often an optimal length that balances respondent engagement while still allowing for a significant level of data collection.
Centiment supports longer surveys, but as a general rule of thumb, we recommend keeping your survey below 75 questions / 12 minutes to preserve respondent engagement.
Use plain language
Be direct and use concise language every person in your target audience will understand. Keep your respondents on track and avoid confusion to maximize data quality. Avoid lengthy question text or response choice options when possible. This is especially true for matrix-style questions, as the amount of information presented can quickly become overwhelming with this format.
It's also best to assume that not all individuals will be familiar with abbreviations or acronyms. Therefore, we recommend spelling out acronyms and avoiding abbreviations.
Closed-ended questions are more-efficient to analyze
Centiment's powerful reporting tools include exports and web reports specifically designed to allow you to draw actionable insights from large data sets quickly. That said, these tools become less effective when too much variability is found in the results.
For example, if you'd like to ask a group of senior software decision-makers, "Which factors are most important when making a purchasing decision for an enterprise accounting software solution?" The most common question format would be a multi-select or an open-text. So let's walk through the pros and cons of each format.
Option 1: Open text question format
The open text format requires the respondent to type out a response. This takes more thought and time to complete, which can become a taxing experience if repeated several times in your survey.
Typical outputs, such as the word cloud shown above, can be cumbersome to interpret and draw detailed reporting from.
Often, similar responses are not aggregated. This can result from terms with the same effective meaning, such as "cost" and "price" or "management" and "mgmt" being reported independently. The end result is that tedious post-processing of the terms is often required to place them into consolidated categories.
Option 2: Multi select question format
The multi select format provides a predetermined list of response options to choose from. In doing so, respondents experience a less taxing experience, and researchers are left with more structured outputs.
A closed-ended question format also leads to a reporting output that makes drawing specific comparative conclusions amongst a dataset efficient. E.g., "70% of software decision-makers consider ROI a key decision-making factor whereas just 29% stated service to be a factor." More advanced analyses, such as a Top 2 Box score, also utilize a closed-ended format.
A potential downside to closed-ended single or multi-select question types is that they can box respondents into a series of choices that may not be applicable. Fortunately, many survey platforms, including Centiment, allows for an "Other" key-in option. This option enables respondents to key in an open-text response if they are not satisfied with the provided response choices.
Survey design tips & tricks
Topic intros and transitions
Does the respondent know why they are taking the survey? Often a concise introduction is appropriate to set expectations for the type of interaction the respondent should expect to follow.
Pertinent details may include why they are being asked to take the survey and how long it's expected to take. Knowing the purpose behind the research can help drive engagement. That said, this information must often stay concealed to avoid any bias in the response data.
When it comes to switching topics, make it a smooth ride for the respondent. Just as you would in a conversation, provide concise transitional language to avoid a jarring experience. E.g., "Thank you for your feedback on which features matter most; now let's discuss how you collaborate when using this software." This kind of brief language helps the respondent mentally transition to the next research topic.
Include visual appeal
Using visual aids can be very beneficial when explaining a concept and can also make the study more engaging by dividing the content. For instance, showing an image of a physical product or a diagram can help the respondent understand something more easily than just text. After all, a picture is worth a thousand words.
Videos can also act as an efficient method of knowledge transfer. That said, it's best to limit your videos to just a couple of minutes. If you have a longer video, consider breaking it into multiple shorter videos.
When presenting a video, employ a timer features that holds the respondent on that page for at least 50% of the duration of the video. This ensures it's not accidentally skipped over.
Dial in your look and feel
Personalize your survey to fit your brand and your audience. Customize the font, adjust the color palate, upload a background image, or present your company logo. Just remember not to draw too much focus away from the survey content.
Demographic questions
To respect your respondents' time, if you'd like to include quotas or any directional balancing in your survey, introduce any associated questions early in the question order. Doing so will allow you to quickly disqualify any respondents whose quota groups have been filled.
You'll want enough demographic questions to segment your data by pertinent categories, but not so many that the demographic section becomes cumbersome to complete. Three to five questions is a typical range in a single survey, but this amount can vary depending on your research and segmentation goals.
If you are interviewing your own contacts on a repeat basis, be sure to store their profile data in your directory or contact management database if your survey software contains such a feature. This will prevent you from having to ask the same profile questions repeatedly.
Utilize a variety of question formats
Although having some consistency in question formats can be helpful, it is recommended to avoid overwhelming respondents with continuous matrix batteries. Even though it is important to keep this in mind, some researchers tend to neglect the human experience.
In other words, keep it interesting, mix up the formats to some degree, and, in general, treat your respondent as you'd like to be treated. They'll reward you with better feedback.
Avoid leading questions
Questions that prompt or encourage a specific answer can skew respondents' responses and negatively impact your data set.
A leading question such as, "Is product A the best one on the market?" is likely to provide a skewed view.
A non-leading question such as: "Which of the following products do you believe is the best on the market?" is much more likely to provide an accurate representation.
Randomize long lists
Individuals tend to select from among the first response choices that apply to them. As such, questions with eight or more response choices often benefit from selection choice randomization. Just remember to pin or anchor certain question types in place, such as none of the above.
Questions with 15+ response choices may be best suited to a dropdown question format. E.g., "What state do you live in?"
Avoid conflicting data inputs
Conflicting or nonsensical inputs can create noise in your data set, raising challenges when you get to your analysis. To avoid this, only introduce questions that are pertinent to each respondent.
Use skip and display logic to skip past or hide questions. For example, if a respondent indicates they are not a software purchasing decision-maker at their organization, suppress downstream questions on this topic.
Tailoring your question paths up front will keep your questions relevant and your data set clean.
Avoid overlaps in your ranges
When providing response choices for a respondent's age, for example, make sure your ranges are consecutive as opposed to overlapping. E.g., 20-29, 30-39, as opposed to 20-30, 30-40. In this example, 30 is being double counted.
Validation/guard-rails
Prevent outliers or nonsensical responses by using validation for open-text questions. Common validation options include US zip codes, text-only (non-numeric), or restricted numeric responses to define a minimum, maximum, or range.
Many survey tools, such as Centiment, also offer an "exclusive" feature. Using the "none of the above" example, if this option is selected on a multi select question, any other previously selected options will be unselected.
In terms of presenting scales and ranges in a closed-ended format, ensure you provide logical response ranges for questions involving numbers. A "not applicable" option can also be used to avoid forcing a selection when a respondent does not have an answer.
Test your survey several times
The "preview" feature in Centiment lets you test your survey without storing responses. Share your preview link with friends or colleagues to get fresh eyes on your study. Test all survey paths, including all skip and display logic outcomes.
If collecting data from Centiment's Audience Panel, we'll start off with a soft launch. This is an early stopping point after 10-20 responses are collected. It provides the researcher a chance to review their initial results to ensure their survey build is acting as expected before proceeding with data collection.
Conclusion
To sum it up, designing an effective survey involves focusing on your analysis from the outset and crafting questions that foster respondent engagement. By comprehending reporting options and structuring your survey to produce clear results, you can create an engaging survey experience that reveals valuable insights.
Be sure to test your survey rigorously and follow the practical tips and tricks discussed in this article to ensure accurate and efficient analysis. Equipped with this knowledge, you will be well-prepared to embark on your research journey.
Get started now by signing up for the Centiment Survey Tool for free or by pricing targeted responses for your next study from Audience Panel.
