Congratulations on designing a great questionnaire that is sure to generate insightful and actionable research results. But before going pressing that scary launch button, it’s imperative that you ensure the questionnaire is the highest quality possible. Here are ten tips to get you there.
Find every typo
You’re almost guaranteed to find at least one typo in your research questionnaire even if you’ve reviewed it carefully ten times. It’s nearly impossible to catch your own typos so make sure a colleague who’s not been part of the questionnaire design process reviews the questions and answers. Preferably someone who has great attention to detail!
Confirm the length
It’s easy to count the number of pages in your draft questionnaire and decide that ten pages equals ten minutes. This may work reasonably well in many cases, but if many of the questions are complicated, the length of interview could be much longer than anticipated. Since your colleagues are helping you find typos, ask them to do a more thorough pre-test and note of the time it takes them to complete the questionnaire. One of the most common complaints research participants have is finding out a survey is far longer than they were told. Let’s try to be more accurate with length of survey estimates.
Simplify the language
Market researchers are so comfortable with acronyms, marketing jargon, and marketing language that we don’t notice it when we’re writing it. Try reading your research questionnaire out loud so that you can hear it for yourself – are these words you would use with your friends? In addition, run your questionnaire through a grammar checker like Grammarly, as well as a Readability Calculator. These tools will help you identify confusing and overly-complicated language that needs to be fixed. The result will be a questionnaire that can be understood by more people, regardless of reading skill and regardless of whether English is their first or fourth language.
Opt-out
There’s nothing more annoying for questionnaire participants than not being able to select an answer that reflects their opinion. As such, check that every single question has an “opt-out” option. It could be something like “Other,” “Don’t know,” “None of the above,” “More than 100,” “Don’t recall,” “Never,” or “Not applicable.”
Consistent scales
When several people have contributed to a questionnaire, it’s easy to include a variety of different scales – everyone has their favourite. Take the time to review every scale to make they are as similar to each other as possible. Convert 7 or 10 point scales back to 5 point scales so that all the questions match each other, and so that they’re easier to answer. And, make sure the language matches, e.g., use “Agree Strongly” throughout rather than switching between “Agree Strongly” and “Strongly Agree.”
Randomize right
By nature, people are biased. When they read long lists, they are more likely to choose answers at the beginning and at the end of the list. This could be detrimental if the “correct” answer is in the middle. Thus, ensure answer options are randomized appropriately. When people know the names of things (e.g., brands, stores, celebrities) or when there is a natural order (e.g., days of the week, volume), don’t randomize. But when there is no inherent order, make sure the answers are randomized (e.g., product categories, concepts, opinion statements, colours, shapes. And more importantly, make sure that the scales, and the “Don’t know” and “Other” options aren’t accidentally included in the randomization process.
Confirm the logic
The longer a questionnaire is, the trickier the logic can get. Try writing out each possible scenario and then answering the questionnaire as if you’re that person. Ask several colleagues to do the same. Take note of whether people are being asked questions that don’t apply to their situation. Review the dataset to determine whether each question is generating as many datapoints as you would expect. Questions with very few or very many answer options might be the result of skip logic errors.
Demographic questions
We’re so used to creating targeted sampling strategies using demographic and psychographic questions that we sometimes to forget to include those questions within the questionnaire. As a minimum, make sure that questions for age, gender, and region are included. Then, as space on the questionnaire permits, incorporate additional questions such as ethnicity, household size, education, income, religion, and more. It’s okay to start with one or two demographic questions but do your best to place as many as possible at the end of the questionnaire.
Preview it on a mobile device
Most market researchers write out their questionnaires on full size computers, perhaps even with extra-large monitors. When we test questionnaires, we get an unrealistic and overly attractive preview of what the questionnaire will look like in field. However, upwards of 50% of questionnaires are now answered on mobile devices like tablets and smartphones, many of which have very small screens. Rather than trusting your monitor, trust your smartphone and test the questionnaire there. Look for questions that scroll down too far, or require scrolling from left to right and make those corrections.
If you’re looking for a more in-depth and theoretical discussion of how to pretest a questionnaire, AAPOR’s Public Opinion Quarterly has an excellent article called Methods for Testing and Evaluating Survey Questions.
And if you’re ready to program your questionnaire to generate data, we’d love to hear from you!
You might like to read these:
