Survey open ends often uncover insight gold, be it through a nugget that beautifully sums up how people are feeling or delivering an unexpected view into why participants think the way they do. However, these questions are not without their challenges. Open-ends are also a notable cause of drop-out as the format requires a higher cognitive effort than their closed-end counterparts.
So how do we optimize open-end questions to obtain rich data while minimizing participant fatigue? Here are five tips to consider when crafting open-ended questions:
- Broad, but not too broad
Open-end questions are just that – open-ended. They offer participants the ability to express an answer in their words rather than be constrained by pre-determined responses.
Developing a question that allows them to express themselves is a balancing act. If the question is too vague then participants may not have enough frame of reference to answer it, and in turn, it may be perceived as too onerous a task resulting in survey drop off. For example, a few years back I was experimenting with video open-ends and posed the question ‘what does good customer experience mean to you?’ In hindsight it is obvious this wouldn’t work: most people don’t sit around thinking about what a good experience means to them.
On the flip side, if the question is too narrow, you’ll generally receive just a few words in response. For example, ‘what did you like about the ad’ will likely return short responses such as ‘it was funny’ without the rich detail of what made it funny.
- Ensuring the question intent is universally understood
When crafting any survey question, you need to consider whether the wording is going to be understood by all participants in the same way. The trickier part with open-end questions arises when navigating the need for broad, but not too broad and creating a question that everyone understands the intent. Even the simplest wording may prove to be problematic. For example, in our recent ‘How You Research’ survey, a response to the open-end question ‘what would make your life easier’, chastised us for seemingly searching for the easy way out, the next quick-fix. However, the intent when crafting the question was for a catch-all way to gather the top of mind challenges and potential solutions. Does this one response mean that the question wording is fundamentally flawed? Not necessarily, but it does bring me nicely to the next tip: pretest the question as widely as possible.
- Don’t skip pretesting
Pre-testing removes the guesswork and quickly helps the question evolve into the most useful version of itself. In an ideal world, you’d conduct cognitive interviews to test all the survey questions. While that is rarely possible, there are several other ways that pre-testing can take place at different stages of the question construction.
In the early stages, it is helpful to recruit colleagues who aren’t directly involved in the study to obtain initial feedback. As the question is refined, turn to friends and family who aren’t in the research industry determine how close – or off – the mark the question wording is compared to your desired response. Finally, if time and budget allow, conduct a soft launch, and evaluate real-time responses to the question. This confirms you are on track or helps iron out any issues that haven’t been caught.
This is what I did for the video open-end question, discussed in tip 1, asking about the ideal customer experience. I quickly saw that the question had a high drop-off rate and, from the responses that were recorded, I understood that it was too broad a question to get meaningful responses. So, I paused field and reworked the wording to ask participants to describe a memorable occasion where they had an excellent customer experience.
- Placement is crucial
Do not underestimate the importance of placement. When doing a drop-off analysis of a questionnaire, it is extremely common to see higher drop-out rates when the open-end is placed towards the beginning of the survey, and fewer drop-offs when positioned towards the end.
Why does this happen? Most dropouts occur early in a survey as people decide whether or not to invest the time to complete the full questionnaire. As open-ends are more demanding, it can be the deciding factor to abandon to survey. Conversely, as people move through the questionnaire, they are more committed to seeing it through, and they are generally more engaged in the topic. Thus, they are willing to deal with the additional burden. However, there are limits to participants’ patience, which brings us to the final tip – the number of open-ends.
- Limit open-end questions
As a rule of thumb, try to keep to a maximum of two open-end questions per survey. There are obviously many exceptions to this rule, particularly if people are responding to a topic that is of keen interest to them, the questions are ones that they can readily answer and/or the financial incentive for their response sufficiently meets the amount of effort required.
Crafting successful open-end questions is not a simple task. It requires taking the time to ensure the question is readily understood by all, is open enough to capture detail while tight enough to feel manageable, is thoughtfully placed, and participants aren’t overburdened by the number of open-ends within the questionnaire. This is very much a mix of art and science and it is important to pretest the questions to ensure insight gold at the end of fieldwork.