No one has an infinite amount of time or patience. So we might reasonably expect respondents to tire and quit a survey that takes too long to finish. But how long is too long? I’ve been running online surveys for more than a decade. One rule of thumb I hear a lot in the research industry and among my colleagues says that the shorter the survey, the better. But definitely no longer than 15 minutes. I’ve had success adhering to this principle, but I do regularly see other industry surveys stretch longer than 35 minutes.
What does the experimental evidence say?
Research has shown that survey respondents do indeed get fatigued. Respondents become less thoughtful about their answers and give less attention to a survey the longer it goes.
A wonderful paper by Pete Cape looked at how data quality differed for a block of questions when they were placed at the beginning of the survey versus when they appeared at the end. Signs of ‘satisficing’ were much more pronounced when the questions appeared at the later stages of the survey. ‘Satisficing’ means to just do enough to get through the task at hand. Rather than providing thoughtful answers, respondents did the minimum required to complete the survey. This includes:
- Not answering optional question
- Fewer characters typed in for open-ended questions
- Spending less time reading and answering questions
- More evidence of ‘cheating’ (answering in a way that allowed them to skip question sections)
Such fatigue effects start to creep in after the first few minutes of taking the survey. They become truly pronounced after the 15-minute mark or so.
Survey length is generally not a determining factor for survey participation. Just because the survey is long, does not mean this will lead to increased dropouts. Once a person decides to take your survey, they tend to soldier on no matter how short or long the survey is.
Figure 1 illustrates a typical example showing the page by page dropout rate of a 15-page survey. Survey abandonment almost always occurs within the first handful of pages within a survey. The majority of dropouts occur on the first page. This is a respondent’s first impression of the survey as they discover what it is all about and make some decisions about whether to continue. Many do not. The second highest number of abandons occur on the next page. Here, respondents are still in the evaluation stages, considering whether they are invested enough in the survey. By the second half of the survey, very few dropouts occur. By this stage, the respondent is engaged in the survey and has decided to see it through completion. If there is an increase in dropouts at this point, we can usually point to other factors – like a complex grid question or very wordy question that places an extra burden on the respondent.
The implication here is that the first few questions set the tone for the rest of your survey. Since respondents are most vulnerable to dropout at the beginning of a survey, it’s important to pay particular care to engage respondents here by avoiding any survey dropout triggers. Questions that require extra reading or thinking should be saved for later. Instead, make early survey questions easy to read, fill out and visually appealing.
How long can it go? Shorter surveys are better than longer surveys, and they should not be longer than 15 minutes. Fatigue effects and poor data quality – not survey participation — is the main concern when surveys become long. People seem to have it in them to complete something they start. But if it goes on too long respondents focus on ‘finishing’ the task rather than providing thoughtful answers to your questions.