Since the early days of the internet, web design conventions have periodically evolved. Designs are pushed by changes in technology, evolving taste and online habits. Take Yahoo’s front page as an example (Figure 1). An internet pioneer, Yahoo’s early web designs were largely text focused, and all almost all of the content fit within a single viewable area of the screen. By 2009, images and more sophisticated graphical and stylistic elements appear.
Yahoo’s website today reveals a couple of current design conventions. Websites now show increasing use of image and multimedia content, and as the mobile web has evolved, it is now commonplace to scroll for content rather than abide by the ‘above the fold’ rules that governed large screen web design for many years. For market researchers, this constant evolution of websites means it’s important to continually evaluate how online surveys are designed.
Figure 1. Yahoo’s front page through the years. Designs from the early days were text-driven and content appears condensed within a single viewable area of the computer screen. By 2009, web page designs now include substantial content ‘below the fold’ requiring vertical scrolling.
Typically, online surveys are designed to include just one question per page. This enables logic checks at every question, ensuring that each person is properly directed through the survey and only receive questions that apply to them. A one question per page design puts all content within a single viewable area of the screen. Instead of scrolling down to see more content (i.e. the next question), the survey participant selects ‘continue’ after each question to go to the next page (Figure 2). Given today’s web design conventions, would a survey design with multiple questions on a page (scrolling design) improve the participant survey experience and completion rates? Besides reducing the burden of the extra mouse clicks, participants can visually see and anticipate the next few questions.
Figure 2. A survey design with one question per page (left); and multiple questions on one page (right). The one question per page requires the participant to select ‘continue’ after each page. The multiple questions on one-page format requires vertical scrolling.
In a recent study, Maritz CX, Dynata, and FocusVision, collected survey data from more than 2500 desktop, tablet, and mobile users to compare both the single question and multiple questions per page formats.
In terms of participant survey satisfaction, neither design won out. Both the single question per page design and the multi-question per page design showed similar levels of survey satisfaction. Completion rates were also very similar for both versions (Figure 3).
Figure 3. Both designs yielded similar survey satisfaction scores and completion rates. Source: Mobilize Me!! – Mobile Survey Design Enhancements – research presented at Insights Association NEXT, June 2019. A webinar of the presentation can be viewed here.
Does either paging design have an impact on the quality of the survey data? We looked at the percentage of participants who straight-lined at the grids questions (Figure 4). We did not find any meaningful difference between the two designs (Figure 5).
Figure 4. In straight-lining, the respondent selects the same scale response for each of the items in the grid. This is one measure of data quality. As researchers, we hope participants vary their scale usage as different items are being rated. Generally, the less straight-lining the better.
Figure 5. We averaged the % of straight-lining across the grid-formatted questions in our survey. No meaningful difference was detected. Source: Mobilize Me!! – Mobile Survey Design Enhancements – research presented at Insights Association NEXT, June 2019. A webinar of the presentation can be viewed here.
At this time, our test did not show any compelling reason in favor of one design or the other. The single question per page design may still prove useful given the convenience to the programmer / survey designer to add a page break after every question. Then decisions don’t have to be made on where to put the page break nor is there worry about the execution of the survey logic. However, as web design conventions and online habits evolve, ongoing evaluations in web survey design, (e.g. repeating the current test and varying the survey length) remain an important area of investigation.