Rating scales are one of the most frequently used types of questions in online surveys. They measure just about everything: customer opinion, behavior, preferences, emotions, etc. For this reason, rating scales are an important point of discussion for insights professionals. They come in all shapes in sizes, so how do we best use them to capture customer insights?
In the past few years, FocusVision, MaritzCX, and Dynata have been collaborating on experimental research to further our understanding of how to use rating scales in online surveys. One area we’ve looked at is the differences between the horizontal and vertical presentation of scales.
Whether to go horizontal or vertical can seem like an arbitrary decision. A desktop display is large enough to accommodate either horizontal or vertical scales, and the rating scale will display nicely on the screen. But for a mobile device, which is typically held in portrait mode, a lengthy horizontal scale may get cut off or has to be shrunk to a small size to stay within the viewable area of the screen, making the survey difficult to read and even harder to indicate a response.
The survey has to be redesigned for the mobile display, such as using either touch input friendly form or a vertical scale to better accommodate the portrait orientation and smaller screen size.
Which design is best? A vertical or horizontal scale?
In our research to investigate best practices in rating scale design for online surveys, we tested dozens of 5 point scales and 10 point scales. We looked at horizontal and vertical arrangements. We compared desktop, tablet, and phone users, collecting data from thousands of survey participants. Here’s a snippet of what we found:
- For fact-based questions, there is virtually no difference in rating scale data whether you use a horizontal or vertical presentation. Fact-based questions ask for factual information such as: What percentage of your household expense is spent on food and groceries? We did not find any difference among users of the different devices, regardless of whether a 5 or 10 point scale was used.
- For opinion-based questions, horizontal and vertical scales yielded comparable data on 5 point scales. Opinion-based questions ask for opinions, such as: How likely are you to visit this restaurant again?
- For an opinion-based question, mixing horizontal and vertical 10 point scales (e.g. using a 10 point horizontal scale on desktop, and a 10 point vertical scale for the same question on a mobile phone) would occasionally lead to significant data differences. Once we maintained consistency in the scale orientation (vertical scale for both desktop and mobile phone OR horizontal scale for both desktop and mobile phone) the difference largely went away.
- Non-mobile friendly scales (scales that were cut off/too small to read) for either fact-based or opinion-based questions led to a significant increase in participants dropping out of the survey and not seeing it through completion
When deciding between a horizontal or vertical scale, either orientation is fine and will lead to comparable data results. This holds true for 5 point and 10 scales; desktop, tablet and mobile phone users. Mixing PC horizontal and mobile vertical is mostly okay, but it seems preferable to stay consistent, especially if using longer scales. Most important is to provide a user-friendly survey experience for all participants. Questions that are too small to read or require horizontal scrolling will cause survey completion rates to suffer.