Horizontal vs. Vertical Rating Scales: Does it Make a Data Difference?

A Picture of Aaron Jue

I was recently asked whether or not there is a difference or change in response to a survey question if the answer options are arranged in vertical or horizontal format.

The question pertained to Decipher’s responsive survey design, where an 11 point horizontal scale renders in horizontal format on a PC device, but in vertical format on a smartphone device. This change to vertical format occurs when an 11 point horizontal scale is too wide to properly fit the narrow screen of a smartphone. There is an advantage to arranging it this way. Research has shown that mobile-friendly designs improve participation rates by as much as 50% for smartphone users.

 Figure 1. PC view – 11 point NPS question, horizontal scale
Impact on Data Quality
Figure 2. A responsive design detects 
that an 11 pt horizontal scale  is too 
wide for the narrow screen width of a 
smartphone. So it renders the scale in
vertical format. 
Back to the original question. Does the change in the scale arrangement affect the data? In other words: If you take the same rating scale question, but show it in horizontal alignment to a PC user and vertical alignment to a smartphone user – will that cause a difference in the data? Are the two comparable and can you aggregate data from PC users and smartphone users together?
Evidence says that yes, you can.
In a split cell test, we investigated  5, 7 and 11 point scales. We found no significant difference in responses between PC users rating horizontally arranged ratings scales and smartphone users rating vertically arranged rating scales. We even looked at PC data from a 5 column x 9 row grid and compared that to breaking out the nine row attributes into individual vertical scaled questions on the smartphone. We did not see a significant difference there, either.
The findings support that there shouldn’t be a concern when utilizing a responsive design in your survey.  It has helped improve participation rates for smartphone users, and we have yet to see evidence that it substantively alters respondent answers to our survey questions.
Figure 3. Frequency response distribution  – 11 point scale
  • Share this article:
  • Share this article on Facebook
  • Share this article on Twitter
  • Share this article on LinkedIn
  • Share this article on Google Plus
  • Share this article through E-mail

Add comment

E-mail is already registered on the site. Please use the Login form or enter another.

You entered an incorrect username or password

Sorry, you must be logged in to post a comment.

2 comments

by Newest
by Best by Newest by Oldest
1

Do you have a published research paper with these findings? It would be useful.

2
Alisa Manjarrez

Hi, Larissa! We're talking about working on something for this. We'll email you when a paper comes out!

Check Out the Latest from Our Resource Library

See how FocusVision has helped some of the worlds' biggest brands in our quantitative and qualitative resources including case studies, white papers, and webinars.

Webinar

Understanding Social Media Video Sharing Behaviors and Motivations

View Resource
White Paper

More than Just a Lighthearted Activity: Understanding Social Media Video Sharing Behaviors and Motivations

View Resource
Case Study

Double Take: Connecting the Dots Between Millennial and Seasoned IT Decision Makers

View Resource
×