I was reminded recently, when my project team switched the response scale for a survey question from an agreement scale to a frequency scale that, too often evaluators focus on developing survey questions while giving less attention to response options. When agreement scales are used as the fail safe response option, the result can be high-quality questions without the most meaningful answers to those questions.

Agreement scales are extremely common in the evaluation world, to the point of overuse. To be fair, evaluators cannot be completely blamed. While well-documented standards for writing survey items and how to design a survey exist, there are far fewer and less established standards for writing response options.

Rather than defaulting to an agreement scale, it is important to match survey response options to the information that is most important to know. Is it that participants agree with something, or is it frequency, importance or some other construct?

There is also a time and place for agreement scales, such as measuring attitudinal constructs. In addition, a Stanford professor’s recent research on construct-specific response options highlights that agreement scales can produce poorer quality data due to acquiescence bias. Respondents agree with statements for a number of reasons, which for most is not because they actually agree with the statement. A survey should make it easy for people to determine how to answer each question directly.

So whether you are in the role of creating surveys or commissioning surveys, before agreeing (or strongly agreeing) to an agreement scale, spend some time considering whether it helps you achieve your goals. Sometimes it will, but oftentimes it is not the best option.