Almost nothing is more frustrating for researchers than observing inconsistent responses after fielding a survey. Unfortunately, inconsistent survey responses from participants is a common problem in survey research. This is particularly true when a survey questionnaire includes multiple questions that ask about very similar things. Inconsistent responses to survey questions lead to many problems including poor reliability, reduced statistical power and bad fit of models.

So, how can we identify the different primary types of inconsistencies we observe in surveys? And how do we fight back against inconsistency survey responses?

There are three primary types of inconsistency we see in survey research – negation, polar opposite, and reversal.  

  • Negation: Inclusion of a negative in a survey question can result in inconsistency responses. For example, using a negative term in a response option like “not talkative” is more likely to result in inconsistent responses than using positive terms like “quiet”.
  • Polar Opposites: Respondents may respond to a survey questions with responses that are the opposite of a response they provided to a different survey question, resulting in inconsistency. For example, a respondent may say they are talkative in response to one survey question and then say they are quiet in response to a different question.
  • Reversal: Inconsistency can occur when a respondent agrees (or disagrees) with survey questions or statements that are very different. Using the example of extroversion versus introversion, respondents may be asked to express their agreement or disagreement with the statements “I see myself as someone who is talkative” and “I see myself as someone who tends to be quiet”. Inconsistency results when a respondent agrees (or disagrees) with both items.

Respondents tend to answer survey questions inconsistently for one of two reasons. First, some respondents lack the motivation to process survey questions in specific detail. They do not take the time they need to fully read and respond thoughtfully to survey questions. In other cases, respondents lack the ability to fully comprehend and respond to complex survey questions.

Researchers at Pennsylvania State University have established that eye tracking is an effective way of evaluating a survey questionnaire to avoid inconsistent responses. Eye tracking can tell if respondents are likely providing inconsistent responses as well as the type of inconsistency. Eye tracking can be included in the pilot testing of a survey questionnaire. This type of testing is particularly important in cases where a survey will be very expensive to field or where results will be used for extensive statistical analysis.