How to Avoid Response Bias in Surveys

What is Response Bias?

Response bias is the tendency for participants to respond inaccurately to questions. This occurs often when people are asked to self-report, for example in a survey, and is often an unconscious behavior.

Why This Matters

Surveys can be an invaluable method to get information from users or customers. In fact, a lot of software companies leverage user testing before building their app. However, many companies unintentionally use biased surveys to inform their business decisions, which can have an effect on the validity of the research and lead companies astray. It is encouraged for any company to acknowledge response bias in surveys and take measures to inhibit it. In this post, four types of response bias will be discussed along with some tips on how to structure surveys to avoid them.

1. Acquiescence Bias

Acquiescence bias occurs when the respondent tends to agree with all of the questions in the survey, also known as “yea-saying”. There are many reasons hypothesized as to why people do this. Some people believe that it is due to the respondents’ desire to please researchers, and others think it’s caused by the respondents’ attempt to relate their experience to the question. Regardless of why this occurs, just know this bias occurs often, and there are ways to avoid it.

Pro-Tip

Avoid asking questions that require the respondent to agree. This can also be known as a leading question. Instead, give them the option to agree or disagree. Take note of the following two examples. Which one avoids bias?

Example A

Example B

Example B is the answer. Example B is considered best practice, as it gives the respondent the option to either agree or disagree rather than leading them to agree.

2. Demand Characteristics Bias

Demand characteristics bias is where a respondent changes their behavior because they are participating in a survey. There are many ways this can occur, but here are a few examples:

• Participants try to figure out the purpose of the survey and change their behavior to align with that purpose.
• Through email surveys, if the respondent receives a friendly greeting, their responses can be influenced.
• The respondent having knowledge of the survey content before taking the survey may cause them to research before taking the survey.
• Anonymous vs. non-anonymous surveys may yield a different answer from the respondent.

Pro-Tip

This is a tough one to avoid, however, you can soften the bias by using a neutral tone in an email survey greeting, avoid giving your participants prior knowledge regarding the purpose or content of the survey, and consider how anonymity would affect the respondents' answers. Additionally, you can try to make the survey fun or engaging for your participants so they do not overthink it.

3. Social Desirability Bias

Social desirability bias is the tendency for the respondent to choose the more socially acceptable response rather than answering truthfully. People tend to want to present their best selves, even if they do not know or have any face to face contact with the researcher.

Pro-Tip

Consider anonymity. If you are going to use Example A, make your survey anonymous to receive more truthful responses. Otherwise, Example B is the best way to increase truthful responding.

Example A

Example B

4. Extreme Responding Bias

Extreme responding bias occurs when participants are likely to choose the most extreme responses. This often occurs when the respondents are given a Likert scale. A Likert scale is a question that uses either five or seven points. This bias also commonly occurs when the respondent has a strong opinion or an emotional reaction to the question.

Pro-Tip

Avoid asking sensitive questions using emotionally charged words or loaded language. Choose your words carefully and consider how they could make the respondent feel.

Example A

Example B

In these two examples, the only difference is one word. In Example A the word eliminated is used whereas in Example B removed is used. The word eliminated has a much more negative connotation than removed and may cause a more extreme response from a participant.

Voila. Four common types of response bias to watch out for when designing surveys. This was by no means an exhaustive list, but feel free to leave a comment if anything additional would be helpful.