Response bias, and why it's your survey's biggest enemy

Response bias can adversely affect your survey outcome. Here's what you can do to fight it.

What makes a survey a powerful tool is its ability to obtain real information and data about the target audience, rather than relying on trial and error methods. However, in order for this data to be useful, it must be both error-free and, most importantly, bias-free.

But what exactly is response bias?

A response bias, or responder bias, occurs when the participant intentionally or unintentionally diverges from truthful responses to the survey questions. This results in inconsistent data, or worse, incorrect analysis of the collected data.

Without an interviewer, response bias has become even trickier to avoid online. Because the responder's answers are unmonitored, it becomes quite a task to try to gauge his/her interpretation of the questions.

It should thus be your top priority to lessen the potential for response bias in your surveys before you release them.

How can you eliminate it?

Below, we have compiled a checklist of five items to help you fight off the possibility of a bias occurring on your survey.

Study your demographic

It goes without saying that understanding your target audience is the key to making a comprehensive survey. This is even more true when you want the survey to be bias-free. You'll need to make sure you tailor your survey to fit your target audience's needs.

Pro tip #1: Choose words that your audience is familiar with.

A poor understanding of your audience base can result in inconsistent responses and partially completed surveys. The good news is, learning about your audience is a fairly easy challenge to overcome. One way of getting to know your target audience better is by finding an answer to these questions:

What do the audience members have in common?

What aspect of your target audience are you specifically interested in learning about? (Example: Age, gender, educational qualification, occupation)

What motive does your target audience have to answer your survey?

After you have gained meaningful insight into the target demographic, you can move to the next item on the checklist: constructing questions.

Frame clear questions

When creating questions for a survey, keep in mind that they must never lead your responders to answer in favor of or against any answer. Be careful not to use strong, polarizing, emotionally-charged questions, as these can upset the respondents and are bound to result in bias.

Pro tip #2: Refrain from using words such as terrorist, calamity, fear, and the like.

Another factor to consider when writing surveys is the user's understanding of your questions. A person who is unfamiliar with your product may not be able to give accurate answers about it. If he/she has to choose an answer at random just to finish the survey, that can degrade your data accuracy.

Below is an example of how a good survey question can outperform a bad one in terms of accurate data collection.

To-do

Which of the following would you prefer for its ease of use?

- Survey Planet

- Pollfish

- Zoho Survey

- None

Not-to-do

Are you satisfied with Zoho Survey's ease of use?

- Yes

- No

Notice that the To-do example has a list of services that the responder might be familiar with. This creates a greater scope in the survey, making it easier to obtain an unbiased answer. On the other hand, the Not-to-do example is a survey-killer. Not only is it hard to understand what the responder actually feels about the service in comparison to others, but also the lack of variety in the answer options makes it harder to tell which answers are biased and which are unbiased.

It is also crucial to steer clear of absolutes (every, all, always, ever) in your survey questions. These force the responder into giving feedback which is inconclusive.

Let's take another look at an example of an impartial survey question and a biased survey question:

To-do

How often do you conduct surveys online?

- Less than 10 times a month

- 10-20 times a month

- Over 20 times a month

- Never

Not-to-do

Do you always conduct surveys online?

- Yes

- No

In the To-do example, the responder has the freedom to choose to answer with absolute certainty. On the other hand, the Not-to-do example creates confusion because of its vague answer choices. The question doesn't quantify the frequency that qualifies as always, and because of this, the answer of a vast majority of the responders would likely be a No. This once again compromises the data accuracy of your surveys and doesn't leave you with much specific information to analyze.

Structure the survey flow

Often, questions that focus on similar ideas, as well as surveys whose answer choices follow a pattern, can create false positives in the survey data. For example, when answer choices are recurring, (say, a-a-b-a-a-b) for a part of the survey, the responder's instinct might be to stick with the pattern and not answer the survey fairly. Randomizing the question patterns and the answer sequence is a standard practice adopted to avoid such bias.

Another method used to combat this problem is to have a branched survey flow, in which survey questions are a logical continuation of the preceding ones. For example, ask the responder which service he/she prefers based on the ease of use. According to the answer, branch to a different question asking to rate the ease of use of the particular service.

Pro tip #3: Present 2-3 questions per page in an online survey.

When all the questions on a survey are presented on a single page, the responder's opinion of the future questions can bias his/her answers for the previous ones. Additionally, the responders might immediately be put off by the length of the survey if it seems endless at first glance.

Provide a thorough answer list

While a lot of thought goes into forming a question, it can seem like not as much effort is put into creating the answer choices. It is essential that you provide the responder with a list of all possible options for the questions.

Pro tip #4: Answer choices such as 'Unsure', 'Neutral', 'Don't know,' etc. make for good opt-outs.

Having multiple answer options whose meanings are closely related can create unavoidable bias, mathematically, as well. To eliminate this, always give the responder a way out through your answer choices. This way, they can make a well-informed decision, which is, as we know, free of bias.

By providing opt-outs, the responder can choose the answer which he/she can relate to the most, instead of skipping the question altogether. This way, you are not left in the dark about unanswered questions and their possible interpretations.

Pro tip #5: Providing a numeric scale wherever possible is a great way to eliminate this error.

Below we have two more examples of survey questions: one which provides answer choices on a numbered scale, and one that doesn't:

To-do

On a scale of 1 to 5 (with 1 being the least and 5 being the most), how satisfied are you with Zoho Survey's ease-of-use?

- 1 (Extremely dissatisfied)

- 2 (Somewhat dissatisfied)

- 3 (Neutral)

- 4 (Somewhat satisfied)

- 5 (Extremely satisfied)

Not-to-do

How do you feel about Zoho Survey's ease-of-use?

- Alright

- Good

- Bad

Interval questions (shown in To-do) give the responder simple and coherent choices to pick from without prejudice. Not-to-do gives two qualitative options inclined towards a positive scale, but only one inclined towards the negative. This gives the positive answers a greater probability of being chosen even if they don't align with the responder's true beliefs.

Now that the audience is analyzed, questions carefully crafted, the survey structured, and answers ascertained, your focus should shift to creating a unique survey design for optimal impact.

Balance design and text

The harmony between text and visuals is paramount for conducting an online survey. Often, out-of-proportion images or page elements can take away from the survey. Lack of proper design can visually irritate a responder and distract them, which consequently results in unintentional bias.

Good design should also extend to mobile platforms as well. Responders prefer brief, to-the-point surveys that can be taken on hand in any situation: during a commute, a short break, or a lazy Saturday afternoon.

Additionally, most responders are comfortable taking these using their mobile phones. If the survey fails to be responsive on mobile, it can result in responders abandoning the survey and, consequently, cause bias to occur due to incomplete data collection.

Now that you've struck these five items off your checklist, you are all locked and loaded and ready to launch your survey, get accurate, unbiased data, and convert it into informative insight. To-do or Not-to-do will never be a question for you again.

Want to build a survey ASAP? Check out Zoho Survey! We have the answers to all your questions. Unbiased.