Get 10 Free Surveys

Survey bias: what are different types and how to avoid them?

By Elisabeth ·

Survey bias: what are different types and how to avoid them?

Taking care how you build customer surveys and which customers you survey comes down to this: if you’re going to do a job, do it well or don’t bother at all. Survey bias threatens the whole purpose of customer surveys. This is why eliminating survey bias is so important when collecting customer feedback.

The fundamental issue stems from the use of customer sampling as a proxy for an entire customer base or cohort. This is when surveys of ‘some’ customers are held up as being representative of ‘all’ customers. But this is only possible when survey bias has been negated. Done right, pollsters can accurately predict the outcome of political elections by asking less than 1% of the voting public. Done wrong, and they’re miles out.

But what about when sampling doesn’t matter and you just want feedback from customer X so you can understand them better? Survey bias creeps in here too. Ask the wrong question in the wrong way and the customer’s feedback will paint a picture that doesn’t match their reality.

What is survey bias?

Like any bias, survey bias is an input prejudice that skews output results. Survey biases are almost always unintentional errors. Sometimes they are more like wishful thinking. In each case, the survey creator has failed to execute a survey exercise with sufficient control over all its variables.

There are many kinds of survey bias, which we’ll explain. The benefit of confronting survey bias is to maximize data accuracy and quality, so you have more reliable data on which to base important decisions:

Eliminating all kinds of bias is very difficult. There’s probably always going to be some. That should not lessen your effort to reduce survey bias. These efforts make a significant difference and produce more valuable customer intelligence to act upon.

Types of survey bias?

The two main kinds of survey bias relate to the two key aspects of customer survey development. 

The following 12 types of survey bias apply to one or other of these primary aspects.

Sampling bias

Sampling bias is when the group of individuals in your sample group aren’t reflective of the group you are trying to survey. A classic example is women’s lingerie. Most lingerie is purchased by women; the people who wear the garments. But a sizable minority are husbands and partners buying them as gifts. Another example is children’s toys. Some are bought by parents, others by friends and relatives who don’t have appropriately aged children. The primary users of the product are children. In both examples, the buyers and users are not exactly the same. So then, how accurate are the survey results if these groups aren’t adequately represented? 

Another example is business software. Business software is typically purchased by a head of department or senior decision maker, validated by a finance decision maker and then implemented by the IT team. Beyond that will be a group of users that interact with the product day-in, day-out. Who gets the customer feedback survey? And what do the results mean?

Survivorship bias

Survivorship bias is the skewing of survey feedback toward recipients of longest tenure. Doing an employee survey? If you don’t do it frequently enough, or fail to insist upon the broadest inclusion criteria, you’ll only hear from those who’ve been in post for a while. Same with customers. The implications should be obvious – you’re not hearing from those who have churned and might have very enlightening input for you to work from.

Non-response bias

Non-response bias is when survey results are skewed by the absence of people who didn’t take part. This strikes at the heart of a major issue with customer feedback surveys – getting customers to respond to them. Organizations want to be ‘democratic’ when making decisions on behalf of customers, like what new features to introduce or opening hours to change. But, as political analyst Larry Sabato once said, “every election is determined by the people who show up.” You really want customers to “show up” or else you could be making decisions influenced only by those who did – and that could be bad for you in the long run.

Self-selection bias

Self-selection bias occurs when survey participation is optional, and the individuals who participate are not equally represented in their group criteria. For example, conducting a survey on a Saturday might omit more people who work on Saturdays (typically lower-paid retail, hospitality, public sector jobs) than those who work weekdays. 

Self-selection bias and non-response bias are closely linked in situations where customer surveys are not popular with recipients. The reason why long-form customer questionnaires are fast going out of fashion is not just because they are seldom completed. It’s also because the only people who do complete them are highly motivated to do so, and hold extremely negative or positive views. This skews results and ignores the perspective of the so-called silent majority.

Recall bias

Recall bias is when the accuracy of survey responses is compromised by the passage of time and its effect upon respondents’ memories of certain events. This is most likely to occur when there is a delay between the event and the survey question asking about the event. The issue is compounded when the event is not of great personal significance to the respondent and so unlikely to be strongly recollected weeks, days or even hours afterwards.

For example, asking how the customer’s hotel stay was the week after the customer has returned home. Unless something significant happened, the response is likely to be expressed as neutral or tending toward positive, rather than genuinely informative as a metric. Asking more specific questions after such a delay, e.g. about the quality of housekeeping, cleanliness of public areas, promptness of room service, etc. will produce yet more unreliable results.

Recall bias is a big problem for organizations who would like to be able to rely on the accuracy of customer feedback, but feel unable to trust it because of delays asking for and receiving responses. Or worse, deciding to act upon such information despite discrepancies in its accuracy.

Confirmation bias

Confirmation bias – also known as acquiescence bias – is when survey responses tend towards the positive. This can happen when questions are structured in such a way that respondents find it easiest to just go along with the basic premise. For example, asking  someone “to what extent do you agree with the following statement” may invite them to be more positive than if the question was posed in a different way. 

Sometimes confirmation bias can be a cultural phenomenon, with some groups more prone to agreeableness and offering positive responses out of politeness. This type of survey bias is closely related to so-called ‘courtesy bias’.

Neutral bias

Neutral bias is similar to confirmation bias, except the tendency is toward the ‘middle’ answer rather than expressing a definitive opinion. Again there is often some psychology at play here with respondents who do not wish to offend or be drawn into a discussion or conflict about problems that have occurred. It is certainly true that a great many negative experiences are expressed as neutral ones in survey exercises. 

In other cases, neutral bias happens when respondents don’t have the time to spend considering answers thoughtfully. They just check the boxes right down the middle. 

Conformity bias

Conformity bias should not be confused with confirmation bias, even though it sounds almost the same. Conformity bias happens when respondents subconsciously try to represent themselves positively, with social traits and behaviors that make them more closely aligned to acceptable norms. 

So, when asking about likes and dislikes (for example), conformity bias can lead some respondents to accentuate positive, and suppress negative, aspects of their character. A good example might be a question asking respondents how much they donate to charities. We all know that donating to charity is a good thing, so the figure we give will be closer to what we want it to be than what it actually is. Likewise, if a survey asks about diet, exercise and ‘vices’ like smoking and drinking. The way respondents report some of these will more closely reflect an aspirational view of themselves – and a version of reality they want others to perceive – rather than what’s really going on.

Question order bias

Question order bias is when the order in which survey questions are posed unduly influences respondents. For example, asking restaurant guests what they thought of their dessert first, followed by their view of the overall dining experience – versus the other way around. Human nature dictates that people are uncomfortable giving knowingly inconsistent responses. So if a customer said their dessert was good, they are less likely to contradict themselves by then saying their overall experience was bad (even though this is entirely possible). 

The impact of question order bias is not especially profound, but the effect can be measured and has been proven to have a subtle conditioning influence on respondents.

Another related form of survey bias is ‘answer option order’ i.e. the order in when response options are listed in multiple choice survey questions. The effect of this is likely to be even smaller.

Extreme response option bias

Extreme response option bias is another form of survey bias that deals with multiple answer options. Sometimes these question types are suitable for the objective at hand, sometimes they aren’t. The reason some customer feedback questions are expressed in different forms is typically to deal with survey bias issues. 

Take the example of a Likert scale question, which will have a list of up to 7 response options:

Now imagine respondents are given the question, “do you agree that violent murder should be a capital offence?” Such a highly emotive subject is likely to elicit responses that sit purely at the opposite ends of the spectrum; those against being people who disagree with capital punishment on a point of principle. In which case, perhaps a different question formulation would be preferable to get a more meaningful and nuanced understanding of different viewpoints.

Demand characteristics bias

Demand characteristics bias is when respondents to a survey are swayed by features of the survey process that alter their behavior. Organizations using surveys should exercise care to ensure that their desire to communicate the purpose of the survey doesn’t unduly affect the results provided. 

For example, being asked about how well their guide/server/agent performed may be perceived by customers as an opportunity to facilitate a reward for that person or earn them a rebuke. Often customer-facing staff who know that customers will soon receive a feedback survey might say, “you’ll get a survey emailed to you very soon, and it really helps us if you give 5 stars”. A survey conducted to scientific standards would tightly control what respondents knew about the survey beforehand, and the conditions and context in which they underwent it.

Interviewer bias

Interviewer bias is when the behavior of a survey interviewer affects the behavior of the respondent, or vice versa. This is only really applicable in research settings where both are communicating directly, such as in a phone call, video conference or face-to-face interaction.

How to combat each type of survey bias?

Let’s look at how these 12 types of survey bias can each be confronted and mitigated. Remember, eliminating all forms of survey bias is nigh-on impossible, but mitigating them as much as you can will boost data accuracy, quality and reliability.

Sampling bias

The issue: You’re only generating feedback from a certain portion of your intended audience.

The solution: Work back to the purpose of the survey and who exactly you need to get feedback from. Design a customer feedback approach that doesn’t leave anyone important out. 

Try this: Look at your audience personas for communications channel preferences and which customer journey touchpoints make sense to launch surveys from. Design surveys for maximum engagement, minimum effort.

Survivorship bias

The issue: You’re only hearing from stable customers who’ve hung around long enough to show up in your results.

The solution: Get more feedback from customers who’ve only purchased once/twice, are likely to churn or who have already churned.

Try this: Follow-up negative feedback from customers to understand the reasons why. Consider ways of surveying ex-customers and individuals who were once ‘prospects’ but who never converted to being customers.

Non-response bias and self-selection bias

The issue: The lack of response and engagement with customer feedback surveys is skewing the results toward the views of the people responding.

The solution: Increase response rates among all groups, particularly those who’ve become disengaged or have lower motivation to express feedback.

Try this: Issue shorter, more engaging single-question surveys embedded in emails you’re already sending.

Recall bias

The issue: The results from survey respondents cannot be relied upon because of the delay in asking them about previous experience/s.

The solution: Maximize recall by reducing the time between the event occurrence and the survey question asking about it.

Try this: Trigger customer feedback surveys as soon as events are completed e.g. customer support ticket closed, salon appointment finished, package delivered, etc. 

Confirmation bias

The issue: Survey respondents are tending to the positive because of a desire to affirm and be agreeable.

The solution: Don’t ask leading questions. Redesign survey questions to elicit more thought provoking responses. 

Try this: Mix up the types of survey questions you ask so that they aren’t all the same response options. 

Neutral bias

The issue: Customers are ‘sitting on the fence’ by giving neutral responses that don’t really tell you anything,

The solution: Redesign survey questions to only offer an even number (2, 4, 6, etc.) of responses, thereby eliminating the ‘middle answer’ and forcing a positive/negative decision.

Try this: Reduce the number of questions that you ask, to decrease the likelihood of disengaged box-checking. 

Conformity bias

The issue: Respondents are being unconsciously dishonest in their survey answers because of how they wish to be perceived.

The solution: Reassure respondents that their answers will be treated confidentially, and make all surveys anonymous.

Try this: Don’t ask loaded questions.

Question order bias

The issue: The way respondents answer questions is being unduly influenced by their running order.

The solution: Change the running order of questions to minimize the possibility of bias.

Try this: Randomize the running order of questions and their response options.

Extreme response option bias

The issue: Respondents are only engaging with the most extreme response options, making it difficult to discern nuance in the results.

The solution: Redesign survey questions to be less absolute and with fewer response options. 

Try this: Reduce the emotiveness of questions as much as possible. 

Demand characteristics bias and interviewer bias

The issue: Survey respondents are being unduly influenced into changing their behavior by the conditions and context of the feedback survey. 

The solution: Police the influence of staff members by asking them not to encourage customers to respond positively to a feedback survey.

Try this: Communicate clearly and transparently the high-level constructive purpose behind the customer feedback survey without going into specifics around the implications for staff.

Examples of biased survey questions

There are 5 main kinds of biased survey questions:

Leading questions

Leading questions lead survey respondents toward a particular answer. They have ‘an agenda’ and this can usually be spotted by the unnecessary use of subjective language.

For example:

How much did you enjoy your stay at the wonderful new Fort Lauderdale Hyatt?

What problems did you encounter with having your car serviced by our team today?

The interference here is that the customer definitely had a problem and must contradict the premise of the question in order to make clear that they didn’t.

Leading questions can be innocent mistakes, though frequently they emerge because of the organization’s enthusiasm to create positive results rather than truthful ones. This is short-sighted. Leading questions are no longer leading when they use objective language and do not present areas of contention as statements of fact. 

Loaded Questions

Loaded questions influence respondents to give answers that don’t necessarily reflect their true opinion or experience. Loaded questions contain inferred assumptions, hence they are also known as assumptive questions.

Here are some examples:

Loaded questions are very efficient at accruing meaning, but the result is biased. A better approach is to deconstruct these questions into simpler steps. For example, by asking if the customer enjoys drinking wine. Then, once qualified, ask each of the subgroup of ‘wine drinkers’ where they most enjoy doing this.

Double-barreled Questions

Double-barreled questions are overly complex because they attempt to ask two questions in a single proposition. Again, this is typically a mistake. However, the results can confuse recipients and lead them to give uncertain or inaccurate responses.

Take these examples:

Survey questions should always examine a single aspect at a time. A double-barreled question invariably works better as two distinct single-barreled questions. 

Absolute Questions

Absolute questions use language that deals in ‘absolute’ concepts like always, all, never, ever and every. They also tend to ask for yes / no answers which close down respondents into giving answers that don’t give space for more representative feedback.

Here are some examples:

As well as cutting down the use of absolute language, it’s worth challenging yourself as to whether it’s necessary to only offer yes/no as answer options. Sometimes it’s warranted, but often it isn’t. At the very least, an option for ‘Other’ or ‘Don’t know’ provides the opportunity to opt out of the binary decision. Typically an answer scale would be better suited to these types of questions.

Confusing Questions

This last group of questions covers a multitude of sins. Poor grammar, question structure and technical jargon often result in confusing questions, which in turn create survey bias as the responses will be affected by the respondents’ lack of understanding. </resources/reasons-to-choose-customer-thermometer/>

Take these examples:

How can Customer Thermometer help prevent survey bias?

Survey bias is a real risk to the integrity of customer feedback data which undermines hard work and investment. If you can safely navigate survey bias, you are primed to benefit from reliable customer intelligence on which to confidently base important business decisions.

To combat survey bias, Customer Thermometer has pioneered the simple customer feedback process, which is harder than it sounds. Our approach combines a robust, proven feedback survey design and distribution system with a user experience that your customers want to engage with. The result is uncommonly high response rates that capture fresh, in-the-moment customer feedback.

Here are just 4 of the ways Customer Thermometer prevents survey bias:

Maximize response rate

The key to maximizing customer feedback response rate is highly engaging surveys that are effortless to complete. We allow brandable, customizable survey designs that extend familiar and reassuring brand experiences, and encourage the use of expressive, emoji-style response icons that customers want to click.

We also make it devastatingly easy for customers to provide feedback – literally via a single click. These core attributes consistently enable industry-leading customer feedback response rates that make survey results representative and stave off non-response and self-selection survey biases.

Facilitate off-the-fence feedback

It’s not all about response option icons and emojis. Customer Thermometer supports the full range of specific response-driven metrics like CSAT, CES and NPS. 

We also recommend our 4-option response card as standard. The options are customizable, but the default setting is for gold, green, amber and green – meaning there is no ‘neutral’ option that allows customers to sit on the fence. And therefore, no neutral bias.

Enable question flexibility and follow up

There are lots of different survey types out there, and we support them all. We just don’t recommend pumping surveys full of lots of questions that demand too much customer effort and see response rates decline. 

In fact, the Customer Thermometer way is the single-question, 1-click customer feedback survey that guarantees the highest response rate. But one with a trick up its sleeve! Once the question is posed and the customer clicks their response, a page can be launched that allows the respondent to answer a follow-up question, leave comments or signal the drivers behind their choice of answer. All of it adds richness and value, without diminishing the core objective of an unbiased survey.

Real time recall

Last but not least, a real USP for Customer Thermometer is straight-line speed, giving you real-time results that maximize the power of customer recall and avoid ‘delay bias’. This is achieved by embedding questions into the notification emails you already send off the back of customer interactions and touchpoints. Secondly, our integrations with a host of CRMs, PSAs and other enterprise platforms means workflows are connected and deduplicated. You get the reporting information you need to make faster, better decisions based on data you can trust. 

Want to get started? Begin your Customer Thermometer trial, for free (no CC details will be requested), whenever you like, for 14 days.