Insights Refresher - Biases

Foundational Testing Concepts Refresher - Biases in Research

What is bias?

• Systematic errors in our thought processes that impact the way we live and work 

• An inclination of temperament or outlook 

• Often associated with prejudice or favoritism 

Biases in Research

• Researcher Bias

• Respondent Biases

• Analysis Biases

Researcher Bias

• Leading questions: Rather than allowing the respondent to answer the question honestly, these types of questions nudge them in a particular direction. 

• Loaded questions: Force respondents to answer in a way that may not accurately indicate their true opinion or experience. 

• Order bias: Questions positioned earlier in your survey might influence how respondents choose to answer questions that appear later in your survey. 

Respondent Bias

• Status quo: Preferring things to stay the same.

  • Ex: Presenting a new & affordable way to stream TV but most participants reporting they would prefer Netflix since it’s what they are used to.

• Availability heuristic: Using info that comes to mind quickly/ easily

  • Ex: A respondent is currently watching “The Voice”. They respond with “The Voice” when asked what their favorite all-time TV show is, despite having many other favorite shows.

Analysis Bias

• Framing effect: Drawing different conclusions from the same information depending on how it’s presented 

  • Ex: Hand sanitizer is labeled as "99% effective" vs. "1% not effective"

• Confirmation bias: Finding and remembering information that confirms our perceptions & beliefs 

  • Ex: Closely looking for patterns in your test data to prove your hypothesis, overlooking the data that says otherwise. 

Multiple Choice

Why do we randomly select our samples?

• For control

• So we don't get the same answers from everyone

• Reduce bias

• Eliminate confounding results

Multiple Choice

You are trying to get a rating from users on their experience when contacting customer service. What is the better way to phrase the question? 

• How disappointing is it when you can't find out how to contact customer service?

• How easy or difficult is it to contact customer service?

Multiple Choice

You are trying to determine whether someone has used a mobile app to book a hotel recently. What is the better way to phrase the question? 

• 2. Which of the following have you done in the past 30 days? 

  1. - Rented a car 

  2. - Applied for a credit card online 

  3. - Bought something on Facebook marketplace 

  4. - Used an app to book a hotel 

  5. - Called a restaurant direction to make a reservation 

• 1. Have you used a mobile app to book a hotel in the past 30 days? 

  2. - Yes

  3. - No

Multiple Choice

You want to determine how much people liked a YouTube video. What is the better way to phrase the question? 

• How much did you enjoy this YouTube video? 

• On a scale of 1-5, with 1 being the worst and 5 being the best, please rate this YouTube video 

Multiple Choice

Which of the questions below is an example of order bias? 

• 1st Q: How interested are you in playing on a company softball team? 2nd Q: What is your favorite sport?

• 1st Q: What is your favorite sport? 2nd Q: How interested are you in playing on a company softball team?

Multiple Select

What are some ways you can avoid bias in your research?

• Ask neutrally worded questions

• Ask questions that are not leading

• Ask yes/no questions

• Send your survey to someone to proofread

• Don't insert your own opinion