Thinking Like a Product Analyst

Foundational Testing Concepts Refresher - Thinking Like a Product Analyst

Survivorship Bias

Survivorship bias is an analytical bias that can lead to concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. 

Twyman's Law

Correlation vs. Causation - Twyman's law states that any data or figure that looks interesting or different is usually wrong. When there’s so much data to work with, it’s easy to get careless and assume that the numbers right under your nose are always telling you the truth.

Simpson's Paradox

Simpson's paradox is a phenomenon in probability and statistics in which a trend appears in several groups of data but disappears or reverses when the groups are combined. 

Multiple Choice

A common analytical fallacy that product analysts should avoid is:

• Larger sample sizes are always better 

• Correlation implies causation

• All data points are equally important 

• Statistical models are always accurate 

Multiple Choice

When analyzing user engagement data for the NBC app, how might survivorship bias impact your conclusions if you only focus on active users and ignore users who have churned? 

• It may lead to an inaccurate assessment of the popularity of certain features or content. 

• It may result in overestimating the effectiveness of marketing campaigns. 

• It may cause difficulty in identifying trends in user behavior. 

• It may lead to inaccurate predictions of future user activity. 

Multiple Choice

Fill in the blank: Survivorship bias can be mitigated by actively seeking feedback from _________ to understand their reasons for being inactive.

• Premium users

• New users

• Churned users

• Current users

Multiple Choice

You're tasked with optimizing the user experience on the NBC app. Your data suggests that a specific genre of content has a high engagement rate, but further analysis reveals that this engagement is primarily driven by a small subset of users. How would you apply the concepts we’ve learned to make informed decisions about promoting content within this genre? Which of the following strategies would best address the situation described in the scenario? 

• Focus solely on promoting content within the popular genre to maximize engagement. 

• Explore ways to diversify content offerings within the popular genre to appeal to a broader audience. 

• Discontinue content production within the popular genre to allocate resources elsewhere. 

• Conduct additional research to identify the specific characteristics of the subset of users driving engagement and tailor content recommendations accordingly. 

Multiple Choice

Can you apply Twyman’s Law to any of the following scenarios?

• A new piece of code breaks your homepage, but after looking at your analytics you see that users are spending more time on it than ever—and therefore must be more engaged. 

• You collect the birthday of every user that signs up for your service. What you find shocks you—nearly 5% of all your users were born on January 1st. 

• Your web analytics team came to you with a surprising revelation. Your e-commerce business—usually active 24 hours a day, 365 days a week—shows no sales, no visitors, no nothing for an entire hour between 2am and 3am on March 12th, 2017. 

• None of the above 

• All the above 

Can you apply Twyman’s Law to any of the following scenarios?

A new piece of code breaks your homepage, but after looking at your analytics you see that users are spending more time on it than ever—and therefore must be more engaged. 

Can you apply Twyman’s Law to any of the following scenarios?

You collect the birthday of every user that signs up for your service. What you find shocks you—nearly 5% of all your users were born on January 1st. 

Can you apply Twyman’s Law to any of the following scenarios?

Your web analytics team came to you with a surprising revelation. Your e-commerce business—usually active 24 hours a day, 365 days a week—shows no sales, no visitors, no nothing for an entire hour between 2am and 3am on March 12th, 2017. 

Answers

• Visitors are spending more time on your homepage, but only because it’s broken and it’s taking longer for them to do what they want to do. 

• The fastest way to fill out your mandatory birthday collection form is to simply pick January 1st from the dropdown menu. 

• In spring, many countries turn clocks one hour forward in a tradition known as Daylight Savings Time—hence the lack of any sales (or any activity, for that matter) during an hour that simply doesn’t exist. https://amplitude.com/blog/twymans-law