WorksheetsWhat is Testing & Validation? INNOVATION AND ENT.WK 3 JAN 25C
Total questions: 87
Worksheet time: 44mins
Fill in the blank: The definition of Testing & Validation is systematic processes to verify that a product works as intended and meets user needs before _________
full market release
initial prototype development
internal team review
customer feedback collection
What is the main purpose of Testing & Validation?
To increase product price
To identify issues early, reduce development costs, and ensure market fit before widespread distribution
To delay product launch
To ignore user needs
According to Weinberg (2008), what is the value of testing?
Testing is a one-time process
Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous
Testing is only for software
Testing is unnecessary
Fill in the blank: The first step in the Testing & Validation Process is ________.
Planning
Execution
Reporting
Review
Which step in the Testing & Validation Process involves creating detailed scenarios to verify functionality and user experience?
Planning
Designing Test Cases
Executing Tests
Iterating
Fill in the blank: ________ is the step where tests are run with real users or automated tools to identify issues.
Executing Tests
Writing Code
Designing Interfaces
Gathering Requirements
Which step in the Testing & Validation Process involves evaluating findings to determine required improvements?
Executing Tests
Analysing Results
Iterating
Planning
Fill in the blank: The final step, ________, involves implementing changes and repeating testing until quality targets are met.
Iterating
Planning
Documenting
Designing
Fill in the blank: Tesla collects driving data from customer vehicles without activating Autopilot to validate algorithms in real-world conditions. This process is known as _________
Shadow Mode Testing
Beta Release Monitoring
Passive Data Logging
Simulated Driving Analysis
Fill in the blank: Over 1 million vehicles contribute to a neural network that improves with each mile driven. This process is called _________
Fleet Learning
Autopilot Mode
Smart Routing
Dynamic Mapping
Fill in the blank: Features are released gradually to limited driver groups before wider distribution based on performance metrics. This process is known as _________
Iterative Deployment
Hotfix Release
Rollback Update
Monolithic Launch
Which of the following tasks should students perform for a mobile banking app's fund transfer feature?
Design three test cases
Write user documentation
Develop a marketing plan
Create a logo
Class Activity: Test Case Design
Test Case Design
Code Implementation
System Deployment
User Training
Class Activity: Test Case Design
Which of the following best describes what you should do during the presentation?
Share your test cases and explain your reasoning
Write new test cases
Grade your classmates' test cases
Ignore the presentation
Which type of user testing involves observing users performing specific tasks?
Usability Testing
A/B Testing
Beta Testing
User Research
Which type of user testing is described as 'Comparing two versions to determine better performance'?
Usability Testing
A/B Testing
Beta Testing
User Research
Which type of user testing involves a limited release to gather real-world feedback?
Usability Testing
A/B Testing
Beta Testing
User Research
Which type of user testing focuses on understanding user needs and behaviors?
Usability Testing
A/B Testing
Beta Testing
User Research
Fill in the blank: "User testing is about giving users a ______ in the design process" (Krug, 2014).
voice
break
deadline
reward
Which usability testing method involves a controlled environment with observation equipment and one-on-one moderation, providing rich qualitative data?
Remote Testing
Lab-based Testing
Guerrilla Testing
Eye-tracking Studies
Which usability testing method enables participants to test from their own locations, allowing for broader geographic reach and natural usage environments?
Lab-based Testing
Remote Testing
Guerrilla Testing
Eye-tracking Studies
Which usability testing method is characterized by quick, informal testing in public places, offering low cost and fast insights from random participants?
Lab-based Testing
Remote Testing
Guerrilla Testing
Eye-tracking Studies
Which usability testing method uses specialized equipment to record precise eye movements and reveals subconscious user behaviors and attention patterns?
Lab-based Testing
Remote Testing
Guerrilla Testing
Eye-tracking Studies
Match the following phases of Airbnb's User Testing with their correct descriptions:
Discovery Phase
Designers create interactive mockups based on research insights for initial testing.
Prototyping
Iterative updates to the platform based on test findings.
Usability Testing
Airbnb researchers conduct contextual inquiries with hosts and guests in their homes (Schleifer, 2016).
Data Analysis
Both in-person moderated sessions and remote unmoderated tests validate new features.
Implementation
Combining qualitative feedback with quantitative metrics from A/B tests.
In the Discovery Phase of Airbnb's User Testing, what do researchers conduct with hosts and guests in their homes?
Contextual inquiries
Usability testing
A/B testing
Surveys
Which phase of Airbnb's User Testing involves designers creating interactive mockups based on research insights for initial testing?
Discovery Phase
Prototyping
Usability Testing
Data Analysis
Implementation
What is the main activity in the Usability Testing phase of Airbnb's User Testing?
Both in-person moderated sessions and remote unmoderated tests validate new features.
Only remote unmoderated tests are conducted to validate new features.
Usability Testing focuses solely on collecting user feedback through surveys.
The phase is dedicated to updating the Airbnb homepage design.
During which phase of Airbnb's User Testing are iterative updates to the platform made based on test findings?
Discovery Phase
Prototyping
Usability Testing
Data Analysis
Implementation
Fill in the blank: The first step in conducting effective user tests is to _________.
Define Objectives
Recruit Participants
Analyze Data
Create Wireframes
Which step involves finding users matching your target demographic?
Define Objectives
Recruit Participants
Create Realistic Tasks
Analyse Findings
Match the following steps to their descriptions:
Collect data without influencing participants
B
Set clear goals and specific questions to answer
D
Identify patterns and prioritize improvements
E
Find users matching your target demographic
A
Design scenarios reflecting actual use cases
C
Fill in the blank: The step where you design scenarios reflecting actual use cases is called _________
Create Realistic Tasks
Conduct Usability Testing
Analyze User Feedback
Develop Wireframes
'Analyse Findings' is the step where you collect data without influencing participants.
True
False
Step 1: Group Formation
How many students should be in each group when forming groups for the food delivery app activity?
1-2
3-4
5-6
7-8
Which of the following is a key component of a usability test script?
Introduction, specific tasks, and follow-up questions
Only an introduction
Only specific tasks
Only follow-up questions
List the scenarios you included in your usability test script and explain why you chose them.
To identify common user tasks and understand their importance.
To evaluate the visual design of the interface only.
To test the speed of the internet connection.
To check the hardware compatibility of the device.
Step 4: Presentation Present your script to the class and explain your methodology. How would you present your script and explain your methodology to the class?
By reading the script aloud and describing the steps taken
By submitting the script without explanation
By only showing the final result
By not presenting at all
Fill in the blank: A/B Testing is a randomized experiment comparing two versions (A and B) to determine which performs better against ________ metrics. (Kohavi et al., 2020).
business
aesthetic
random
subjective
Fill in the blank: Key components of A/B Testing include control version, variation, random user assignment, statistically significant sample, and clear ________ metrics.
success
color
location
timing
Which of the following is NOT an application of A/B Testing?
Weather forecasting
Emails
Landing pages
Product features
Checkout flows
What is the first step in the A/B Testing Process?
Identify Metrics (Define measurable success indicators)
Launch the experiment without planning
Analyze results before collecting data
Ignore user segmentation
Which step in the A/B Testing Process involves creating a testable prediction of outcome?
Identify Metrics
Form Hypothesis
Create Variations
Run Experiment
Match the step number with its description in the A/B Testing Process:
Deploy to randomized user segments
C
Design alternative version with specific changes
E
Define measurable success indicators
B
Evaluate results and apply winning version
A
Create testable prediction of outcome
D
Fill in the blank: The step in the A/B Testing Process where you 'Deploy to randomized user segments' is called ________.
Run Experiment
Collect Data
Analyze Results
Define Hypothesis
'Analyse & Implement' is the final step in the A/B Testing Process where you evaluate results and apply the winning version.
True
False
According to the slide, how many concurrent A/B tests does Booking.com run?
10+
100+
1,000+
10,000+
Booking.com tests everything from minor UI elements to major features.
True
False
Fill in the blank: Booking.com's culture prioritizes _______ evidence over opinions, regardless of seniority.
experimental
anecdotal
theoretical
subjective
What is the definition of P-value? Fill in the blank: P-value is the _________.
Probability results occurred by chance
Probability that the null hypothesis is true
Probability that the alternative hypothesis is true
Probability that the experiment was conducted correctly
What is the importance of Sample Size in A/B testing? Fill in the blank: Larger samples _________.
increase result reliability
decrease the accuracy of results
make tests inconclusive
reduce the need for statistical analysis
What does Confidence Interval represent in statistical significance? Fill in the blank: Confidence Interval is the _________?
Range containing true value
Average of all values
Median of the data
Standard deviation of the sample
What is the importance of Statistical Power? Fill in the blank: Higher power _________?
reduces false negatives
increases sample size
increases false positives
decreases statistical significance
Class Activity: A/B Test Hypothesis
Element Selection: Which element would you choose to test in an A/B test?
CTA button
Headline
Product image
Pricing display
Navigation
Class Activity: A/B Test Hypothesis
Which of the following is an example of creating an alternative version based on psychological principles or user feedback for your chosen element?
Changing the button color to red based on user feedback
Keeping the original design unchanged
Ignoring user feedback
Removing the element entirely
Write a formal hypothesis: "Changing X to Y will increase Z by N% because..." Fill in the blanks to form your hypothesis.
Answers will vary. Example: Changing the CTA button color to green will increase click-through rate by 10% because green is more attention-grabbing.
Changing the CTA button color to blue will decrease click-through rate by 10% because blue is less visible.
Changing the CTA button color to red will have no effect on click-through rate because color does not matter.
Changing the CTA button color to yellow will decrease click-through rate by 5% because yellow is distracting.
What metrics would you use to measure the success of your test?
Test coverage, defect density, and user satisfaction
Test duration and number of team members
Project budget and timeline only
Number of meetings held
What feedback did you receive from your partner, and how did you refine your hypothesis?
I received constructive feedback and refined my hypothesis accordingly.
I ignored the feedback and did not change my hypothesis.
I received no feedback and kept my hypothesis the same.
I received negative feedback and abandoned my hypothesis.
What method involves structured questionnaires to collect quantitative and qualitative data on specific topics?
Surveys
Case Studies
Experiments
Focus Groups
Which market feedback method involves moderated discussions with representative users to explore perceptions and preferences?
Focus Groups
Surveys
One-on-One Interviews
Field Trials
What is the method of monitoring online conversations to identify trends, sentiment, and emerging issues called?
Social Listening
Content Marketing
Influencer Outreach
Search Engine Optimization
Which method involves one-on-one conversations to deeply understand individual experiences and motivations?
Customer Interviews
Surveys
Focus Groups
A/B Testing
Which type of analysis involves statistical analysis of numeric data?
Quantitative Analysis
Qualitative Analysis
Descriptive Analysis
Comparative Analysis
Which type of analysis uses frequency counts and distributions?
Quantitative Analysis
Qualitative Analysis
Comparative Analysis
Descriptive Analysis
Which type of analysis involves correlation and regression analysis?
Quantitative Analysis
Qualitative Analysis
Descriptive Analysis
Comparative Analysis
Which tools are used for Quantitative Analysis?
SPSS, Excel, R, Python
Photoshop, Illustrator, InDesign, Premiere
WordPress, Wix, Squarespace, Shopify
AutoCAD, SolidWorks, Revit, SketchUp
Which type of analysis involves thematic coding of text responses?
Qualitative Analysis
Quantitative Analysis
Descriptive Statistics
Inferential Statistics
Which type of analysis uses sentiment analysis of comments?
Qualitative Analysis
Quantitative Analysis
Descriptive Analysis
Predictive Analysis
Which type of analysis involves content analysis of interviews?
Qualitative Analysis
Quantitative Analysis
Descriptive Analysis
Statistical Analysis
Which tools are used for Qualitative Analysis?
NVivo, Atlas.ti, MAXQDA
SPSS, SAS, R
AutoCAD, SolidWorks, CATIA
MATLAB, Simulink, LabVIEW
Which step in action planning involves prioritizing insights by impact?
Action Planning
Goal Setting
Resource Allocation
Implementation
Which step in action planning involves creating actionable recommendations?
Action Planning
Goal Setting
Monitoring Progress
Evaluating Outcomes
Which step in action planning involves defining implementation timelines?
Action Planning
Goal Setting
Resource Allocation
Performance Review
Which step in action planning involves establishing feedback loops?
Action Planning
Goal Setting
Resource Allocation
Risk Assessment
How many active customers does Monzo Bank have?
1M+
3M+
5M+
10M+
How many community members participate in Monzo Bank's product forums?
1K+
3K+
6K+
10K+
What percentage of feature adoption is based on community-suggested improvements at Monzo Bank?
50%
60%
70%
80%
What is the average feedback response time at Monzo Bank?
12hrs
24hrs
48hrs
72hrs
Fill in the blank: The first step in integrating testing, validation, and feedback is ________, which involves understanding user needs and problems.
Research
Implementation
Prototyping
Testing
Fill in the blank: The second step in integrating testing, validation, and feedback is ________, which involves creating potential solutions.
Design
Testing
Evaluation
Implementation
Fill in the blank: The third step in integrating testing, validation, and feedback is ________, which involves validating with users.
Test
Design
Plan
Implement
Fill in the blank: The fourth step in integrating testing, validation, and feedback is ________, which involves building and deploying solutions.
Implement
Design
Evaluate
Plan
Fill in the blank: The fifth step in integrating testing, validation, and feedback is ________, which involves gathering performance data.
Measure
Design
Plan
Report
Who said the following quote? "The key is to embrace failure as a learning opportunity."
Ries
Steve Jobs
Elon Musk
Peter Drucker
Fill in the blank: Every £1 spent on testing saves £___-___ in development and post-launch fixes.
10-100
1-5
2-8
5-20
Which key takeaway emphasizes prioritizing user needs identified through systematic testing?
Testing as Investment
User-Centricity
Data-Driven Decisions
Continuous Improvement
A/B testing provides empirical evidence to resolve design and strategy debates. Which key takeaway does this statement describe?
Testing as Investment
User-Centricity
Data-Driven Decisions
Continuous Improvement
Testing and feedback should form an ongoing cycle throughout product lifecycle. Which key takeaway does this statement describe?
Testing as Investment
User-Centricity
Data-Driven Decisions
Continuous Improvement
