ISWS 2024 - User centered Evaluation tutorial

Copyright © Cefriel – All rights reserved

Bertinoro, June 12th 2024

User-Centered
Evaluation

Irene Celino
[e-mail]

Multiple Choice

Warm-up question: in which Italian region are we now?

• Tuscany

• Emilia-Romagna

• Sicily

• Algarve

Multiple Choice

Who's your best friend for user evaluation analysis?

• An anthropologist

• A statistician

• A philosopher

• A rich sponsor

Multiple Choice

How many users do you need to involve in a user evaluation study?

• 30

• 300

• 3,000

• It depends

Multiple Choice

Ok, the number of required evaluators depends... on what?

• The number of people I know

• The complexity of the evaluation task

• The size of the target user population

• The money I can invest to pay evaluators

Multiple Choice

Now, say you need around 100 evaluators. Therefore you involve...

• 100 friends from the target population

• 100 representative users from the target population

• The first 100 people available

• Ignore statistics, only the people I can get

Multiple Choice

You failed to involve enough users in your evaluation. Now what?

• You cannot include your evaluation in a paper

• You have to keep going, even for months, until you reach the right number

• You are doomed, better give up your PhD

• You can still derive some insights from the data you collected

Multiple Choice

And how many users do you involve in a qualitative evaluation study?

• Exactly the same as quantitative evaluation

• 3 evaluators per each "category" of users in the target population

• At least 10% w.r.t. quantitative evaluation

• Nah, 4 or 5 users in total are more than enough

Multiple Choice

What do you do in a "thinking out loud" method?

• You clearly express your opinion on why the evaluated system is beautiful

• You let the evaluators scream to release tension before the evaluation

• You ask the evaluators to describe what they do and think during evaluation

• You shout out to evaluators to hurry up

Multiple Choice

What's best: qualitative or quantitative user evaluation?

• Always quantitative: numbers don't lie

• Always qualitative: humanity is not explained by numbers

• Both qualitative and quantitative would be the best!

• Neither of them: I hate people

Multiple Choice

When can you run an evaluation campaign, say of a software tool?

• A drawing on a napkin could be enough to evaluate

• Only when the user interface design is finished

• Only when the main features are implemented

• Only when all bugs are solved

Multiple Choice

What is the evaluation protocol?

• The permission from the ethics committee to run the evaluation

• The official paper on which you report the evaluation results

• The rules and behaviours you suggest to evaluators to positively assess your work

• The instructions to execute the tasks and the set of evaluation questions you ask to participants

Multiple Choice

What are A and B in A/B testing?

• Two variants of the same evaluated variable

• Two runs of the same testing procedure

• Two participants to the same study

• Two possible outcomes of the same experiment

Multiple Choice

OK, you got it! So, in A/B testing you will test...

• Two system variants with the same group of users

• Two groups of users with the same system variant

• Two system variants with two similar but distinct groups of users

• Two system variants with two groups of users with different characteristics

Multiple Choice

Say you want to evaluate your system usability. The first thing you do is...

• Make sure your system is usable

• Re-define usability in your context

• Use your system multiple times to correct bugs

• Look for a pre-existing usability test

Multiple Choice

What should NEVER be changed or adapted in a STANDARDIZED user evaluation questionnaire?

• The minimum number of factors/dimensions of the evaluation

• The order of the items/questions in the questionnaire

• The maximum number of factors/dimensions of the evaluation

• The order of the predefined answers to the items/questions

Multiple Choice

So, in the end, what was this "User-Centered Evaluation" tutorial?

• Just a gamified lecture

• A proper user evaluation study

• A sort of user evaluation study

• A complete waste of my time