SAN6106

Mental Models & Human Error

SAN6106
Prof. Dr. Adviye Ayça ÜNLÜER
2024-2025 Fall Term
Meral ŞENTÜRK

Note to the Audience

This presentation is prepared using information derived from Foundations and Fundamentals in Human-Computer Interaction (1st Edition) edited by Constantine Stephanidis and Gavriel Salvendy. It also includes additional data and activities not covered in the textbook. For a comprehensive understanding, please refer to the original text.

A brief introduction to mental models

 Representation and characteristics of mental models

 Why to study mental models

 Important aspects of studying mental models

 How to elicit mental models

 How to analyze mental models
Research Samples
Demonstration

​Ch6 Human Error

Origins and the Use of Term

 Basic Premises of Human Error

 Anatomy of an Error

 Human Error Analysis

 Human Error Taxonomies

 Human Error in HCI
Guidelines
Research Samples
Final Remarks-Gaps and Future Work

​Ch5 Mental Models

Content

• ​Mental models are lack clear boundaries

• Schema or mental model dilemma

• Shares common ground and triggers cognitive maps

• Involves semantic networks

• Conceptual models can be interpreted as mental models (Allen, 1997)

​​Subset of Mental Representations

​​Mental Model vs. Schema

Study Methods of MMs

• Phase 1 Behavioral Methods: Stimulu-response (observable behaviors and the impact of environmental stimuli)

• Phase 2 Cognitive Psychology: Brain as a computer

• Phase 3 Neuroscience Method: Biological underpinnings of human thinking and reasoning

  the brain (i.e., neuroscience) versus the mind (i.e., cognitive psychology)

Domains Studying Mental Models

• Organizational Behavior and Management

• Philosophy

• Linguistics and Semantics

• Public Health and Medicine

• Psychology

• Cognitive Science

• Educational Sciences

• Neuroscience

• Information Engineering and Artificial Intelligence

• Economics and Behavioral Economics

Key Domains of MMs in HCI

• Learnability and Onboarding

• Metaphors and Analogies in Design

• Cross-Platform Consistency

• Personalization and Adaptive Interfaces

• Mental Model Testing and Evaluation

• User Interface Design

• Usability and User Experience (UX)

• System Feedback and Affordances

• Information Architecture

• Error Prevention and Recovery

• Cognitive Load and Efficiency

Objectives of Studying MMs in HCI

• Ensure Consistency Across Platforms

• Adapt to Individual Needs

• Innovate Responsibly

• Reduce Development Costs

• Inform AI and Automation Design

• Enhance User Experience

• Improve Usability

• Support Effective Decision-Making

• Reduce Cognitive Load

• Facilitate Learnability and Onboarding

​Research Types

Various study types in mental model research include descriptive, experimental, and comparative studies to understand and compare users' interpretations, as well as longitudinal and cross-sectional studies to track changes and capture diversity in mental models. Other methods, like case studies, field studies, and evaluation studies, provide in-depth or real-world insights, while action research and phenomenological studies allow iterative refinement and exploration of users' lived experiences.

​Research Approaches and Data Strands

Mental model research can be approached through qualitative, quantitative, and mixed methods.

-Behavioral Data: This strand involves tracking users' actions within a system, such as navigation patterns, key strokes, time on tasks, and error frequency.

-Physiological Data: Biometric data, such as eye-tracking, heart rate, or skin conductance, can indicate user focus, stress, or engagement levels.

-Self-Reported Data

-Contextual Data

-Historical Data

Human factors in MMs

• Overconfidence

• Inconsistent Behavior

• Learning Difficulties

• Error Propagation

• Inflexibility

• Confirmation Bias

• Resistance to Change

• Misalignment with Reality

• Over-Simplification

• Tunnel Vision

Mental Models in HCI

Interaction Design Principles
Mental Models vs.Conceptual Models
Gulf of Execution and Gulf of Evaluation (Norman, 1983)

​Elicitation of Mental Models

Elicitation approaches are typically categorized along two dimensions: (1) the technique used to extract thought processes, which can be verbal, visual, or a hybrid of both; and (2) the context of elicitation, either situated or non-situated [Jones et al., 2014].

Non-diegetic
Outside the main environment
Post-Interaction

Non-situated

Diegetic
Context-Specific Insights
Natural Interaction

​Situated

Context of Eliciting MMs

Thought Bubbles: A Proxy into Players’ Mental Model Development
Mohaddesi et al., 2023

The study introduces Thought Bubbles as a method for eliciting mental models in interactive virtual environments, specifically tested in a supply chain game with 250 participants. The findings of the study show that Thought Bubbles can reveal cognitive behavior by highlighting how disruptions and information availability shape players' mental models and decision-making based on their behavior profiles.
Theory: Situational Analysis
Inductive Coding

“It doesn’t just feel like something a lawyer slapped together.”Mental-Model-Based Privacy Policy for Third-Party Applications on Facebook
Paudel et al., 2023

The study conducted semi-structured interviews using drawings and think-aloud protocol with 32 participants. The participants had misconceptions regarding third-party applications’ data sharing practices with varied mental models. Based on these findings, researchers created mental model-based privacy policy design that prompts users to consider a specific scenario and provides information to help them understand their misconceptions. To evaluate designs, the study then conducted an online study with 26 participants over Amazon Mechanical Turk (MTurk). The results showed that using mental models helped users comprehend the message in the privacy policy, connect them to the design, and grabbed their attention.

Smart TV for Older Adults: A Comparative Study of the Mega Menu and Tiled Menu. Ouyang, X., Zhou, J. (2018)

This study examined the impact of different menu layouts on the user experience of older adults navigating smart TV (STV) interfaces. Navigation challenges are linked to interface design and users’ mental models. Two menu designs, mega menus and tiled menus with tags, were created and tested. Eye-tracking was used to assess layout effectiveness, and card sorting with path diagrams helped analyze users' mental models. Involving 30 older adults, the study found that while mega menus reduced physical effort, tiled menus led to greater user satisfaction. No significant difference was observed in mental model similarity between designers and older adults across both layouts. However, older adults with higher spatial ability had mental models more aligned with those of designers.

Spatial Memory Test

Test for spatial and visual memory, based on the Corsi Block Test.

https://www.memorylosstest.com/visual-span/

Informed Consent: https://forms.gle/JqHeTDWcLp3JP6uA6

Training for Closed Card-sorting: https://study.uxtweak.com/cardsort/jakGKF178hJtKXjMEF7pF

Training prior to card-sorting

Test for spatial and visual memory, based on the Corsi Block Test.

Evaluating User Mental Models in University Information Systems: A Case Study of YTU OBS

Chapter 6: Human Error
To err is human,
To err is to Play.

Imagine the perspective of a nurse working in a fast-paced emergency room, managing critical decisions while interacting with the electronic triage system. The NASA-TLX assessment captures the nurse's experience, giving us insights into the system's impact on workload and usability.

​

Draw

Please rate your experience on the following scales (0 = Low, 100 = High):

1. Mental Demand: __________

2. Physical Demand: __________

3. Temporal Demand: __________

4. Performance: __________

5. Effort: __________

6. Frustration Level: __________

Cognitive Load vs. Task Load

• Print vs. Interactive Web 2.0 educational materials

• YTU Starverse class (e.g., navigating the virtual environment, participating in group discussions, completing assignments).

Etymology

• Latin roots "errate" (a wandering or mistake), Old French influence "errer" (in a similar meaning)

• By the 14th century, "error" entered Middle English, keeping the dual meaning of a physical wandering and a deviation from correctness or truth.

• It came to primarily mean "a mistake" or "incorrect belief or action," a sense that has continued in modern English.

Equi-Availability Myths

Myth 1: If data is physically available, then its significance should be appreciated in all contexts.
Myth 2: If people demonstrate knowledge in some context, then that knowledge should be available in all contexts.

Woods et al., 2010
Behind Human Error

Latent errors occur at the blunt end.





Reason (1996)

​Blunt end

Active errors occur at the sharp end of the process.

​Sharp end

Anatomy of an Error

​Basic Premises for Research Human Error

• Loose coupling between process and outcome

• Erroneous actions and assessments should be taken as the starting point for an investigation not an ending

• Hindsight biases

• Incidents evolve through the conjunction of several failures/factors
  some of the contributing factors to incidents are latent in the system

  (Hollnagel, 1993; Woods et al., 2010)
  

Perspectives on Safety Management and Accident Analysis (Read et al., 2021)

​Perspective	​Typical conceptualisation of human-in-system behaviour	​Typical Unit of Analysis	​Error-as-cause	Safety-I or Safety-II
​Mechanistic	​Complicated	​Micro-the human	​Often	I
​Individual	​Complicated	​Micro-the human	​Often	I
​Interactionist	​Complicated	​Meso-the human,context-sometimes the organisation	​Often	I
​Systems	​Complex	​Macro	​-	I or II

Wording

• mode error (Sarter and Woods, 1995)

• slip/lapses of actions (Norman,1981; Reason and Mycielska, 1982)

• Hollnagel (1991;1993) calls this difference phenotye and geneotype and proposes the term "erroneous actions" (...there is no such thing as "human error"¹⁹⁹³).

• fixations or cognitive lockup (De Kreyser and Woods, 1990)

• vagabording (Dorner, 1993)

Human Error Methods & Taxonomies

Literature is full of extensions of methods and taxonomies.
*reliability and applicability issues

Retrospective vs. Prospective Methods/Integrated Methods

Domain specific vs. generic methods and taxonomies

The most influential researchers and theoreticians are Rasmussen, Hollnagel, Woods, Reason, Avizienis et al., and Norman.

system tailoring and task tailoring

​Theoretical Underpinnings of Methods in HCI

Schema Theory (Barlett, 1932): perception is dynamic, emphasizing interaction between knowledge and environmental information.
Perceptual Cycle Model (PCM) (Niesser, 1976): perception as an interaction with the environment rather than a purely internal cognitive process, foundational to HCI and design thinking.
Ecological Psychology (Gibson, 1979): perception as an interaction with the environment rather than a purely internal cognitive process, foundational to HCI and design thinking.
Distributed Cognition (Hutchins, 1995): expands cognition beyond the individual, considering it as distributed across people, tools, and environments.

Design Guidelines and Sources

1. Nielsen Norman Group (NNG) – [NNG Usability Heuristics](https://www.nngroup.com/articles/ten-usability-heuristics/)

2. ISO (International Organization for Standardization) – [ISO 9241-210](https://www.iso.org/standard/52075.html)

3. W3C (World Wide Web Consortium) – [WCAG Guidelines](https://www.w3.org/WAI/standards-guidelines/wcag/)

4. U.S. Department of Health and Human Services (HHS) – [HHS Usability Guidelines](https://www.usability.gov/)

5. Apple Human Interface Guidelines – [Apple HIG](https://developer.apple.com/design/human-interface-guidelines/)

6. Microsoft UX Design Guidelines – [Microsoft UX Design Guidelines](https://learn.microsoft.com/en-us/windows/uwp/design/)

7. U.S. National Institute of Standards and Technology (NIST) – [NIST Usability Resources](https://www.nist.gov/publications/usability-and-user-centered-design)
Additional Guidelines and Websites

1. NASA Human Factors Design Guidelines – [NASA Human Factors Design](https://humansystems.arc.nasa.gov/groups/HFES/)

2. The U.S. Department of Defense (DoD) Human Factors Engineering Program – [DoD Human Factors Engineering](https://www.acq.osd.mil/se/initiatives/humanfactors.html)

3. Health and Safety Executive (HSE) Guidelines – [HSE Guidelines](https://www.hse.gov.uk/humanfactors/)

Relevant Error Management Frameworks

Error Management in Human-Robot Interaction (HRI)

Honig and Oron-Gilad (2018) developed the RF-HIP (Robot Failure Human Information Processing) Model, which, while designed for HRI, is believed to be applicable to a broader field of HCI. The model has three main parts:

   Communicating Failures: Transmitting failure-related information through various channels (visual, audio, speech, etc.).

   Perceiving and Comprehending Failures: Ensuring the operator can perceive and understand relevant information.

   Solving Failures: Supporting operators' decision-making to resolve problems.

Human Error Management in HCI

Martinie et al. (2015) proposed a three-level framework for dealing with human errors in HCI:

  System Level: Monitoring the operator’s status to ensure activities remain within acceptable boundaries.

   Interaction Level: Preventing incorrect inputs by restricting input options to valid selections.

   Operator Level: Ensuring operators receive adequate training to avoid errors.

Model Touch Pointing and Detect Parkinson’s Disease via a Mobile

Game
Ling et al., 2024 -8 researchers from various fields

This study explores how people with Parkinson’s Disease (PD) use touch screens, aiming to (1) understand how PD affects touch pointing behaviors and (2) detect PD through these behaviors. To do this, researchers created a mobile game called MoleBuster, where users complete a series of touch-based pointing tasks. In the study with 40 participants, the study found that PD participants moved their fingers more slowly and had more variability in their movement times. Surprisingly, they also made fewer errors than non-PD participants of similar age, likely because they focused more on accuracy than speed. Additionally, the analysis showed that a model called the Finger-Fitts law was more accurate than the traditional Fitts' law for predicting how PD participants pointed, suggesting it could guide better mobile design for people with PD. The study also developed a CNN-Transformer neural network model to detect PD, using touch data and participants' comfort ratings for finger movement. This model achieved high accuracy, with an AUC of 0.97 and a sensitivity of 0.95 in a leave-one-user-out cross-validation.

*(Convolutional Neural Network - Evrişimli Sinir Ağı)

One of the First Fatalities of a Self-Driving Car: Root Cause Analysis of the 2016 Tesla Model S 70D Crash
Ergin, 2022

The recent technological developments have increased the prevalence of automated vehicles and vehicles with Advanced Driver Assistance Systems (ADAS) within the roadway traffic. Consequently, different safety-critical concerns rise for the usage of self-driving vehicles. The present study has investigated a crash between a semi-trailer of a Freightliner Truck and an SAE Level 2 automated Tesla Car. Operated during the autopilot mode engaged, the 40-year-old Tesla Driver hit and traveled under the mid aspect of the semi-trailer without taking any evasive actions prior to the crash and instantly deceased after the initial impact. The contributory factors of Human Error and Equipment Failure have been analyzed using specific tools of the root cause analysis: Five Whys Technique and Barrier Analysis respectively. The analysis has emphasized the importance of situational awareness while driving automated vehicles and showed that safety barrier features of ADAS may fail and should not be over-relied. The potential reasons to over-rely automated systems were discussed, and recommendations that target the safety of automated vehicle drivers have been made.

Misattribution of Error Origination: The Impact of Preconceived Expectations in Co-Operative Online Games
Minalovic and Pitt, 2021

This paper presented experimental results from a series of online games designed to investigate whether the theory of Artificial Social Constructivism, specifically the impact of training (education), could be effectively applied to current human computer relationships. It presented results that showed a tendency for players to doubt themselves and blame themselves for errors that occurred when playing with an AI player. This trend increased in likelihood as the difficulty of the game increased. The fact that even a minority of participants could be influenced to blame themselves and doubt their actions in a short period of time is a concerning trend. Therefore it is important to consider the extent of potential manipulation on human users. This is especially relevant given that individuals in the future will probably have prolonged exposure to artificial intelligence and smart devices. In the extreme, there is a risk that preconceived expectations of AI and misattribution of error attribution could be leveraged by designers (consciously or unconsciously) to create self-doubt in users and manipulate them to act against their own best interests.

Challenges and Gaps in Research

Taxonomy of HCI Errors: A need for a more systematic classification of HCI-relevant human errors, cognitive mechanisms, and contributing factors.

Validation of Countermeasures: Existing countermeasures for preventing human error lack sufficient validation, necessitating more rigorous evaluations.

 Human-Automation/AI Interaction (HAI): As automation and AI evolve, there is an increasing need for research into HAI, with particular attention to models, situation awareness, transparency, and trust in automated systems (Oswald et al., 2022).

​References

Jia, Z. (2024). Mental Models. In Foundations and Fundamentals in Human-Computer Interaction (pp. 151-166). Taylor & Francis.
She,M. et al. (2024). Human Error. In Foundations and Fundamentals in Human-Computer Interaction (pp. 167-214). Taylor & Francis.

See the "Bibtex" file in the drive folder for secondary and additional sources.

​Design Tools

Quizizz
Utweak

​Data for this PPT

Google Drive link that includes all relevant data and materials used in this PowerPoint presentation: https://drive.google.com/drive/folders/1uRSukV_WlIdWQsOB8_TtTFgam7vmK5zJ?usp=drive_link

Thank you for your attention!
Happy to hear your questions related to today’s topics on mental models and Human-Computer Interaction. Your insights and feedback are valuable, so please feel free to share!

​Q & A
Session