Design Aspects of Technology-Based Assessment

Design Aspects of TBA

• Design Model and Principles

  • ADDIE

  • Bloom's Digital Taxonomy

  • TPACK

  • SAMR

• Guidelines and Features

  • Authentic

  • Reliable

  • Valid

  • Transparent

  • Feasible

  • Fairness

​TSLB3533 | Kho Chung Wei

• ​Dimension of Assessment

  • Item Development

  • Scoring

  • Administration

  • Reporting

  • Interpretation

Success Criteria

By the end of this topic, I will be able to:

• Apply the design model and principles in developing my own TBLA

• Explain the features in my own TBLA

• Analyse the dimension of assessment in my own TBLA

​TSLB3533 | Kho Chung Wei

Design Model and Principles

ADDIE

• Instructional design model

• Created by Florida State University for the military in the 1970s

• Hierarchical and linear model --> more interactive and dynamic

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Analyse

• Goal-setting stage, needs analysis

• Addresses the following questions:

  • What is the typical background of the learners?

  • What do the learners need to accomplish at the end?

  • What are the desired learning outcomes in terms of knowledge, skills etc.?

  • Review existing instructional strategies employed. Are they adequate?

  • What instructional goals does it focus on?

  • Determine the various options available in terms of learning environment.

  • Determine limiting factors to the overall goal of the project.

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Design

• Overview of the design, storyboards, prototypes

• Determine the following:

  • Types of media to be used, available resources, user interface

  • Level and types of activity to be generated during the study

  • Pedagogical approach (e.g. behaviorist, constructivist)

  • Time frame for each activity

  • Mental processes. knowledge and skill developed after each task

  • The roadmap of how the study or project will appear on paper

  • Learners' feedback mechanism

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Development

• Drafting, production and pilot-testing

• Address the following questions:

  • Is the time frame being adhered to in relation to what has been accomplished in terms of material? Are you creating materials as per schedule?

  • Are the materials produced up to task on what they were intended for?

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Implementation

• Share with learners

• Consider the following:

  1. Record keeping, the actual data

  2. Affective feedback given by teachers and students

  3. Suitability to learners

  4. Ways to deal with any possible errors

  5. Back-up strategy

  6. A small scale, or a large scale implementation

  7. Learners working independently, or with constant guidance

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Evaluation

• Formative (at every stage) vs Summative (at the end)

• Determine the following:

  1. Factors or criteria to measure effectiveness

  2. Data collection - when and how

  3. A system for analysing learners' feedback and responses

  4. Basis to revise certain aspects

  5. Method by which reliability and content validity can be observed

  6. Method to measure clarity of instructions

  7. The target audience of the evaluation report

​TSLB3533 | Kho Chung Wei

Open Ended

Briefly explain how you would apply the ADDIE model in your technology-based language assessment project.

Design Model and Principles

Bloom's Digital Taxonomy

• Purpose: To inform teachers of how to use technology and digital tools to facilitate student learning experiences and outcomes

• Aim: To expand upon the skills associated with each level as technology becomes a more ingrained essential part of learning

• Focus on how the tools can act as vehicles for transforming student thinking at different levels, not on the tools themselves

• Created by Andrew Churches in 2008

• Add verbs that address forms of learning and creating that reflect the digital age to the Bloom's revised taxonomy

​TSLB3533 | Kho Chung Wei

Design Model and Principles

​TSLB3533 | Kho Chung Wei

Open Ended

Briefly explain how the Bloom's digital taxonomy is useful in your technology-based language assessment project.

Design Model and Principles

TPACK

• Proposed by Punya Mishra and Matthew J. Koehler in 2006

• Based on Lee S. Shulman’s (1986) pedagogical content knowledge

• Outlines how content (what is being taught) and pedagogy (how the teacher imparts that content) must form the foundation for any effective educational technology integration

• Specific technological tools (hardware, software, applications, associated information literacy practices, etc.) are best used to instruct and guide students toward a better, more robust understanding of the subject matter

​TSLB3533 | Kho Chung Wei

Design Model and Principles

TPACK

• Effective implementation of technology in the classroom acknowledges the dynamic, transactional relationship among content, pedagogy, and the incoming technology.

• The unique contexts of different schools, classrooms, and cultures mean that every situation will demand a slightly different approach to edtech integration.

• No one combination is applicable in every setting.

​TSLB3533 | Kho Chung Wei

Design Model and Principles

TPACK

• CK = teachers’ own knowledge of the subject matter

• PK = teachers’ knowledge of the practices, processes, and methods regarding teaching and learning

• TK = teachers’ knowledge of, and ability to use, various technologies, technological tools, and associated resources

​TSLB3533 | Kho Chung Wei

Design Model and Principles

TPACK

• TCK = teachers’ understanding of how the subject matter can be communicated via different edtech

• TPK = teachers’ understanding of how particular technologies can change the teaching and learning experiences by introducing new pedagogical affordances and constraints

​TSLB3533 | Kho Chung Wei

Design Model and Principles

TPACK

• PCK = teachers’ understanding of how teaching-learning strategies and assessment methods can be suited to the subject matter

• TPACK = teachers' understanding of of how to teach their students a subject matter effectively using technology

​TSLB3533 | Kho Chung Wei

Open Ended

Based on the TPACK framework, what can you conclude about using technology in language assessment? What should be the focus when developing a technology-based language assessment?

Design Model and Principles

SAMR

• Developed by Ruben Puentedura in 2010 based on research into how the use of digital technology was transforming classroom-based teaching and learning

• A spectrum that lays out four tiers of online learning, presented roughly in order of their sophistication and transformative power

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Substitution

• Replacing traditional activities and materials with digital versions

• No substantial change to the content, just the way that it is delivered

• Goal: to keep things simple - no need to reinvent the wheel

• Helps to provide synchronous and asynchronous versions

Augmentation

• Incorporating interactive digital enhancements and elements like comments, hyperlinks, or multimedia

• Content remains unchanged, but students can now take advantage of digital features to enhance the lesson

​TSLB3533 | Kho Chung Wei

Design Model and Principles

Modification

• Using a learning management system to handle the logistical aspects of running a classroom, like tracking grades, messaging students, creating a calendar, and posting assignments

• Opens up new channels of communication, which can help students who have traditionally been marginalized to participate

Redefinition

• Fundamentally transformed learning, enabling activities that were previously impossible in the classroom

• Bring authentic audiences into virtual classroom

• Allow students to tackle real-world problem

​TSLB3533 | Kho Chung Wei

Multiple Choice

A teacher scans worksheets, converts them into PDFs, and post them online. She also delivers the content of her lesson via Google Meet and later shares the recording with students who could not attend the class.

This is an example of...

• Substitution

• Augmentation

• Modification

• Redefinition

Multiple Choice

• A teacher gets her students to go on a virtual field trip to the places that were mentioned in a story that they have read. She also connects them with other students from those places via email exchanges. She then invites the author of the story to join a Google Meet session to chat about his work and answer questions.

This is an example of...

• Substitution

• Augmentation

• Modification

• Redefinition

Multiple Choice

A teacher uses the online forum function in Google Classroom to discuss issues related to the topic learnt. During the class via Google Meet, she gets her students to respond using the poll function, ask questions in the text chat and respond via the reaction feature.

This is an example of...

• Substitution

• Augmentation

• Modification

• Redefinition

Multiple Choice

A teacher gamifies her quizzes using platforms like Socrative and Kahoot. She also creates virtual bulletin boards using Padlet which allows students to post comments, questions, links and pictures. She also allows her students to present using digital portfolios. 

This is an example of...

• Substitution

• Augmentation

• Modification

• Redefinition

Poll

Think about your technology-based language assessment project. Which tier of the SAMR model are you targeting?

• Substitution

• Augmentation

• Modification

• Redefinition

e-Learning Task

• Item Development

• Scoring

• Administration

• Reporting

• Interpretation

• Authentic

• Reliable

• Valid

• Transparent

• Feasible

• Fairness

Read up on the following subtopics in Bachman and Dambock (2016), Hong Jiao and Lissitz (2017), Brown (2013) and International Test Commission and the Association of Test Publishers (2022):

We'll play a game related to these subtopics in the next class.

​​TSLB3533 | Kho Chung Wei