Development of Technology-Based Assessment (Part 2)

Development of TBA

• Analyse 🙋

  • Curriculum Mapping

  • Appropriate and Suitable Technology Tools

  • Theoretical Underpinning (SLA Theory)

• Develop

  • Item Development

  • Response Format

  • Scoring Rubric

  • Psychometric Validation of Item (Reliability & Validity)

​TSLB3533 | Kho Chung Wei

Success Criteria

By the end of this topic, I will be able to:

• Map the types of language assessment to the learning content and learning outcome in the curriculum

• Decide on the most appropriate and suitable technology tool(s) based on my curriculum mapping

• Explain the theoretical underpinning behind the TBLA that I have planned

• Develop the items, response format and scoring rubric for my TBLA

• Validate the psychometric properties of my TBLA

​TSLB3533 | Kho Chung Wei

Item Development (Assessment Development)

1. Develop an assessment task template

2. Create multiple assessment tasks from the template

3. Combine assessment tasks to make classroom-based assessments

• Target language use (TLU) domain: A specific context outside the assessment itself in which students need to perform language use tasks

• Target language use (TLU) task: A language use task that students may need to perform in one or more of their TLU domains

​TSLB3533 | Kho Chung Wei

Open Ended

State a possible TLU task that primary school students need to perform.

Developing an Assessment Task Template

1. Select a TLU task that might serve as a basis for developing assessment tasks.

2. Describe this TLU task.

3. Modify the TLU task to provide the basis for an assessment task template.

4. Develop a method for recording the students’ performance for the assessment task template.

5. Evaluate the assessment task template in terms of assessment qualities of meaningfulness, generalizability, impartiality, consistency, and practicality.

​TSLB3533 | Kho Chung Wei

Step 1: Selecting a TLU task

• Provide:

  • a short descriptive label for the TLU task

  • a short descriptive label for the area of language ability that is engaged by the TLU task

​TSLB3533 | Kho Chung Wei

Step 2: Describing TLU tasks

• Describe the TLU task(s) that you have identified in Step 1.

• Describe:

  • the activities and procedures to be followed

  • the Task Characteristics of the TLU task.

​TSLB3533 | Kho Chung Wei

Step 2: Describing TLU tasks

• Describe the TLU task(s) that you have identified in Step 1.

• Describe:

  • the activities and procedures to be followed

  • the Task Characteristics of the TLU task.

​TSLB3533 | Kho Chung Wei

​TCs: aspects or features of language use tasks that provide a way to describe the task with more precision than simply giving it a label

Step 3: Modifying a TLU Task

• Reasons:

  • The TLU task itself might not be practical to use as is as an assessment task.

  • The TLU task itself cannot be used as is because it does not enable us to arrive at assessment records for individual students.

  • It might be necessary to give the assessment tasks in a secure way, so that students cannot share their answers or information about the assessment with other students either prior to or during the assessment.

  • The TLU task itself cannot be used as is because of the students’ prior familiarity with the task’s topical content.

​TSLB3533 | Kho Chung Wei

Step 3: Modifying a TLU Task

• Modify the TLU task that you have described in Step 2.

• You will proceed as follows:

  1. TLU activities and procedures

     1. specify the reasons for making the modifications to the TLU task

     2. make the modifications to the TLU task.

  2. TLU TCs

     1. specify the reasons for making the modifications to the TLU TCs

     2. make the modifications to the TLU task TCs.

​TSLB3533 | Kho Chung Wei

Step 4: Developing a Method for Recording/Scoring Students’ Performance

• Specify the type of assessment record

• Specify a scoring method, the procedures for scoring, and the person who will score students’ assessment performance.

​TSLB3533 | Kho Chung Wei

Step 5: Evaluating the Assessment Task Template in terms of Assessment Qualities

• How well will the assessment tasks based on this template elicit the ability I want to assess? (meaningfulness)

• How closely will the assessment tasks based on this template correspond to the TLU task? (generalizability)

• How well will the assessment tasks based on this template avoid sources of bias? (impartiality)

• How consistent will the assessment records be? (consistency)

• How practical will the assessment tasks be to administer and score? (practicality)

​TSLB3533 | Kho Chung Wei

Multiple Choice

Can we decide on a student's language ability in a specific TLU domain through a single assessment task?

• Yes
• No

Open Ended

Why do we need multiple assessment tasks?

Reasons for Multiple Assessment Tasks

• A single assessment task...

  • is generally not sufficient for providing scores that are consistent

  • may not elicit enough performance for score-based interpretations to be meaningful

  • may not adequately sample the different tasks in the TLU domain

​TSLB3533 | Kho Chung Wei

Creating Multiple Assessment Tasks

1. Create one task following the assessment task template. That is, for each assessment task template, you will create one model assessment task that follows the assessment task template.

   • A model assessment task is an assessment task that is created following the specifications in an assessment task template. This assessment task then serves as a basis for creating additional assessment tasks.

2. Create additional tasks by:

   1. keeping the definition of the ability to be assessed the same

   2. keeping the attributes of the students (e.g. age, ability level, etc.) the same

   3. changing one or more of the TCs.

​TSLB3533 | Kho Chung Wei

Combining Assessment Tasks: Use a Blueprint

• Blueprint = a detailed plan that specifies the content and format of an assessment, and the procedures and instructions for administering an assessment

​TSLB3533 | Kho Chung Wei

Using Blueprint to Guide Assessment Development

• Maintain quality control

  • Ensure the assessment tasks you include in the assessment adequately sample both the content to be assessed and the tasks in the TLU domain, to assure that your interpretations will be meaningful and generalizable

  • Ensure the assessment will provide sufficient performance from your students to yield consistent assessment records

  • Ensure different forms of an assessment are equivalent in the content they assess and in the characteristics of the tasks they include.

​TSLB3533 | Kho Chung Wei

Using Blueprint to Guide Assessment Development

​TSLB3533 | Kho Chung Wei

Response Format

• Selected response

  • Students select one or more responses from among several possible responses that are given.

  • Typical tasks that elicit selected responses are multiple-choice and matching items.

  • Useful for testing specific areas of language knowledge, for example, grammatical or lexical knowledge, or specific aspects of listening and reading comprehension.

​TSLB3533 | Kho Chung Wei

Response Format

• Limited production response

  • Students’ responses can vary from a single word or phrase (oral or written), to a single sentence or utterance.

  • Typical tasks that elicit limited production responses are short answer and completion items.

​TSLB3533 | Kho Chung Wei

Response Format

• Extended production response

  • Students’ responses are longer than a single sentence or utterance, and can vary from two sentences to longer stretches of discourse, such as a written paragraph or essay, or, in speaking, a conversation or a speech.

  • Typical tasks that elicit extended production responses are those that are generally included in essay/composition exams or oral interviews.

​TSLB3533 | Kho Chung Wei

Scoring Methods

• Two main types of scoring methods:

  • The score consists of the number of tasks or items successfully completed.

  • The score is defined in terms of several levels on one or more rating scales of language ability.

​TSLB3533 | Kho Chung Wei

Multiple Select

• Assessment tasks with which response format can be scored by the number of tasks or items successfully completed?

• Selected response

• Limited production response

• Extended production response

Multiple Select

• Assessment tasks with which response format can be scored using several levels on one or more rating scales of language ability?

• Selected response

• Limited production response

• Extended production response

Scoring By Number of Tasks Successfully Completed

• The criteria for scoring will be specified in a scoring key.

  • Scoring key = the correct responses to assessment tasks and how many points each correct response counts

• For selected response items: 1. A, 2. C, 3. B (1 point per correct response)

• For limited production response items:

​TSLB3533 | Kho Chung Wei

Scoring as Levels of Language Ability

• The criteria for evaluating the quality of students’ responses are defined as several levels on one or more rating scales of language ability.

  • A rating scale (sometimes called a “scoring rubric”) specifies different levels on the ability to be assessed and provides descriptors for each level.

  • Two types of rating scales:

    • Global rating scales (“holistic” scale): the students’ performance is rated in terms of its general quality or level of ability; provides a single rating or score for a student’s assessment performance.

    • Analytic rating scales: the students’ performance is rated in terms of multiple components, each of which is rated on a separate scale; provides a “profile” of several different ratings or scores for a student's assessment performance.

​TSLB3533 | Kho Chung Wei

Multiple Choice

This is an example of...

• a global rating scale

• an analytic rating scale

Multiple Choice

This is an example of...

• a global rating scale

• an analytic rating scale

Practical Task (Previously)

• Based on the non-digital primary ESL classroom assessment instruments in Task 2 Product Development in EDUP3153 Assessment in Education, identify the learning outcomes and learning contents that are assessed in the instruments.

• Then, construct a curriculum map that contains the following:

  • Year level

  • Learning content

  • Learning outcome

  • Types of language assessment (Include all appropriate types of language assessment, not only the ones from the instruments)

  • Appropriate and suitable technology tool(s) to develop a TBLA

  • The theoretical underpinning of the proposed TBLA

​TSLB3533 | Kho Chung Wei

Practical Task (Continuation)

• Based on the curriculum map you have constructed:

  • Develop the assessment task template(s)

  • Create at least one assessment task from each template

    • If there is only one assessment task template, create at least two assessment tasks from the template.

  • Prepare a blueprint to combine the assessment tasks into a technology-based classroom assessment.

  • Prepare scoring keys and/or rubrics that are appropriate to the response format of the assessment tasks.

​TSLB3533 | Kho Chung Wei