Implement and Evaluate Technology-Based Assessment

Implement and Evaluate Technology-Based Assessment

• Implement

  • Administration

  • Collection

• Test Reporting

• Interpretation of Result

​TSLB3533 | Kho Chung Wei

Success Criteria

By the end of this topic, I will be able to:

• Administer the technology-based language assessment that I have developed

• Collect assessment data from a sample of the target population

• Report the assessment data in terms of the pupils' performance and psychometric properties of the assessment

• Interpret the assessment results

​TSLB3533 | Kho Chung Wei

Practical Task (4 hours)

1. Find a sample of the target population which your TBLA was designed for.

2. Administer the TBLA to the sample either in person or through a proxy (e.g. the English teacher in school) following the administration procedure you have developed.

3. Collect the responses from the sample.

4. Score the responses.

​TSLB3533 | Kho Chung Wei

Poll

Which type of response does your TBLA require?

• Selected response

• Limited production response

• Extended production response

Word Cloud

What should you include when reporting the assessment results to individual pupils (and their parents)?

Reporting to Stakeholders: Individual Pupil/Parent

• For selected / limited production response items, report:

  • The score for each item

  • Explanation if the items are scored less than the maximum score

• For extended production response items, report:

  • The profile of analytic ratings

  • Description of the ratings given as applied to the pupil's response

• For all types of assessment, report:

  • The composite score (as raw score / part of total possible score / percentage)

  • Suggestion for improvement

​TSLB3533 | Kho Chung Wei

Word Cloud

What should you include when reporting the assessment data to the school administrator?

Reporting to Stakeholders: School Administrators

• Overall performance of the pupils:

  • Visual representation - score distribution

  • Measures of central tendency = mean, mode, median

  • Measures of variability = standard deviation, range, interquartile range

  • Interpretation of the pupils' performance (where they are, what's the next step)

• Psychometric properties of the assessment:

  • Reliability

  • Validity

  • Interpretation of the psychometric properties (what does it mean)

​TSLB3533 | Kho Chung Wei

Multiple Choice

Is it possible to have high validity and low reliability?

Think about if it is possible to make a valid inference about someone's weight using an unreliable measuring scale.

• Yes

• No

Multiple Choice

Which of the following represents high reliability and high validity?

Psychometric Property: Reliability

• The extent to which assessments are consistent

• The consistency of the results we obtain from an assessment

• Factors affecting reliability:

  • The number of tasks / items

  • The difficulty level of the tasks / items

  • The spread of scores produced by the assessment

  • The training of the assessors

  • Clarity of scoring key / scoring rubrics and marking procedures

  • Clarity of tasks / items

  • Affective factor of the pupils (e.g. tiredness, anxiety)

​TSLB3533 | Kho Chung Wei

Psychometric Property: Reliability

• Determining reliability:

  • Consistency across time —> Test/retest reliability

  • Consistency across tasks —> Internal consistency (split half, Cronbach alpha)

  • Consistency across markers —> Inter-rater reliability

• Reliability coefficient: between 0 and 1 (Nunnally, 1967)

​TSLB3533 | Kho Chung Wei

Psychometric Property: Reliability

• For classroom assessment, reliability = sufficiency of information

  • “Does this assessment provide me with enough information to make a judgement of each student’s level of accomplishment with regard to this learning?” (Smith, 2003, p. 26)

  • "A wide range of assessments can serve the purpose of a long test—the more sources of information, with demonstrable evidence for validity, the

    more likely dependable decisions can be made” (Taylor & Nolan, 1996, p. 11)

  • Triangulation = using different sources of assessment evidence as the basis for any decision making (Davies, 2000)

​TSLB3533 | Kho Chung Wei

Psychometric Property: Validity

• The accuracy of an assessment — whether it measures what it is supposed to

• Should not be considered as a fixed property of an assessment instrument

• Evaluation of the quality of the interpretations and decisions that are made on the basis of an assessment result — that is, how well the inferences we make or actions we take on the basis of an assessment result can be justified

​TSLB3533 | Kho Chung Wei

Psychometric Property: Validity

• Four major considerations for validity argument:

  • Content considerations —> Evaluating how well the assessment tasks represent or sample the learning domain

  • Construct considerations —> Examining the extent to which the assessment result can be used to make inferences about the existence of a certain construct (specific psychological characteristics / traits)

  • Criterion considerations —> Looking at how well the assessment results compare with or predict other measures recorded on a separate assessment or criterion

  • Consequential considerations —> Evaluating the consequences of using the assessment results

​TSLB3533 | Kho Chung Wei

Psychometric Property: Validity

• Do the tasks match the learning intentions we are interested in?

• Does the test cover a wide enough range of content?

• Are there enough items or tasks to cover the scope of what is being assessed

• Do the tasks require use of the desired skills and reasoning processes?

• Is there an emphasis on deep, rather than surface knowledge?

• Are the directions for the assessment task clear?

• Are the questions unambiguous?

• Are the time limits sufficient?

• Do the tasks avoid favouring groups of students more likely to have useful background knowledge—for instance, boys or girls?

• Is the language used suitable?

• Are the reading demands fair?

​TSLB3533 | Kho Chung Wei

e-Learning Task (2 hours)

1. Read the materials attached under this topic.

2. Make your own notes.

3. Do your own Internet search on:

   1. how to visually present the score distribution

   2. how to calculate the measures of central tendency, and the measures of variability

   3. how to calculate the different reliability coefficients.

​TSLB3533 | Kho Chung Wei

Tutorial Task (2 hours)

1. Based on the assessment data that you have collected using the TBLA that you have developed:

   1. Write a report of the assessment data for the individual pupils in your sample

   2. Write a report of the assessment data for the school administrator

2. Make sure that you include an interpretation of the assessment results in your report.

​TSLB3533 | Kho Chung Wei