Validity and Reliability & Survey Research

Validity & Reliability

Survey Research

Methods of Research

Validity & Reliability

Objectives

• Distinguish between validity and reliability.

• List the major types of evidence used to support the valid interpretation of test scores

• Define construct underrepresentation and construct-irrelevant variance and explain their relevance to the validity of test scores

• Distinguish between convergent and discriminant evidence of validity

• Distinguish between random and systematic errors of measurement and their relationship to validity and reliability of test scores.

Definition of Validity

• Validity was defined as the extent to which an instrument measured what it claimed to measure

• Standards for Educational and Psychological Testing (1999) defines validity as “the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests”

Messick (1995) identified two problems that threaten the interpretation (validity) of test scores:

• Construct underrepresentation refers to assessment that is too narrow and fails to include important dimensions of the construct.

• Construct-irrelevant variance refers to the extent to which test scores are affected by variables that are extraneous to the construct

VALIDATION

The process of gathering evidence to support (or fail to support) a particular interpretation of test scores is referred to as validation. 

The Standards for Educational and Psychological Testing lists three categories of evidence used to establish the validity of score-based interpretations: evidence based on content, evidence-based on relations to a criterion, and construct-related evidence of validity.

Multiple Choice

It involves the test’s content and its relationship to the construct it is intended to measure.

• Evidence Based on Relations to a Criterion

• Evidence Based on Test Content

• Construct-Related Evidence of Validity

Evidence Based on Test Content

The Standards defines content-related evidence as “The degree to which the sample of items, tasks, or questions on a test are representative of some defined universe or domain of content.” 

Evidence-based on test content is especially important in evaluating achievement tests.

Multiple Choice

It refers to the extent to which test scores are systematically related to one or more outcome criteria.

• Evidence Based on Relations to a Criterion

• Evidence Based on Test Content

• Construct-Related Evidence of Validity

Types of Criterion-Related Validity Evidence

• Concurrent validity evidence is the relationship between scores on a measure and criterion scores obtained at the same time.

• Predictive validity evidence is the relationship between scores on a measure and criterion scores available at a future time.

Choosing the Criterion

• The worth of the entire procedure depends first and foremost on the relevance of the criterion.

• The criterion must also first be reliable. 

• The criterion should also be free from bias.

Validity Coefficient

The coefficient of correlation between test scores and criterion is called a validity coefficient (r_xy). 

Validity coefficients indicate whether the test will be useful as a predictor or as a substitute measure.

The nearer the coefficient is to 1.00 (+ or −), the stronger the evidence is that the test is useful for the stated purpose.

Multiple Choice

It focuses on test scores as a measure of a psychological construct.

• Evidence Based on Relations to a Criterion

• Evidence Based on Test Content

• Construct-Related Evidence of Validity

Strategies Used to Gather Construct-related Evidence

• Related measures studies

• Known-groups technique

• Intervention studies

• Internal structure studies

• Studies of response processes

Related Measures Studies

The aim is to show that the test in question measures the construct it was designed to measure and not some other theoretically unrelated construct.

“Relationships between test scores and other measures intended to assess similar constructs provide convergent evidence, whereas relationships between test scores and measures purportedly of different constructs provide discriminant evidence” (Standards, p. 14).  

Other Strategies

Known-groups technique: The researcher compares the performance of two groups already known to differ on the construct being measured. 

Intervention studies: To apply an experimental manipulation and determine if the scores change in the hypothesized way.

Internal structure studies: The test is measuring the construct it is supposed to be measuring. 

Studies of response processes: To look at the evidence-based response processes of individuals actually taking the test.

VALIDITY OF CRITERION-REFERENCED TESTS

The basic approach to determining content validity is to have teachers or subject matter experts examine the test and judge whether it is an adequate sample of the content and objectives to be measured.

Another approach that has been used is to administer the test and divide the examinees into two groups: masters versus nonmasters.

Multiple Choice

Identify the type of validity evidence (content, concurrent criterion, predictive criterion, or construct related) being gathered.

1. A test administered to applicants for law school correlates .65 with first semester grades in law school.

• content related

• concurrent criterion

• construct related

• predictive criterion

Multiple Choice

Identify the type of validity evidence (content, concurrent criterion, predictive criterion, or construct related) being gathered.

2. A high school teacher administers a standardized chemistry test and correlates the scores with the scores that students earned the next day on a teacher-made chemistry test.

• content related

• concurrent criterion

• construct related

• predictive criterion

RELIABILITY

The reliability of a measuring instrument is the degree of consistency with which it measures whatever it is measuring.

On a theoretical level, reliability is concerned with the effect of error on the consistency of scores. There are two kinds of errors: random errors of measurement and systematic errors of measurement. Random error is error that is a result of pure chance. Systematic errors, on the other hand, inflate or depress scores of identifiable groups in a predictable way. Systematic errors are the root of validity problems; random errors are the root of reliability problems.

SOURCES OF RANDOM ERROR

• The individual being measured may be a source of error

• The administration of the measuring instrument may introduce error

• The instrument may be a source of error

APPROACHES TO RELIABILITY

• The first method indicates the amount of variation to be expected within a set of repeated measurements of a single individual.

• The consistency of a set of scores is also indicated by the extent to which each individual maintains the same relative position in the group. 

VALIDITY AND RELIABILITY COMPARED

Open Ended

What is a term/phrase that you can relate to RELIABILITY AND VALIDITY?

SURVEY RESEARCH

OBJECTIVES

• Define the terms related to survey research

• Describe the four categories of surveys classified according to their scope and focus.

• Distinguish between longitudinal and cross-sectional surveys.

• State the merits and disadvantages of the interview as a data-gathering technique

• Describe data-gathering techniques used in survey research.

Multiple Select

Which of the following questions would best be answered by survey methods?

• Do voters in our school district think we should raise taxes in order to build new classrooms?

• Do people who have taken driver education have fewer accidents than people who have not?

• What do voters consider the most important issues in the upcoming election?

• What do school principals consider to be the major problems in their schools?

• Does dividing second-grade math students into ability groups produce greater math achievement than doing math instruction in a single group?

TYPES OF SURVEYS

• SURVEYS CLASSIFIED ACCORDING TO FOCUS AND SCOPE

• SURVEYS CLASSIFIED ACCORDING TO THE TIME DIMENSION

SURVEYS CLASSIFIED ACCORDING TO FOCUS AND SCOPE

Population - refers to the entire group of individuals to whom the findings of a study apply. A survey that covers the entire population of interest is referred to as a census

Sample - smaller portion of the population. A survey that studies only a portion of the population is known as a sample survey

SURVEYS CLASSIFIED ACCORDING TO FOCUS AND SCOPE

Tangibles, such as what proportion of children ride school buses and the average class enrollment.

Intangibles, such as attitudes, opinions, values, or other psychological and sociological constructs.

If you classify surveys on the basis of their scope (census versus sample) and their focus (tangibles versus intangibles), four categories emerge: (1) a census of tangibles, (2) a census of intangibles, (3) a sample survey of tangibles, and (4) a sample survey of intangibles.

SURVEYS CLASSIFIED ACCORDING TO FOCUS AND SCOPE

• A Census of Tangibles - When you seek information about a small population, such as a single school, and when the variables involved are concrete, there is little challenge in finding the required answers.

• A Census of Intangibles - The task will be more difficult because this census deals with constructs that are not directly observable but must be inferred from indirect measures. 

SURVEYS CLASSIFIED ACCORDING TO FOCUS AND SCOPE

• A Sample Survey of Tangibles - When investigators seek information about large groups, the expense involved in carrying out a census is often prohibitive.

• A Sample Survey of Intangibles - The public opinion polls are examples of studies measuring intangible constructs. Opinion is not directly observable but must be inferred from responses made by the subjects to questionnaires or interviews. 

Multiple Choice

What type of survey is illustrated in the following examples?

1. Randomly selected teachers are asked how many years experience they have teaching?

• Census of tangibles

• Census of intangibles

• Sample survey of tangibles

• Sample survey of intangibles

Multiple Choice

What type of survey is illustrated in the following examples?

2. The superintendent of School District 214 has Iowa Test of Basic Skills scores for all second-graders in his district.

• Census of tangibles

• Census of intangibles

• Sample survey of tangibles

• Sample survey of intangibles

Multiple Choice

What type of survey is illustrated in the following examples?

3. The state superintendent of schools’ list of the enrollment in each of the state’s 1143 schools.

• Census of tangibles

• Census of intangibles

• Sample survey of tangibles

• Sample survey of intangibles

Multiple Choice

What type of survey is illustrated in the following examples?

4. Some students in Shaw School are given a physical fitness test to get an estimate of the fitness of all the students in the school.

• Census of tangibles

• Census of intangibles

• Sample survey of tangibles

• Sample survey of intangibles

SURVEYS CLASSIFIED ACCORDING TO THE TIME DIMENSION

• Longitudinal Surveys - gather information at different points in time in order to study the changes over extended periods of time. Three different designs are used in longitudinal survey research: panel studies, trend studies, and cohort research.

• Cross-sectional surveys study a cross section (sample) of a population at a single point in time. 

SIX BASIC STEPS INVOLVED IN SURVEY RESEARCH

• Planning

• Defining the population

• Sampling

• Constructing the instrument

• Conducting the survey

• Processing the data

STANDARD ERROR OF THE SAMPLING PROPORTION

CALCULATING THE STANDARD ERROR

STANDARD ERROR OF THE SAMPLING PROPORTION

CALCULATING SAMPLE SIZE

CONSTRUCTING THE INSTRUMENT

FORMAT OF QUESTIONS: Open-ended questions or Close-ended questions?

STRUCTURE OF QUESTIONS:

1. Completion, or fill-in, items

2. Checklists

3. Scaled items

4. Ranking items

5. Likert-type items

CONSTRUCTING THE INSTRUMENT

WRITING SURVEY QUESTIONS

1. Questions should be short, simple, and direct.

2. Phrase questions so that they can be understood by every respondent.

3. Phrase questions so as to elicit unambiguous answers.

4. Phrase questions so as to avoid bias that may predetermine a respondent’s answer.

5. Avoid questions that might mislead because of unstated assumptions.

6. Avoid leading questions, which imply the desired response.

CONSTRUCTING THE INSTRUMENT

WRITING SURVEY QUESTIONS

7. Avoid questions that may elicit embarrassment, suspicion, or hostility in the respondent.

8. Avoid “double-barreled” questions, which attempt to ask two questions in one.

9. Make sure the alternatives to each questionnaire item are exhaustive—that they express all the possible alternatives on the issue.

10. Keep the questionnaire as brief as possible so that it requires a minimum of the respondents’ time.

11. Make sure the respondents have the information necessary to answer the questions.

STATISTICAL ANALYSIS IN SURVEYS

CONTROLLING VARIABLES IN A SURVEY ANALYSIS

STATISTICAL ANALYSIS IN SURVEYS

STATISTICS FOR CROSSTABS

Sample Study

Teaching and Learning in Higher Education: Perspectives of New Millenium Higher Education Learners (NML)

by Merlita C. Medallon

ISSN 2094-8905 (Print)

Ask and you shall know.

Thank you for participating! :)