Methodology: Quantitative

Preparing a Research Proposal: Methodology (Quantitative)

By KHO CHUNG WEI

In this topic, you will:

• Discuss research designs using the quantitative approach

• Determine the most appropriate sampling technique and sampling size for a research problem

• Identify the most suitable research instrument for a research study

• Explain how to conduct a pilot study

• Explain how to ensure validity and reliability in a research study

• Explain the most appropriate data collection procedure

• Identify the different data analysis methods

TSLB3353

Fundamentals of Educational Research in ESL

Multiple Select

Choose the research designs that use the quantitative approach.

• Experimental

• Survey

• Case study

• Ethnography

• Action research

Quantitative: Research design

​Research design = the outline, plan, or strategy that is used to answer RQ​

TSLB3143

Fundamentals of Educational Research in ESL

Quantitative: Research design

TSLB3143

Fundamentals of Educational Research in ESL

Quantitative: Sampling

TSLB3353

Fundamentals of Educational Research in ESL

​Population (target population) = the set of all elements

Sample = a set of elements taken from the population

Elements = basic units in a population​

N = population size

n = sample size​

Parameter = ​numerical characteristic of population

Statistic = numerical characteristic of a sample​

Sampling error = difference between the value of a sample statistic and the population parameter​

​Sampling frame = a list of all the elements in the population

Response rate = ​Percentage of elements in a sample who participate in a research study (≥70%)

Quantitative: Sampling

TSLB3353

Fundamentals of Educational Research in ESL

Random ​Sampling Techniques (Random Selection)

• Simple random sampling

  • ​Equal probability of selection method (EPSEM)

• Systematic sampling

  • ​Sampling interval, k = N/n

  • Starting point = a randomly selected number between 1 and k

  • Beware of periodicity (presence of cyclical pattern in the sampling frame)

• Stratified random sampling

  • Stratification variable ​= the variable on which the population is divided

• Proportional stratified sampling

• Disproportional stratified sampling​

• ​Cluster sampling (one-stage / two-stage / probability proportional to size)

Open Ended

Why do we use a random sampling technique?

Quantitative: Sampling

TSLB3353

Fundamentals of Educational Research in ESL

Random Assignment

• Randomly assigning the elements in the sample to different groups

• Purpose: to produce comparison groups that are similar on all possible factors at the beginning of the experiment

Quantitative: Sampling

TSLB3353

Fundamentals of Educational Research in ESL

Nonrandom ​Sampling Techniques

• Convenience sampling

  • ​those who are available / volunteer / can easily be recruited

• Quota sampling

  • ​determine the appropriate sample sizes (quotas) for the groups identified as important and takes convenience samples from those groups

• Purposive sampling

  • ​specify the characteristics of the population of interest and locate individuals with those characteristics

• Snowball sampling

  • ​each research participant is asked to identify other research participants

Multiple Choice

Tehnically speaking, can we generalise the results from a nonrandom sample to the population?

• Yes

• No

Sampling

TSLB3143

Fundamentals of Educational Research in ESL

Sampling Size

• ​Also depends on efficiency of sampling techniques:

  ​Stratified < Simple < Cluster

Fill in the Blanks

Sampling

TSLB3353

Fundamentals of Educational Research in ESL

Sampling Size

• ​Also depends on the data analysis procedures and the expected effect size

Quantitative: Research instruments

TSLB3353

Fundamentals of Educational Research in ESL

1. Test

2. Questionnaire

3. ​Interview protocol (mostly close-ended)

4. ​Observation checklist (more specific and detailed)

​* You may also modify existing instruments with permission from the authors.

Task 1

TSLB3353

Fundamentals of Educational Research in ESL

Recently, the English Unit conducted an English Enhancement Programme with the aim of enhancing TESL students' English proficiency level to at least a C1 (based on the CEFR scale). The programme was carried out as a seminar and a workshop. However, due to time constraint, the workshop was only made available to some students. All the students however have attended the seminar.

​

As part of an evaluation research, you would like to find out the participants' opinions and experience of the programme, the knowledge they have gained from the programme and their behaviours towards enhancing their English proficiency level after attending the programme.

​

Write three close-ended items and two open-ended items for the questionnaire that you will use to collect data for the research study.

Quantitative: Research instruments

TSLB3353

Fundamentals of Educational Research in ESL

Principles of Questionnaire Construction

1. ​Make sure the questionnaire items match the research objectives.

2. ​Understand your research participants.

3. ​Use natural and familiar language.

4. ​Write items that are clear, precise, and relatively short.

5. ​Do not use "leading" (suggest certain answer) or "loaded" (emotionally charged words) questions.

6. ​Avoid double-barreled (combine two or more issues/attitudes) questions.

7. ​Avoid double negatives (two negatives).

8. ​Determine whether an open-ended or close-ended question is needed.

9. ​Use mutually exclusive and exhaustive response categories for close-ended questions.

Quantitative: Research instruments

TSLB3353

Fundamentals of Educational Research in ESL

Principles of Questionnaire Construction

10. ​Consider the different types of response categories available for close-ended questionnaire items.

11. Use multiple items to measure abstract constructs.

12. Consider using multiple methods when measuring abstract constructs.

13. ​Use caution if you reverse the wording in some of the items to prevent response sets in multi-item scales.

14. Develop a questionnaire that is properly organised and easy for the participants to use.

15. Always pilot test your questionnaire.​

Quantitative: Research instruments

TSLB3353

Fundamentals of Educational Research in ESL

Quantitative: Research instruments

TSLB3353

Fundamentals of Educational Research in ESL

Quantitative: Pilot Study

TSLB3353

Fundamentals of Educational Research in ESL

• ​Pilot study = a preliminary test of the research instrument

• 5-10 individuals similar to your sample

• ​Ask them to fill out the research instrument and note any points of confusion

• ​Use think aloud technique

  • participants verbalise their thoughts and perceptions while filling out the instrument

  • Audio tape / video record / carefully write down ​what they say

  • Helpful to determine whether participants are interpreting the items the way you intended

• Also administer to others under conditions similar to actual study

Quantitative: Pilot Study

TSLB3353

Fundamentals of Educational Research in ESL

Check for:

• ​the duration it takes to complete the instrument

• confusing / threatening items

• consistent understanding of the items

After participants have completed the instrument:

• Discuss the instrument with them individually or in groups

• Explain the purpose of the instrument and ask if anything was left out

• Check if the instructions are clear and if any items stood out

• Ask them to comment on the appearance and clarity

• Check if there are too many "I don't know", "Neutral", "Does not apply"​

​Revise & pilot test again until all the issues are ironed out

Quantitative: Reliability & Validity

TSLB3353

Fundamentals of Educational Research in ESL

​Reliability = consistency or stability of the scores

​Validity = accuracy/appropriateness of the inferences or interpretations you make from the scores

Systematic error = an error that is present every time an instrument is used

​

Multiple Select

Which of the following statements are True? Choose 3 statements.

• Reliability is a necessary condition for validity.

• Validity is a necessary condition for reliability.

• Reliability does not guarantee validity.

• Scores that are valid may or may not be reliable.

• Scores that are reliable may or may not be valid.

Quantitative: Reliability

TSLB3353

Fundamentals of Educational Research in ESL

Reliability coefficient = ​correlation coefficient that is used as an index of reliability

​​Maximum reliability coefficient = +1.00 (perfect reliability)

Reliability coefficient​ ≤ 0 (no reliability at all)

​

Quantitative: Reliability

TSLB3353

Fundamentals of Educational Research in ESL

• Test-retest reliability = a measure of consistency of scores over time

• ​Equivalent-forms reliability = the consistency of a group of individuals' scores on alternative forms of a test measuring the same thing

• ​Internal consistency = the consistency with which the items measure a single construct​

  • Split-half reliability = ​a measure of the consistency of scores obtained from two equivalent halves of the same test (Spearman-Brown formula)

  • Coefficient alpha = a reliability estimate that can be thought of as the average of all possible split-half correlations

• Interscorer reliability = the degree of agreement or consistency between 2 or more scorers, judges or raters

Quantitative: Validity

TSLB3353

Fundamentals of Research in Education

• Validity evidence = empirical evidence and theoretical rationales that support the inferences or interpretations made from scores

Quantitative: Validity

TSLB3353

Fundamentals of Educational Research in ESL

• Content-related evidence = validity evidence based on a judgment of the degree to which the items, tasks, or questions on a test adequately represent the construct domain of interest

  Ask:

1. Do the items appear to represent the thing you are trying to measure?

2. Does the set of items underrepresent the construct’s content (i.e., have you excluded any important content areas or topics)?

3. Do any of the items represent something other than what you are trying to measure (i.e., have you included any irrelevant items)?​

Quantitative: Validity

TSLB3353

Fundamentals of Educational Research in ESL

• Evidence based on internal structure

1. ​Determine how many dimensions or constructs the instrument measure using factor analysis ( = a statistical procedure that analyzes correlations among items and tells you the number of factors present, i.e. unidimensional or multidimensional instrument)

2. ​Examine the homogeneity of the items ( = how well the different items measure the same construct)

   1. ​item-to-total correlation

   2. coefficient alpha

Quantitative: Validity

TSLB3353

Fundamentals of Educational Research in ESL

• Evidence based on relations to other variables

  • Criterion-related evidence = validity evidence based on the extent to which scores from an instrument can be used to predict or infer performance on some criterion such as a test or future performance

    • ​Concurrent evidence = criterion scores obtained at the same time

    • Predictive evidence = criterion scores obtained at a later time

  • Convergent evidence = based on the relationship between focal scores and other independent measures of the same construct

  • Discriminant evidence = focal scores are not highly related to scores from other instruments that are designed to measure different constructs

Quantitative: Validity

TSLB3353

Fundamentals of Educational Research in ESL

• Evidence based on relations to other variables

  • Known group evidence = groups that are known to differ on the construct do differ on the instrument in the hypothesised direction

  • Consequential validity = degree to which the instrument is used appropriately, works well in practice, and does not produce any negative or abnormal social and psychological consequences

Quantitative: Data Collection Procedures

TSLB3353

Fundamentals of Educational Research in ESL

• Data collection procedure = technique for physically ​obtaining the data to be analysed in a research study

• ​Tests

  • one​-on-one or in a group situation

  • online or offline or pen-and-paper

• Questionnaires

  • one-on-one or in a group situation

  • in-person, Internet, or mail

Quantitative: Data Collection Procedures

TSLB3353

Fundamentals of Educational Research in ESL

• Quantitative interviews

  • ​​in-person, telephone, or tele-conferencing

  • questions and response categories are fixed in advance and included on the interview protocol

  • interviewer reads the interview protocol, respondents choose from the response categories, and interviewer marks the responses

• Quantitative observation (standardised and structured)

  • ​time-interval sampling = observing during time intervals specified in advance

  • event sampling = observing during and directly after a specific event has occurred

  • checklist, video-recordings

Quantitative: Data Analysis Methods

TSLB3353

Fundamentals of Educational Research in ESL

• Descriptive statistics

  • Frequency distributions

  • Measures of central tendency: mean, mode, median

  • ​Measures of variability: standard deviation, variance, range

  • Measures of relative standing: percentile ranks, z-scores

• Inferential statistics

  • Relationship: Correlation, Regression, Contingency table

  • Comparison: t-test, ANOVA

  • Hypothesis testing

Poll

How do you feel about the lectures on this topic?

Open Ended

Q&A / Reflection / Issue

Please ask at least ONE question.

OR Tell us your thoughts on what you have learned.

OR State ONE issue/difficulty that you have faced.