Flexible Learning

Classroom Assessment in
Flexible Learning

JOSHUA C. MARTINEZ
OIC-Dean, College of Computer Studies
Ateneo de Naga University

Word Cloud

Describe in 1 word Flexible LEarning?

Classroom Assessment in
Flexible Learning

JOSHUA C. MARTINEZ
OIC-Dean, College of Computer Studies
Ateneo de Naga University

Math Response

$$\sqrt[2]{x+y}$$

Type answer here

Correct Answer: 12

Deg^°

Rad

To equip educators with
the knowledge and skills
to effectively leverage

EdTech tools for
assessing learning

outcomes in the context

of flexible learning.

Session Flow

Explore EdTech Tools in creating,
administering, and analyzing assessment

Define Flexible learning and identify its
key characteristics

Review of Formative Strategies and
Summative Techniques

Experience using selected EdTech Tools
for classroom assessment

01

03

02

04

Understanding Flexible
Learning

Assessment Strategies
and Techniques

Tools in Assessing
Learning Outcomes

EdTech Tools
Demo

01: Understanding
Flexible Learning

A Primer

What is Flexible Learning?

Flexible learning encompasses a wide range of
educational approaches that allow students to
learn at their own pace, in their own time, and in
a way that best suits their individual needs.

What is Flexible Learning?

Flexible learning encompasses a wide range of
educational approaches that allow students to
learn at their own pace, in their own time, and in
a way that best suits their individual needs.

It often includes online, hybrid, or blended
modalities.

Online Modality

Definition: All instruction and assessments are conducted entirely online.
Students access learning materials, engage in discussions, and complete
assignments through digital platforms (e.g., learning management systems
like Google Classroom, Moodle, Blackboard).

Synchronous vs. Asynchronous:

●Synchronous: Real-time classes through video conferencing tools (e.g.,
Google Meet, Zoom, MS Teams).

●Asynchronous: Self-paced learning where students access
pre-recorded lectures, readings, and assignments on their own schedule.

Interaction: Limited or no face-to-face interaction; all communication occurs
digitally via forums, emails, or virtual meetings.

Hybrid Modality

Definition: A mix of online and face-to-face (in-person)
instruction, with a portion of learning happening in the
classroom and the rest online.
Schedule: Typically, a fixed schedule where students attend
physical classes on certain days and complete online work
on other days.
Interaction: Combines in-person interaction with digital
learning, allowing flexibility but still requiring campus
presence for part of the instruction.

Blended Modality

Definition: A type of hybrid learning that deeply integrates online and
face-to-face components. The in-person and online portions are
designed to complement each other, with significant online activities
that are not just supplementary but integral to the course.
Example: In-class discussions paired with online collaborative
activities or assessments that extend learning. For instance, students
might engage in face-to-face problem-solving sessions after having
completed online modules or readings.
Interaction: Seamless integration between physical and digital
learning experiences, fostering continuous engagement across both
environments.

Key Differences

Online is fully remote, whereas hybrid requires
some in-person attendance.

Blended learning goes beyond basic hybrid
structures by tightly connecting online and in-person
elements to reinforce one another, making online
work a core part of the learning experience rather
than a separate component.

Flexible Learning Characteristics

● Asynchronous and Synchronous Sessions
● Self-Paced Learning
● Integration of Technology
● Student Autonomy

Flexible Learning Characteristics
Asynchronous Learning

Students access learning materials, complete assignments, and participate in
discussions on their own schedule without the need to be present online at the same
time as their instructor or classmates.

Characteristics:

●Flexibility in time and pace where students can review materials, watch lectures,
and complete tasks at their convenience.

●Interaction occurs through discussion boards, recorded video lectures, or emails.
Communication is delayed, allowing students more time to reflect and respond.

●Best for students with varied schedules or those balancing work and family, as it
accommodates different time zones and commitments.

Flexible Learning Characteristics

Synchronous Learning

Real-time, scheduled sessions where students and instructors interact live, often
through video conferencing (e.g., Zoom, Microsoft Teams).

Characteristics:

●Live interaction that allows for immediate feedback, real-time discussions,
and engagement through activities like live Q&A, group work, and
presentations.

●Provides a fixed schedule, offering more routine and accountability for
students who may benefit from a structured timetable.

●Best for students who prefer live engagement and need real-time guidance or
collaboration opportunities.

Flexible Learning Characteristics

Self-Paced Learning

Students have control over the speed at which they complete learning activities
and progress through the course material.
Characteristics:

●Learners can take additional time on topics they find challenging or move
faster through content they understand well.

●Offers freedom from strict deadlines, though there may still be end-of-term or
end-of-module completion dates.

●Students need to manage their time effectively, requiring strong
self-discipline and motivation.

●Best for mature learners, such as graduate students, who are self-directed
and can manage their workload independently.

Flexible Learning Characteristics

Integration of Technology

Technology is a core component of flexible learning, supporting content delivery,
communication, collaboration, and assessment.
Characteristics:

●Learning Management Systems (LMS) Platforms like Google Classroom host
course materials, assignments, and discussions.

●Interactive Tools like video conferencing platforms (Meet, Zoom), collaboration tools
(Google Docs, Padlet), and multimedia content (videos, simulations) are integrated
to create dynamic learning experiences.

●Assessment Technology:for Quizzes, automated grading, plagiarism detection tools
(Turnitin), and interactive assessments (e.g., Kahoot, Quizlet) enhance learning.

●Best for engaging diverse learning styles through multimedia content and facilitating
remote communication and collaboration.

Flexible Learning Characteristics

Student Autonomy

Learners have greater control over their education, making decisions about when, how, and
what they learn, within the structure of the course.

Characteristics:

●Students may be given options in assessments (e.g., choosing between a research paper
or multimedia presentation) or the ability to set their own deadlines within a broader
timeframe.

●Students must take initiative in managing their learning, balancing time for lectures,
readings, assignments, and other obligations.

●Self-Directed Learning where students are expected to independently explore topics,
conduct research, and engage with the material beyond the provided content.

●Best for Graduate students, as they are typically more mature, motivated, and capable of
handling the demands of independent learning.

Key Principles of Assessment in FL

●Authenticity. Assessments should mirror real-world tasks and
challenges relevant to the field of study.

●Relevance. Assessments should align with the course learning
objectives and reflect the skills and knowledge students are expected
to acquire.

●Accessibility. Assessments should be accessible to all students,
regardless of their learning styles, disabilities, or location.

●Feedback. Regular and constructive feedback is crucial for student
learning and should be provided in a timely and meaningful manner.

●Technology Integration. Leverage technology tools to enhance the
assessment process, streamline grading, and provide personalized
feedback.

02: Assessment Strategies
and Techniques

A Review

What is Assessment?

What is Assessment?

In the context of education……

Process

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation

Measurement

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation → systematic assessment based on standards

Measurement

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation → systematic assessment based on standards

Measurement → quantifying the extent of something

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation → systematic assessment based on standards

Measurement → quantifying the extent of something

Students' Knowledge

Skills

Abilities

Home

Open in new tab

Copy link

Graphing

y = 2x + 5

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation → systematic assessment based on standards

Measurement → quantifying the extent of something

Students' Knowledge → Understanding acquired

Skills → proficiency of performing tasks

Abilities → acquired/innate capacities or talents

What is Assessment?

In the context of education……

Process → series of interrelated steps taken to achieve a goal

Evaluation → systematic assessment based on standards

Measurement → quantifying the extent of something

Students' Knowledge → Understanding acquired

Skills → proficiency of performing tasks

Abilities → acquired/innate capacities or talents

Goal:Determine the level of achievement or progress

What is Assessment?

In the context of education……

Assessment is a process of evaluating and
measuring students' knowledge, skills, and
abilities in order to determine their level of
achievement or progress.

What is Assessment?

In the context of education……

Assessment is a process of evaluating and
measuring students' knowledge, skills, and
abilities in order to determine their level of
achievement or progress. It involves collecting
evidence of learning through various methods,
such as tests, quizzes, projects, presentations,
and observations.

Why do we do
assessments?

Why do we do
assessments?

The PRIMARY PURPOSE
is to provide FEEDBACK.

1. Provide Feedback

Through assessments, teachers can offer
constructive feedback (useful, specific, and
actionable) to students. This feedback is crucial
for students' self-awareness and development,
helping them understand what they are doing well
and where they need improvement.

Why do we do assessments?

2. Monitor Students Progress

Regular assessments enable teachers to track
students' academic growth over time. This helps in
identifying trends, celebrating achievements, and
addressing learning gaps promptly.

Why do we do assessments?

3. Inform Instructional Decisions

Assessments provide teachers with detailed
information about students' strengths and
weaknesses. This data helps in tailoring instruction
to meet individual learning needs, ensuring that all
students can progress effectively.

Why do we do assessments?

4. Set Learning Goals

Assessments help in setting realistic and
achievable learning goals for students. By
understanding their current level of performance,
teachers can guide students in setting and striving
towards new academic objectives.

Why do we do assessments?

5. Evaluate Instructional Effectiveness

Assessments allow teachers to evaluate the
effectiveness of their teaching strategies and
instructional methods. If students are not meeting
learning targets, teachers can adjust their
approaches to improve outcomes.

Why do we do assessments?

6. Ensure Accountability

Educational assessments hold teachers, students,
and schools accountable for academic
performance. They provide a basis for evaluating
the quality of education and ensuring that
educational standards are being met.

Why do we do assessments?

7. Guide Curriculum Development

Data from assessments can inform curriculum
development and revisions. Understanding which
areas students struggle with or excel in can help
educators design a curriculum that better meets
learners' needs.

Why do we do assessments?

8. Support Student Motivation and
Engagement

Assessments, particularly formative ones, can
engage students in the learning process by making
learning objectives clear and providing
opportunities for self-assessment and goal setting.

Why do we do assessments?

9. Communicate with Stakeholders

Assessments provide valuable information that
teachers can share with parents, administrators,
and other stakeholders. This communication is
essential for building a supportive educational
environment and ensuring that all parties are
aligned in supporting student success.

Why do we do assessments?

In summary,

Educational Assessment is a CRITICAL

TOOL in the teaching and learning

process. It provides the necessary data

to guide instruction, support student

learning, evaluate teaching effectiveness,

and ensure that educational goals are

being met.

Major Forms of Assessment

Major Forms of Assessment

Source: https://www.celt.iastate.edu/instructional-strategies/evaluating-teaching/assessment-overview/

TIMING

PURPOSE

NATURE

METHOD

Importance in the Learning Process:
Formative Assessments

1. Inform Instructional Decisions
Feedback from the assessment help teachers identify areas where students may
need additional support and adjust instruction accordingly.

2. Support Student Learning
Encourage students to monitor their own progress, identify areas for improvement,
and take ownership of their learning.

3. Identify Learning Gaps
Helps identify students' strengths and weaknesses, allowing teachers to address
learning gaps and provide targeted interventions.

4. Guide Instructional Adjustments
The feedback enables teachers to make timely adjustments to their instructional
strategies, pacing, and content delivery.

Importance in the Learning Process:
Summative Assessments

1. Measure Learning Outcomes
Provides a comprehensive snapshot of students' knowledge and skills at a specific
point in time. They assess overall mastery of content and provide accountability for
instructional effectiveness.

2. Provide Feedback on Achievement
Offers feedback to students on their overall performance and achievement to help
students gauge their progress and identify areas for further growth.

3. Inform Decision-Making
They provide data for evaluating the effectiveness of instructional programs and
informing educational policies.

4. Promote Goal Setting
By understanding their current level of achievement, students can establish realistic
targets and work towards continuous improvement.

Other Types of Assessment

Other Types of Assessment

Diagnostic Assessment
Conducted at the beginning of a learning unit to
evaluate students' prior knowledge, skills, and
understanding of the topic. It helps teachers
identify students' strengths and weaknesses and
tailor instruction to meet their individual needs.

Other Types of Assessment

Performance Assessment
Involves tasks or projects that require students to
apply their knowledge and skills in real-world
contexts. It assesses students' ability to perform
specific tasks and demonstrate their
understanding through practical application.

Other Types of Assessment

Portfolio Assessment
Portfolio assessment involves compiling and
evaluating a collection of students' work over time
to showcase their progress, achievements, and
learning outcomes. It provides a comprehensive
view of students' growth and development across
various tasks and assignments.

Formative Assessment Strategies

Formative Assessment Strategies

Continuous Feedback Methods
Peer Assessment and Self-Assessment
Observational Techniques
Interactive Activities
Technology-Enhanced Formative Assessments

Formative Assessment Strategies:

Continuous Feedback Methods

Providing ongoing feedback to students during the learning process for guiding
instruction, monitoring progress, and supporting student learning.

●Quizzes and Polls. Provides immediate feedback on student understanding.

●Regular Check-ins. Periodic check-ins with students, either one-on-one or in
small groups, allow teachers to assess individual progress, address
misconceptions, and provide personalized feedback.

●Verbal Feedback. Providing verbal feedback during classroom discussions,
group work, or individual tasks can help clarify concepts, reinforce learning, and
guide students in the right direction.

●Discussion. Providing open-ended questions to stimulate discussion & critical
thinking.

●Questioning Technique. Employ wait-time to encourage deeper response

Formative Assessment Strategies:

Peer Assessment and Self-Assessment

●Peer Assessment. Students review and provide feedback on each
other’s work, fostering collaboration and critical evaluation skills.

●Self-Assessment. Students reflect on their own learning and identify
areas for improvement, promoting self-regulation.

Formative Assessment Strategies:

Observational Techniques

Involve teachers observing student behavior, interactions, and performance to
gather insights into learning progress.

●Learning Journals and Reflective Diaries. Students document their learning
experiences and reflections, providing insights into their understanding and thought
processes.

●Anecdotal Records. Teachers maintain brief notes on student behavior and
performance, useful for identifying trends and tailoring instruction.

●Checklists and Rubrics. Structured tools that outline criteria for success, helping
students understand expectations and self-monitor their progress.

Formative Assessment Strategies:

Interactive Activities

Engage students in hands-on tasks, discussions, and collaborative exercises to
assess understanding and promote active learning.

●Think-Pair-Share Students think individually, discuss with a partner, and share with
the class, promoting active participation and deeper understanding.

●Exit Tickets. Quick, end-of-class activities where students write down what they
learned, questions they have, or reflections on the lesson.

●Concept Maps and Mind Mapping. Visual tools that help students organize and
relate concepts, enhancing comprehension and retention.

Formative Assessment Strategies:

Technology-Enhanced Formative Assessments

Leverage digital tools and platforms to gather data, provide feedback, and
assess student learning

●Online Quizzes. Platforms like Google Forms or Kahoot! allow teachers to
create and distribute online quizzes for immediate feedback and
assessment.

●Interactive Platforms . Tools like Nearpod or Socrative enable teachers to
deliver interactive lessons, conduct real-time assessments, and engage
students through multimedia activities.

Summative Assessment Strategies

Summative Assessment Strategies:

Traditional Methods
Alternative Methods
Technology-Enhanced Summative Assessments

Summative Assessment Strategies:

Traditional Methods

●Written Exams. These assessments typically consist of multiple-choice
questions, short answers, or essay questions for evaluating knowledge retention
and understanding.

●Standardized Tests. Designed to measure students' knowledge and skills in a
consistent and uniform manner across a larger population

●Final Projects or Presentations. Students demonstrate their understanding by
completing a final project or presenting their work to showcase their knowledge,
skills, and creativity.

Traditional methods offer a familiar and established framework for evaluation but
may have limitations in assessing higher-order thinking skills or real-world
application of knowledge.

Summative Assessment Strategies:

Alternative Methods

Provide a more comprehensive and authentic evaluation of student learning. These methods
focus on performance-based tasks and real-world application of knowledge.

●Performance Assessments. Students demonstrate their skills and knowledge through
real-world tasks such as creating a product, conducting experiments, or solving complex
problems.

●Portfolios. Showcase students' work samples, projects, and reflections to demonstrate
their growth and achievement over time. They provide a holistic view of students' abilities
and progress.

●Simulations or Role-Playing. Students engage in simulated scenarios or role-playing
activities to assess their ability to apply knowledge and skills in practical situations.

Alternative methods encourage critical thinking, creativity, and the integration of knowledge
but may require more time and resources for implementation and evaluation.

Summative Assessment Strategies:

Technology-Enhanced Summative Assessments

●Online Quizzes and Tests. Web-based platforms allow for the creation and
administration of quizzes and tests that can be automatically graded, providing
immediate feedback to students and saving teachers time.

●Digital Simulations. Interactive simulations provide virtual environments where
students can apply their knowledge and skills to solve problems or make
decisions.

●Automated Essay Scoring. Advanced algorithms and artificial intelligence can
be used to assess and score essays, providing consistent and objective
evaluations.

Technology-enhanced summative assessments offer advantages such as increased
efficiency, flexibility, and the ability to provide instant feedback, but they may also
require access to technology and careful consideration of validity and reliability.

Effective Implementation of Assessments:

●

Ensure assessments are aligned with learning objectives and
instructional strategies.

●

Use a variety of assessment types to capture different aspects
of student learning.

Best Practices and Challenges

Best Practices and Challenges

Balancing Formative and Summative Assessments:

●

Combine frequent, low-stakes formative assessments with
periodic, high-stakes summative assessments to provide a
comprehensive view of student progress.

●

Use formative assessments to inform and improve summative
assessments.

Best Practices and Challenges

Addressing Common Challenges and Resistance:

●

Challenges. Time constraints, student anxiety, and potential
bias.

●

Strategies. Provide clear instructions, create a supportive
assessment environment, and use data to make informed
instructional decisions.

03: Tools Assessing
Learning Outcomes

Explore EdTech Tools in creating, administering, and analyzing
assessment

Importance of Technology in Assessment

●

Technology enhances the efficiency and effectiveness of
assessments by providing diverse, scalable, and flexible tools
that can cater to various learning styles and needs.

●

EdTech tools facilitate real-time feedback, data collection,
and analysis, helping teachers make informed instructional
decisions.

●

They also engage students more actively, making the
assessment process more interactive and less intimidating.

Criteria for Selecting Appropriate EdTech Tool

1. Alignment with Learning Objectives
Choose EdTech tools that align with your specific learning objectives and
assessment goals. The tools should allow you to assess the desired knowledge, skills,
and competencies effectively.

2. Validity and Reliability
Ensure that the EdTech tools have been validated and demonstrate reliability in
measuring student learning outcomes. Look for evidence of the tool's effectiveness
and its alignment with established assessment standards.

3. Ease of Use
Select tools that are user-friendly and intuitive for both teachers and students. The
tools should be easy to navigate, with clear instructions and minimal learning curve to
maximize efficiency and minimize potential technical difficulties.

Criteria for Selecting Appropriate EdTech Tool

4. Flexibility and Customizability
Look for tools that offer flexibility in adapting to different assessment needs and
formats. The ability to customize assessment parameters, question types, and
settings can help meet the specific requirements of different subject areas, grade
levels, or assessment purposes.

5. Data Security and Privacy
EdTech must tools prioritize data security and privacy. Verify if it complies with
relevant data protection regulations and have safeguards in place to protect student
information.

6. Accessibility and Inclusivity
Ensure that the tools can accommodate diverse learners, including students with
disabilities or specific learning needs. The tools should provide options for accessibility
features such as text-to-speech, closed captioning, or adjustable font sizes.

Criteria for Selecting Appropriate EdTech Tool

7. Feedback and Reporting
Look for tools that provide meaningful feedback (data analytics) and robust reporting
features that offer clear and detailed insights into student performance, allowing for
effective analysis and informed decision-making.

8. Integration and Compatibility
Consider how the EdTech tools integrate with existing technology infrastructure and
learning management systems. Ensure compatibility with the devices and platforms
commonly used by students and teachers.

9. Cost and Support
Evaluate the cost-effectiveness of the EdTech tools and consider the level of support
provided by the tool developers. Explore whether there are training resources,
technical support, or user communities available to assist with implementation and
troubleshooting.

It is essential to evaluate the
suitability of tools for your

specific educational context and

the needs of your students.

Assessment Strategies and Techniques

FORMATIVE ASSESSMENT
Continuous Feedback Methods
Peer Assessment and Self-Assessment
Observational Techniques
Interactive Activities
Technology-Enhanced Formative
Assessments

SUMMATIVE ASSESSMENT
Traditional Methods
Alternative Methods
Technology-Enhanced Summative
Assessments

What tools should I use?

EdTech Tools for Assessment

Learning Management Systems
Assessment Creation Tools
Interactive and Engaging Tools
Portfolio Tools
Rubric and Grading Tools
Feedback and Reflective Tools

Learning Management Systems

A software application or platform that facilitates the
management, delivery, and tracking of educational courses,
training programs, and learning materials.

It provides a centralized online environment where educators
can create, organize, and deliver content, as well as monitor and
assess student progress.

Students, on the other hand, can access course materials,
participate in activities, submit assignments, and engage in
discussions through the LMS.

Learning Management Systems

Learning Management Systems

1.

Assessment and Grading. LMS enables the creation and
administration of assessments, such as quizzes, tests,
assignments, and surveys. It often includes features for
automated grading, providing feedback to students, and
generating reports for teachers.

2.

Progress Tracking. LMS tracks and records student
progress, including completion of activities, assessment
scores, and overall course performance. This information
helps educators monitor student engagement and identify
areas where additional support may be needed.

Assessments available in
Google Classroom

●Quick Survey

●Discussion Board

●Exit Tickets

●Assignment Tasks (written tasks, projects, group work, etc.)

●Tests/Quizzes/Surveys

●Practice Test

●Rubric

●Originality Reports

●Grading and Feedback

Google Classroom

DEMO

Assessment Creation Tools

●Google Forms and Microsoft Forms. Easy-to-use platforms for
creating surveys, quizzes, and polls. Features include automatic
grading, branching logic, and data export to spreadsheets for
analysis.

●Quizlet. Tool for creating flashcards, quizzes, and interactive
games that help with memorization and understanding.

●Quizizz. Gamified quiz platform that allows for self-paced
learning and instant feedback.

●Classmarker. online platform that provides tools for
creating, administering, and grading online assessments
and exams

●Exam.net. A comprehensive online assessment platform that
offers a secure, user-friendly, and efficient solution for
creating and administering digital assessments.

Quizizz DEMO

Interactive and Engaging Tools

●

Kahoot! Game-based learning platform where
teachers can create quizzes and interactive
lessons.

●

Mentimeter. Tool for creating live polls,
quizzes, and word clouds to engage students
in real-time.

●

Poll Everywhere. Platform for real-time
audience response, suitable for quizzes, polls,
and interactive Q&A sessions.

Kahoot DEMO

Portfolio Tools

● Seesaw. Digital portfolio platform

where students can document and
share their learning progress with
teachers and parents.

● Mahara: Open-source e-portfolio

system that supports personal
learning and development plans.

Rubric and Grading Tools

●

Rubistar, Quick Rubric. Online tools for creating and
customizing rubrics for various assignments and projects.

●

GradeMark. Part of Turnitin, provides tools for annotating
and grading student papers with integrated plagiarism
detection.

●

SpeedGrader. Canvas tool for efficient grading and
feedback, supporting various media types and rubrics.

Feedback and Reflective Tools

●

Padlet. Collaborative platform where students can post
reflections, ideas, and multimedia content.

●

Flipgrid. Video discussion tool that allows students to record
and share short video responses and reflections.

●

WordPress, Blogger. Platforms for students to maintain
reflective journals or blogs, documenting their learning
experiences and insights.

04: EdTech Tools
Demo

Explore EdTech Tools in creating, administering, and analyzing
assessment

References

Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through
classroom assessment. Phi Delta Kappan, 80(2), 139-148.

Brookhart, S. M. (2013). How to Assess Higher-Order Thinking Skills in Your
Classroom. Alexandria, VA: Association for Supervision and Curriculum Development.

Harlen, W. (2006). Assessment of Learning. In A. J. Croxford, (Ed.), The SAGE
Handbook of Curriculum and Instruction (pp. 197-215). Sage Publications.

Heritage, M. (2010). Formative assessment: What do teachers need to know and do? Phi
Delta Kappan, 92(2), 90-97.

Northern Illinois University. (n.d.). What assessment tools are available? Flexible
Teaching.

Popham, W. J. (2009). Assessment literacy for teachers: Faddish or fundamental?
Theory into Practice, 48(1), 4-11.