Scoring guide Changes

Create connection. Value conversation.

Scoring guide
Changes

Self-directed training

June, 2023

2

2

Summary

02 Layout

03 In-Between Scores

06 Vocabulary

07 Language Application

01 Overview and Objectives

05 Pronunciation

7.1. Paraphrasing.
7.2. Coherence and idea organization.
7.3. Active Listening and Comprehension.
7.4. Critical Thinking and Problem-Solving.
7.5. Role–Play.
7.6. Positive Language.
7.7. Empathy.

08 Interview Flow

​04 Grammar

Create connection. Value conversation.

01

In this training lesson, you will explore the recent changes made to the language scoring guide. The language scoring guide is a vital tool for assessing language proficiency, and it's important to stay updated on any modifications. Through this lesson, you will learn about the revisions and enhancements that have been implemented. Additionally, you will have opportunities to assess your understanding through knowledge checks.

​Lesson Overview:

Create connection. Value conversation.

01

• By the end of this training lesson, you will be able to:

• Understand the key changes made to the language scoring guide.

• Identify the reasons behind these modifications.

• Apply the revised scoring guide to evaluate language proficiency accurately.

• Successfully complete knowledge checks to reinforce your understanding.

​​Lesson Objectives:

​Let's Start

Create connection. Value conversation.

Layout

02

Changes and color coding

5

5

Layout

Visual aid

The color coding in the layout will help us find the descriptor faster, according to the level that the candidate demonstrates during the interaction.

Level

Score

C1

5 - 4.5

B2+

4

B2

3.5

B1+

3

B1

2.5

A2+

2

A2

1.5

A1

1

Pre A1

0.5

No English

0

6

6

Easy to move

Visual aid

You will have the capability to navigate vertically and horizontally without losing the visual guidance provided by the color coding, which corresponds to the desired score you are seeking.

7

Create connection. Value conversation.

In- Between
Scores

03

Explanation, color coding, and usage.

8

8

In- Between
scores

How to identify them.

These scores will also have a different color coding. They will be assigned the color 'Gold Accent 4, lighter 80%,' which can be found in the default palette.

9

9

What
changed?

Comparison – Old Version

The previous scoring guide employed the word

"some" in the in-between descriptors to highlight
positive aspects of the scores above, which made it
challenging for us to ascertain if candidates had an
intermediate score in any of the skills we evaluate.

10

10

What
changed?

Comparison – New Version

The new scoring guide not only encompasses all the
elements of the previous scores but also emphasizes
that an in-between score has the potential to influence
a specific proportion of the scores above. We have
successfully established a balanced distribution where
the in-between score retains a 50/50 control over the
positive features of the score above it.

11

11

The positive
Features

Color Coding

To enhance the visibility of the positive attributes associated with higher scores, we have incorporated a visual aid through color coding. This feature will effectively highlight the advantageous aspects of scores above the in-between range, simplifying the evaluation process when candidates fall within this category.

In this case, we have 6 positive features of this descriptor.

Create connection. Value conversation.

Pronunciation

05

Changes

Pronunciation

The general descriptor is now included for each score within this subcategory.

​This change is implemented to serve as a reminder that the general descriptor applies to each of the pronunciation subcategories

14

Pronunciation

General descriptor

Word endings, -es & -ed

In this subcategory, we now have an area of opportunity for scores 4.5 and 5.

15

Pronunciation

General descriptor

Word endings, -es & -ed

Articulation

In this subcategory, we now have an area of opportunity for scores 4.5 and 5.

Knowledge Check

Multiple Choice

In the category of pronunciation, the general descriptor has been relocated.

Where can you now find the general descriptor?

• Within this subcategory

• Right side

• There is no general descriptor

Multiple Choice

Pronunciation

In the subcategory word endings, -es & -ed, an area of opportunity (AOO) was introduced.

Could you please specify the score at which this change was implemented?

• 3

• 1

• 5 - 4.5

• 3.5

Multiple Choice

Pronunciation

In the subcategory articulation, an area of opportunity (AOO) was introduced. Could you please specify the score at which this change was implemented?

• No English

• 2.5

• 5

• 3.5

Create connection. Value conversation.

Vocabulary

06

Changes

17

17

Idiomatic
Expressions

Being more specific

The previous descriptor for scores 4.5 and 5 in the C1 level stated that the candidate had the ability to control common idiomatic expressions and colloquialisms.

However, it is essential to note that English speakers employ these expressions within specific contexts based on the topic of conversation. To address this, we have included the following specification:

18

18

Connectors

AOO added to 5

One of the significant debates in past calibrations regarding the highest scores revolved around considering candidates with these scores as "perfect" and assuming they do not make any mistakes in preposition usage.

In light of this, we have identified a noteworthy area of improvement for the 5 (C1) score.

19

19

Connectors

AOO added to 4.0

As with the previous score, we also added an Area Of Opportunity to the 4 (B2+) score.

Knowledge Check

Multiple Choice

Vocabulary range & control

Regarding idiomatic expressions and colloquialisms, there have been edits made for scores 4.5 and 5.

Should test takers use idiomatic expressions and colloquialisms in their responses or are they not required to do so?

• They need to use diomatic expressions and colloquialisms in their answers.

• There was no change made to this subcategory.

• It implies that test takers have the option to incorporate such language in their responses if they choose to do so.

Multiple Select

Connectors usage

In the subcategory of the usage of connectors areas of opportunity (AOO) were introduced.

Could you please identify the specific changes that were implemented?

More than one answer is correct.

• An area of opportunity (AOO) was added to the score 4.

• An area of opportunity (AOO) was added to the score No English.

• An area of opportunity (AOO) was added to the score 4.5 & 5.

• 
  An area of opportunity (AOO) was added to the score 3.5.

Create connection. Value conversation.

Language
Application

07

Changes

21

Paraphrasing

7.1

22

Score ponderation: 0 to 1

The new weight for this category will prevent it from significantly impacting the overall score of the assessed person.

Paraphrasing

What’s changing?

This Language Application category will include additional information in the Scoring Guide as well as the Interview Flow.

1

23

“Not applicable” as an
option

Associates from certain campaigns are not required to include paraphrasing as part of their call flow. This option will allow the LLS to properly assess such contexts.

Paraphrasing

What’s changing?

This Language Application category will include additional information in the Scoring Guide as well as the Interview Flow.

2

24

Number of details inside the descriptors

To favor convenience, the LLS will see the number of details required to reach each score inside their descriptors.

Paraphrasing

What’s changing?

This Language Application category will include additional information in the Scoring Guide as well as the Interview Flow.

3

25

New Instructions

Based on the experience of the previous interview flow, the instructions of the Paraphrasing exercises have been modified to be as explicit and specific as possible when guiding our counterpart through the expectations of the activity.

Paraphrasing

What’s changing?

This Language Application category will include additional information in the Scoring Guide as well as the Interview Flow.

4

26

Paraphrasing

What’s changing?

This Language Application category will include additional information in the Scoring Guide as well as the Interview Flow.

Additional paraphrasing stories

The LLS will have more paraphrasing stories to work with during interviews, diagnostics, and follow-ups. They contain a similar amount of details to be applicable to our scoring criteria.

5

Knowledge Check

Multiple Choice

The score of 0 in the Paraphrasing category was modified.

Can you identify and explain the specific change that was implemented?

• Doesn't paraphrase

  when required to do so.

• Doesn't paraphrase

  when required to do so.

  Does not provide any detail related to the task.

  (0 details)

• Doesn't paraphrase

Multiple Choice

In the paraphrasing subcategory, have new scenarios been introduced with a number of details to provide more options for evaluation?

• Yes

• No

Multiple Choice

How are we scoring paraphrasing?

• 
  From 0 to 1. Not including N/A.

• 
  From 0 to 5. Including N/A.

• 
  From 1 to 5. Not including N/A.

• 
  From 0 to 1. Including N/A.

Multiple Choice

Paraphrasing

Where can we locate the new scenarios?

• CEFR

• Manual and examples

• 
  Interview Flow

• Glossary

27

27

Coherence and Idea

Organization

7.2

28

28

Changes and
Updates

AOO added to 5 and 4.5 (C1)

Regarding "Coherence and idea organization," we have incorporated a significant aspect of improvement into the top score range (C1 4.5 and 5). This decision stems from our numerous deliberations on this subject during prior calibrations, rendering it challenging to assign this specific score to candidates who fully embody the criteria for this proficiency level.

Knowledge Check

Multiple Choice

In the subcategory Coherence & idea organization, an area of opportunity (AOO) was introduced.

Could you please specify the score at which this change was implemented?

• 5 - 4.5

• 1

• 3

29

Active Listening and

Comprehension

7.3

30

Active Listening and
Comprehension

The interviewer or the interlocutor is not part of
the descriptor

Allusions to the interviewer or interlocutor’s delivery of the message have been removed from this category’s descriptors to better focus on the listening and comprehension skills of the person being evaluated.

31

Critical Thinking

&

Problem-Solving

7.4

32

Competencies being evaluated:

1. Identifying the problem and understanding the issue or problem.

2.Analyzing possible solutions.

3.Proposing a solution that is appropriate to the task.

No English

Subcategory: Critical
Thinking/Problem Solving

Score Range: 0 - No English

Option No. 2: Candidate provides an
answer that is not related to the task.

Critical
Thinking/Problem
Solving

(Role-Play)

Subcategory: Critical Thinking/Problem-Solving
Evaluation Method: Role-Play

Critical Thinking/Problem-Solving competency is assessed solely through
the role-play activity.

Context of Production was removed for this subcategory due to its
complexity as it involves a role-play activity.

33

Positive
language

7.6

Evaluation Method: Role-Play

Positive language competency is assessed
solely through the role-play activity.

Positive
language

Subcategory: Positive
language
Role-Play Evaluation

Competencies being evaluated:

1.Incorporates positive words and
phrases.

2.Avoids using negative words.

3.Focuses on what can be done.

4.Tone of voice.

Knowledge Check

Multiple Choice

Positive language

When do we evaluate positive language?

• Role-play

• In the beginning

• Throughout

• Paraphrasing exercise

Multiple Choice

Positive language

When assessing the No English score, do we consider whether the communication approach can be perceived as rude or the use of negative language?

• True

• False

35

Empathy

7.7

Evaluation Method: Role-Play

Empathy is assessed solely through the role-
play activity.

Empathy

Subcategory: Empathy

Role-Play Evaluation

Competencies being evaluated:

1.Understanding others.

2.Empathize with the customer
(addresses feelings).

Score Range: 0 - No English

Option No. 2:
The speaker does not show understanding or concern for the feelings of
others.

He/she doesn't address feelings. Focuses only on the main problem.

37

Difference between
empathy and positive language

Empathy

Empathy refers to the ability to understand and share the feelings of another person. It involves being able to put oneself in someone else's shoes, see things from their perspective, and respond with sensitivity and understanding. Empathy is a key component of effective communication, particularly in situations where there may be conflict or disagreement.

Positive Language

Positive language, on the other hand, refers to using language that is constructive, supportive, and uplifting. It involves focusing on what can be done rather than what cannot, using words that convey encouragement and hope, and avoiding language that is negative or critical. Positive language can help to create a more positive atmosphere and improve communication by reducing defensiveness and encouraging cooperation.

While empathy and positive language are both important for effective communication, they serve different purposes.

Empathy is focused on understanding and responding to the feelings and perspectives of others, while positive language is focused on creating a positive and constructive atmosphere for communication.

Knowledge Check

Multiple Choice

Empathy

When evaluating the No English score, do we consider whether the speaker demonstrates a lack of understanding or concern for the feelings of others and if they fail to address emotions and solely focus on the main problem?

• We only evaluate: No communication possible

• Yes, it's considered under this category.

• 
  No, it's not considered under this category.

Create connection. Value conversation.

Interview
Flow

08

Changes

Navigation

The new Excel format for the tool allows the use of filters to search for the options you need at different moments of the assessment.

The tool is now part of the LF file and includes additional information.

The new
Interview
Flow

​1

Verb Tenses

This category has been added to the questionnaire table to ensure the LLS can ask questions in the tense they need to evaluate.

The tool is now part of the LF file and includes additional information.

The new
Interview
Flow

​2

New roleplay dynamics & scenarios

The new roleplay scenarios found in the interview flow allow flexibility of speech for the LLS while giving very specific details of the context. The person being assessed will also get enough information to understand how to solve the issue.

The tool is now part of the LF file and includes additional information.

The new
Interview
Flow

​3

Conceptually difficult questions

To ensure the accurate assessment of advanced speakers, a series of conceptually difficult questions were added and marked as such.

The tool is now part of the LF file and includes additional information.

The new
Interview
Flow

​4

The tool is now part of the LF file and includes additional information.

Additional paraphrasing stories

The new
Interview
Flow

​5

New paraphrasing exercises were added and categorized per topic, thus ensuring variety in the different evaluation stages of the Language Workshop.