An introduction to (English) Language testing

An introduction into
(English) Language
testing

Bart Deygers
Cel Diversiteit & Gender / Taalbeleid @ Ghent University
CNaVT
An introduction to (English) Language testing

What is assessment?
Why assess?
Assess what?
An introduction to (English) Language testing

History
Concepts
A bit of history
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing history

Concepts
Construction “[Until about 1980], language was basically
Criteria seen to be grammar: that eventually came to
Teaching
be regarded as too distant, too abstract.”
(Davies 2008)
Closing
An introduction to (English) Language testing

History Language testing history

Concepts
Construction “[In the 1980s], language was reckoned to be
Criteria a set of real life encounters and experiences
Teaching
and tasks, a view which took „real life‟ testing
so seriously that it lost both objectivity and
Closing
generality.”
(Davies 2008)
An introduction to (English) Language testing

History Language testing history

Concepts
Construction “[From the 1990s] there has been a
Criteria compromise between these two positions,
Teaching
where language is viewed as being about
communication but in order to make contact
Closing
with that communication it is considered
necessary to employ some kind of distancing
from the mush of general goings on that
make up our daily life in language.”

(Davies 2008)
An introduction to (English) Language testing

History Language testing now

Concepts
Construction Focus on:
Criteria  Methodology
Teaching
 Practical advances
 Performance-affecting factors
Closing
 Performance assessment
 Ethical issues
(Bachman 2000)
An introduction to (English) Language testing

History
Concepts
Some key concepts
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing definitions

Concepts
Test
Evaluation
Test
Assessment An often formalised (collection of) task(s),
Reliability designed to determine a test taker’s ability,
Validity knowledge or intelligence.
Face validity (Cf. Dochy 1996, 2002)
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing definitions

Concepts
Test
Evaluation
Test
Assessment
Reliability
Validity Evaluation
Face validity The judgement made about a test taker’s ability,
Authenticity
knowledge or intelligence, based on his/her test
Construction performance.
Criteria (Cf. Douglas 2000, Lynch 2003)

Teaching
Closing
An introduction to (English) Language testing

History Language testing definitions

Concepts
Test
Evaluation
Test
Assessment
Reliability
Validity Evaluation
Face validity
Authenticity

Construction Assessment
Criteria
Judging the ability of a learner based on a test or
otherwise and using this judgement as a
Teaching constructive element in learning over time.
Closing (Cf. Gipps 1994, Lynch 2005)
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Reliability
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Reliability
Evaluation
Assessment
Reliability
If a test w, taken by student x, is graded twice by
Validity teacher y, student x will receive two identical
Face validity scores.
Authenticity

Construction If a test w, taken by student x, is graded by

Criteria teacher y and teacher z, student x will receive
two identical scores.
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Reliability
Evaluation
Assessment
Reliability
Do test scores correctly reflect the learners’
Validity actual ability?
Face validity
Authenticity
How can you draw conclusions based on test
Construction results if you are not sure about the results?
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Reliability
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Increasing reliability through
Evaluation
Assessment  Identical criteria for students and tutors
Reliability
Validity
 Transparent scoring
Face validity  No chain questions
Authenticity  Rubric
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Validity
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Validity
Evaluation
Assessment
Reliability
Write an essay on the consequences of
Validity climate change.
Face validity Time: 30 minutes
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Validity
Evaluation To what extent does the test really test what it is
Assessment meant to test?
Reliability
Validity
Face validity How can you evaluate a specific ability if you are
Authenticity
not measuring that ability?
Construction
Criteria Make sure you and your students know what
Teaching you want to test!
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Face validity
Evaluation The learners’ perception of how valid a test is.
Assessment
Reliability
Validity How can you expect test takers to take the test
Face validity results seriously if they do not take the test
Authenticity
seriously?
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Face validity
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Face validity
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Authenticity
Evaluation Does the test include situations that are similar to
Assessment what the learners will face ‘in real life’?
Reliability
Validity
Face validity How can you determine somebody‟s language
Authenticity
performance in reality in the task does not
Construction correspond to reality?
Criteria
Teaching Authenticity matters mainly in productive,
communicative tasks.
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test Authenticity
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
“We don‟t use if-clauses. If you know something,
you write about it. If you don‟t know for sure, you
don‟t mention it.“
An introduction to (English) Language testing

History Language testing concepts

Concepts
Test
Evaluation
Assessment
Reliability
Validity
Face validity
Authenticity

Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History
Concepts
Some thoughts on test
Construction construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Test construction: questions

Concepts
Construction WHY WHAT HOW
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Test construction: questions

Concepts
Construction WHY
Criteria
 Determine entry level
 Student evaluation
Teaching  Motivational
Closing  Punishment

WHAT

HOW
An introduction to (English) Language testing

History Test construction: questions

Concepts
Construction WHY Test purpose
Criteria
Teaching
WHAT Test specifications:
- What learners?
Closing - Target language situation?
- Which skills?
- Which methods?
HOW
An introduction to (English) Language testing

History Test construction: questions

Concepts
Construction WHY Test purpose
Criteria
Teaching
WHAT Test specifications
Closing
HOW Task types
An introduction to (English) Language testing

History Test construction: questions

Concepts
Task types
Construction
Criteria discrete point / integrated / non authentic /
Teaching simulated authentic / genuine authentic /
Closing multiple choice / ranking / hotspot / true-false /
matching / structuring / fill in the gaps / cloze /
C-cloze / semi-open / open answer / diary /
portfolio / syllabus task / problem-based task /
product assessment / process assessment /
oral / written / computer-based / paper-based /
self assessment / peer assessment/ co
assessment/ tutor assessment / in-class
observation / fixed-point testing / norm
referencing / criterion referencing / …
An introduction to (English) Language testing

History
Concepts
A word or two about criteria
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History What about the CEF?

Concepts
Construction
Criteria
CEF
Rubric
Teaching
Closing
An introduction to (English) Language testing

The CEF
History
°2001 / Council of Europe
Concepts
Construction
Goals:
Criteria - Encouraging reflection
CEF
Rubric - Fuelling discussion
Teaching - Creating common language
“One of the aims of the Framework is to help partners to
Closing
describe the levels of proficiency required by existing
standards, tests and examinations in order to facilitate
comparisons between different systems of qualifications.”
An introduction to (English) Language testing

CEF: system
History
Concepts
C2
Construction Skilled User
C1
Criteria
CEF
Rubric
B2
Teaching Independent user
B1
Closing

A2
Basic user
A1

Full text
Overview
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: influence
History
Concepts IELTS
Construction TOEFL
DIALANG
Criteria CNaVT
CEF
Rubric
Teaching EUROPASS
Closing
Handboeken
CEF
Didactiek
Talenscholen
An introduction to (English) Language testing

CEF: problem solved?

History
Concepts
Construction
Criteria
CEF
Rubric
Teaching
Closing
An introduction to (English) Language testing

CEF: give it a go
History
Concepts www.ceftrain.net
Construction
Criteria
CEF
Rubric
Teaching
Closing
An introduction to (English) Language testing

CEF: relating tests

History
Step 1 Step 2 Step 3
Concepts
Specification Standardisation Validation
Construction (CEF-training) (Test analysis)
Criteria
CEF
Rubric
Internal validity Linking benchmarked Verifying psychometric
Teaching items to the CEF test quality
Closing
External validity Linking test answers Independent study
to the CEF

Implementation Confirmation
An introduction to (English) Language testing

History Holistic rubrics

Concepts
Construction
Criteria
CEF
Rubric
Teaching
Closing
An introduction to (English) Language testing

History Dichotomous rubrics

Concepts Yes No
Construction
Well-paced flow 1 0
Criteria
CEF Message is clear 1 0
Rubric
Acceptable pronunciation 1 0
Teaching
Effective use of grammar 1 0
Closing
Effective use of vocabulary 1 0
….
An introduction to (English) Language testing

History Band rubrics

Concepts
Construction
Criteria
CEF
Rubric
Teaching
Closing
An introduction to (English) Language testing

History
Concepts
Language testing and
Construction Language teaching
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Testing and teaching

Concepts
Washback
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Testing and teaching

Concepts
Washback
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Language testing concepts

Concepts
Washback
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Testing and teaching

Concepts
Motivation
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Testing and teaching

Concepts
Motivation
Construction
Criteria
Teaching
Closing

Reliability // Fairness
Authenticity // Realness
(Face) Validity // Credibility
An introduction to (English) Language testing

History Testing and teaching

Concepts
Motivation
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History
Concepts
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History
Concepts
[teaser: testing the test]
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

Descriptive statistics
How difficult is my test?
Average
An introduction to (English) Language testing

Descriptive statistics
How difficult is my test?
Average
Standard Deviation
An introduction to (English) Language testing

Descriptive statistics
How difficult is my test?
Average
Standard Deviation
Voorbeeld:
max = 100 | Ave = 50 | StDev = 10
68,3 % = 40 – 60
95,4 % = 30 – 70
99, 7% = 20 - 80
68,3% max 1 x SD
95,4% max 2 x SD
99,7% max 3 x SD
An introduction to (English) Language testing

Descriptive statistics
Average or median?

How well-off is the average employee of

Peters & Sons?
An introduction to (English) Language testing

Descriptive statistics
How difficult is my test?
Average or median?
An introduction to (English) Language testing

Descriptive statistics
An introduction to (English) Language testing

Descriptive statistics
An introduction to (English) Language testing

Descriptive statistics
An introduction to (English) Language testing

Descriptive statistics
An introduction to (English) Language testing

Correlations
Is this test as difficult as last year’s
test?
Is group A as proficient as group B?
An introduction to (English) Language testing

Correlations
Test Test
12
1 2
1 1 10

2 2
8
3 3
Series1
4 4 6
Series2

5 5
4

6 6
2
7 7
8 8 0
0 2 4 6 8 10 12
9 9
10 10

Corr = + 1
An introduction to (English) Language testing

Correlations
Test Test
1 2
1 10
2 9
3 8
4 7
5 6
6 5
7 4
8 3
9 2
10 1

Corr= - 1
An introduction to (English) Language testing

Correlations

1,00 6,00
2,00 3,00
3,00 5,00
4,00 1,00
5,00 6,00
6,00 8,00
7,00 2,00
8,00 4,00

Corr= + .05
An introduction to (English) Language testing

Correlations
An introduction to (English) Language testing

Correlations
An introduction to (English) Language testing

Split-half reliability
Is the level of difficulty consistent within the
test?
An introduction to (English) Language testing

Split-half reliability
Correlation between 2 test halves


Set 1 Set 2 80 ,0 0




#1 #2 


TOT_100
70 ,0 0 

#3 #4 





#5 #6 60 ,0 0


… …






11 ,0 0 12 ,0 0 13 ,0 0 14 ,0 0 15 ,0 0 16 ,0 0

Corr = + 1 TOT_20
An introduction to (English) Language testing

Split-half reliability

Set 1 Set 2 16 ,0 0  

#1 #2

SCORE_ADMITTED


 

#3 #4 14 ,0 0

  

#5 #6 12 ,0 0  

… … 


10 ,0 0

2,00 3,00 4,00 5,00 6,00

EX_4
Corr = + .66
An introduction to (English) Language testing

Discriminating potential
Does this question separate high
achievers from weaker students?
An introduction to (English) Language testing

Cronbach’s Alpha
An introduction to (English) Language testing

Cronbach’s Alpha
An introduction to (English) Language testing

Cronbach’s Alpha
An introduction to (English) Language testing

History
Concepts
Something to take home
Construction
Criteria
Teaching
Closing
An introduction to (English) Language testing

History Task 1: Increase your self-esteem

Concepts
Construction Know what the CEF is!
Criteria http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf
Teaching
Know what TOEFL and IELTS are!
Closing www.toefl.org
www.ielts.org

Remember something about reliability and validity!

An introduction to (English) Language testing

History Task 2: Test construction

Concepts
Construction For one of your classes, create a test which
Criteria is motivating, valid and reliable.
Teaching
Closing