Running head: DIAGNOSTIC ASSESSMENT FAMILIARIZATION 1

Diagnostic Assessment Familiarization

Luay Askar

California State University, Monterey Bay

December 10, 2017

IST522 Instructional Design

Dr. Jeanne Farrington

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 2

Diagnostic Assessment Familiarization Project Definition

The proposed training module is designed to introduce instructors at the Defense

Language Institute (DLI) to the principles of the Diagnostic Assessment (DA). The targeted

audience is newly joined and seasoned instructors, aiming at familiarizing the former to the basic

principles and the process of conducting the DA, and as refresher training for the later. This

familiarization training is not a replacement for the DA specialist certification training, but it will

help the teachers to conduct informal DA for their students in order to support them during the

learning process. Formal DA reports are written by certified DA specialists only according to the

current regulations.

The learning process is a collaborative work of both the instructor and the learner.

Implementing the student-centered approach in the classroom has changed the role of the teacher

from Sage on Stage to Guide on the Side (King, 1993). The teacher needs to facilitate, to

prepare materials, and to assess the students. There are different types of assessments using

different tools, all aim at gathering data and the teachers require being familiar with the

diagnostic tool they will be using for the assessment. Assessment is the process of gathering

data. More specifically, assessment is the ways instructors gather data about their teaching and

their students learning (Hanna & Dettmer, 2004).

The mission of the DLI is to provide culturally based foreign language education, training,

evaluation and sustainment to enhance the security of the nation. At the DLI, the time allocated

to each language program differs depending on the category of the language being taught.

Recently, the DLI set up a new standard for the students to pass the final exams with the goal of

achieving an Interagency Language Roundtable (ILR) level of 2+/2+/2, in reading, listening and

speaking skill modalities, consecutively. This goal is a challenge for both the students and the
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 3

teachers. For the students, acquiring a new language is not a personal goal but also a job-related

goal, thus requiring more effort and dedication. As for the teachers, their standards surpassed the

traditional teaching standards to include and not limited to; implementing more advanced

technology in the classroom as well as implementing diagnostic teaching and differentiated

instruction in their instructional model. Traditional teaching following the rule one size fits all

is no longer acceptable, if we are aiming to achieve the DLI goals of a proficiency level of

2+/2+/2 in the three skills(listening, reading, and speaking).

In order to implement diagnostic teaching and differentiated instruction, the instructor

requires knowledge of the diagnostic tools and thus the DA. Alderson (2004) came up with a list

of features that can help to clarify the benefits of the DA, of these:

Identify strengths and weaknesses in a learners knowledge or use of language.

Have a focus on weaknesses that lead to remediation in further instruction.

Enable detailed analysis and report of responses to items or tasks.

Provide feedback that can be acted upon.

Therefore, this familiarization training is to introduce the teachers to the basics of the DA,

what and how to prepare material for the DA, how to construct the DA questions, when to

conduct it, which student requires the DA, and how to conduct it.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 4

Literature Review

Many of the studies conducted about assessment approach the topic from the learners

perspective, while few articles approach the topic from the teachers perspective. In addition,

studies are mainly directed towards the different types of assessments and specifically towards

Diagnostic Assessment (DA). Review of literature was done to support the proposed DA

familiarization training.

An article by Blood (2011) and another by Dunn & Mulvenon (2009), both talking about

assessment from a different perspective. Diagnostic assessment is a dynamic process used to

assess, evaluate and give feedback to learners. Diagnosis, according to Blood (2011), is a term

familiar in the medical field but in second language acquisition, broadly speaking, diagnosis is

any form of assessment whether formative or summative that yields feedback. According to

Dunn & Mulvenon (2009), Although an assessment may be designed and packaged as a

formative or summative assessment, it is the actual methodology, data analysis, and use of the

results that determine whether an assessment is formative or summative (p.2). Blood also

emphasized on the post-assessment feedback to be ample, detailed and innovative.

The two authors emphasized the importance of the content of the assessment in revealing

more accurate data; Blood questioned whether the tasks used by the teacher to construct

assessment material are appropriate for a formative or a summative assessment, while Dunn &

Mulvenon questioned if the assessment will still be called formative if the teachers use it for

grading.

In conclusion, differentiation between the two forms of assessment (formative and

summative), should not be according to their nomenclature, the main points of differentiation

one should keep in mind are; where is the assessment used, when is it used and how is it used.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 5

Mentioning that the most important question teachers should keep in their mind is; what they are

going to do with the collected data? Answering this question will give the appropriate

description or name of the assessment.

In a study conducted in Colombia by Mendoza (2009), there was a difference between

teachers perception to and the use of assessment in the classroom and there is a tendency to use

traditional assessment instead of alternative assessment(p.55). These findings are confirmed by

the study of Tsagari (2016) about the assessment orientation of teachers in two Mediterranean

countries, where teachers concentrate on the traditional use of assessment with contents limited

to assessing vocabulary and grammar and lack creativity and higher order thinking. In both

articles, the assessment used by the teachers tends to be more summative than formative. In

addition, both articles recommend that teachers need more training in language assessment

despite the two articles studied teachers in different countries.

A study carried out by Mertler (2003), comparing the in-service teachers with the

preservice teachers (recent graduates) regarding assessment literacy. This article confirms

what Mendoza (2009) and Tsagari (2016) recommend in their studies; that teachers need more

training in regard to assessment. In addition, Mertler mentioned, Previous studies have reported

that teachers feel and actually are unprepared to adequately assess their students(p.22). Also, the

teachers believe that they did not have sufficient training to build up the skills in assessing the

learners (p.22). He compared in this study between the preservice and the in-service teachers in

regard many standards including learnersassessment and found that; in-service teachers scored

higher than preservice. Mertler questioned whether on-the-job (in the classroom) assessment

training is preferred than training during the undergraduate study. This study highlights the
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 6

importance of understanding the concept of assessment, training and experience that will be

gained within classroom implication.

Analysis

Needs Assessment

Identifying the problem. The DLI is aiming towards teachers adhering to student-

centered teaching methodology, with the teachers being facilitators that guide the students

through their learning process. Also, the teachers are required to prepare teaching materials and

activities to enhance learning, in addition to preparing an assessment material to diagnose areas

that need improvement for each learner. This should be followed by an interventional

instructional plan to overcome the difficulty. Being a trainer for the DA specialist certification

course, I have noticed that most of the participants lack the basic knowledge about the DA. This

problem has a direct negative influence on the outcome of the teaching-learning process, and

thus on achieving the organizational goal.

Learner Analysis

Identifying the target audience. Data collected from learners analysis was provided

from four sources; from the direct supervisors through their class observations and from follow-

up checks of teachers performance. The other source are direct interviews (sensing sessions)

conducted by the academic specialist and the dean with the teachers. The third source is the

questionnaire survey given to the teachers by the end of the DA certification course

(retrospective data collection). The last source is a questionnaire I conducted recently through

Survey Monkey (see Appendix A for the questionnaire). The responses collected from the survey

include suggestions from the participants that, if they had been enrolled in familiarization

training, it would have made their certification course more productive. Another issue is the
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 7

diversity of knowledge about the basic principles of the DA among the participants attending the

certification course. This varies from a novice with lack of knowledge to experts requiring

honing of their experience. The results of the survey conducted using Survey Monkey revealed

that around 30% of the responders have little information about DA, and 39% less often conduct

DA and another 39% moderate frequency. Also, around 50% less often been exposed to DA

report. These figures confirm that the audience varies between novice with lack of knowledge to

experts with the limited practice of DA. Collectively, all these analyses support the proposal of

conducting familiarization training that will cover the issues of DA, as well as set the solid

ground for the DA certification course, which is a long training process. In addition, such

training will be considered as a norming session for the certified DA specialists.

Context Analysis

Logistic information. This training is faced with a logistical problem of time constraints.

The time that will be allotted to the training, how it will affect the teaching hours of the teacher,

and if there are any drawbacks on the learning process should be taken into consideration. The

training for the diagnostic teaching is already being conducted at the DLI by faculty development

specialists over four hours, usually one morning or afternoon. The proposed DA familiarization

training is not a substitute for the DA specialist certification training. As the DA specialist

certification training requires 64 hours of face-to-face instructions in addition to assignments,

practicum and the submission of a final portfolio. This will increase the out-of-classroom time

for the instructors and thus their teaching hours which is part of the teachers standards.

During the proposed familiarization training, the trainees will not conduct DA for a

student as in the DA certification training. Therefore, it will start as blended with instructor-led

for the first iteration so as to pinpoint areas that require modification or adjustment. Then the
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 8

following iteration will be self-instructional, asynchronous e-learning delivered to the

participants designed on whatever platform is available at DLI. Also, instruction on how to

proceed in the training will be given as a Portable Document Format (PDF). In addition, the

designed model will include videotaped DA conducted with a volunteer. The time intended for

the blended course is 1-2 days and for the e-learning around 4-6 hours, but the latter will depend

on the instructors time as they can do the training at their pace.

This training will save time for the organization and for the instructor and will not require

a classroom or a conference room to conduct the training. The teachers can undertake the

training at their workplace and they can pause the training in order to attend a class or to perform

any other task. In addition, the availability of MacBook and IPads for each teacher in addition to

the high-speed internet will facilitate the use of the self-instruction and asynchronous training.

Content Analysis

The DA certification model consists of five modules. Two main modules can be used for

the familiarization training that contains the practical information required for the DA interview

process.

The first module will contain information related to selecting passages to be used

during the DA interview. The level of this material is according to the ILR

descriptors, so the audience should be familiar with these descriptors.

o These descriptors will be provided to the audience as an attachment of soft copy

or they go to the link http://govtilr.org/Skills/ILRscale1.htm.

o The audience will be provided with listening and reading sample passages at

different levels. The audience will assign a level of the passages and check their

selection with justification.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 9

o After selecting the level appropriate passage, the audience will learn how to

construct questions covering the global functionality, vocabulary, grammar and

sociocultural features. The sample of structured questions will be provided as an

attachment.

The second module will cover the DA interview for the four skills (speaking, reading,

and listening).

o The audience will be provided with the electronic forms of speaking, reading,

listening skill and for the recommendation.

o Filled forms will be provided for reference and to have an idea how to complete

the report and recommendation forms.

o In addition, there will be videos of a real interview the audience can watch.

Design

Learning Objectives

The proposed DA familiarization training is linked in concept to diagnostic teaching.

Diagnostic teaching is based on using diagnostic tools during the teaching-learning process in the

classroom, while the DA is mainly conducted outside the classroom. The benefit of the former is

the ability to identify the learners areas that require improvement within the teaching hour, and

that could be related to that specific hours contents or contexts. Also, it gives the teachers an

idea of their lesson plan, activities, teaching strategies, and what sort of modification can he

implement. As for the DA, the desired outcome is broader and gives an idea of what major issue

is taking place in the learning process of that particular learner. This DA is neither related to

specific teaching hour, nor to a certain topic. It gives an overall reflection of the learners

performance from the beginning of the course to the current time.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 10

The goals of this DA familiarization training include and are not limited to:

Increase awareness of the instructors about the importance of DA in revealing the

areas of strength and areas that need improvement at the current time for that

specific learner/student.

Explain to the teachers the importance of ILR levels and their implementation in

DA through a selection of level-appropriate texts.

Demonstrate to the instructors how to conduct DA interview for the three skills;

reading, listening and speaking.

Illustrate to the teachers how to fill and complete DA report forms for the three

skills and that for the recommendations.

The instructor at the end of this DA familiarization training will:

Identify and select level-appropriate reading and listening material following the

IRL descriptors as a guideline with 85% accuracy.

Create and construct DA questions, guided by the sample questions given, to use

them during the interview that will cover the global function, vocabulary,

grammatical, and socio-cultural features, with 85% accuracy.

Demonstrate knowledge of the DA protocol from memory, and the steps to be

taken into consideration before, during, and after the interview with 100%

accuracy.

Analyze the DA report written by the DA specialist, following the protocol as a

guideline with 85% accuracy.

Test Instruments

During the training, two types of test items will be used; declarative and procedural.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 11

From memory, the audience will be given a matching quiz. They will match the

ILR descriptive with the level with 100% accuracy.

Given the IRL descriptors, the audience will read or listen to a passage and will

assign a level to that passage with 85% accuracy.

From memory and from the handout, the audience will organize the steps of a DA

interview given to them randomly with 100% accuracy.

Watching a real video of a DA interview, the audience will fill the form for that

skill and compare with a filled report with 85% accuracy.

Participants will revive feedback in different ways; the matching activity will give an

immediate feedback upon failure to match correctly. Feedback for assigning a level for passages

(reading and listening) will be given when the participant selects an incorrect level accompanied

by a justification. For the organizing activity, being procedural, the participants will receive

feedback after submitting the answer.

Media and Delivery System Decision

Following the analysis, a suitable delivery mode for the participants is the blended both

instructor-led and self-instruction. Thus, the former will be synchronous and the latter will be

asynchronous. The instructor-led may be required in the beginning as a delivery mode, later on,

the delivery mode for the training will be self-instructed. This mode will enhance knowledge

acquisition, knowledge retention and problem-solving that are required for this training. In

addition, self-instruction mode will reduce the out-of-class hours, and is more appropriate

because the contents are relatively stable, and the training will be repeated more often. A good

example of the online use of DA is the Online Diagnostic Assessment(ODA). The purpose of this
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 12

ODA is to assess students reading and listening skills. The idea of the OAD training supports

the applicability of performing an online training for the teachers on how to perform the DA.

Participants will receive instructions on the same platform the training will be delivered.

There will be step-by-step instructions on the opening page in addition to a PDF file attached to

download and print if required. All the material will be available for the participants to access

through their computers, as each participant is possessing a MacBook that contains the Mac side

and virtual window side, in case the participants are not familiar with Mac. I will consider using

Adobe Captivate in designing this training.At the DLI, the Universal Curriculum and Assessment

Tool (UCAT) is becoming popular as a Learning Management System (LMS). The UCAT can be

accessed online or it can be used as standalone, where the participants can download the training

and access it from anywhere. There will be certain material that will be provided using a

different software as Movie Media Player for pre-recorded videos.

Instructional Strategies and Activities

This training module follows the basics of the ADDIE model, the analysis is performed

for both the organization and learners(teachers). The design of the training module consists of

different sections; one section presents the ILR descriptors and activities related to them like

matching, assigning level for passages and constructing questions for the passages. The other

section of the module includes an introduction to DA interview protocol, the interview report

forms, step by step instruction on how to complete these forms. The last section will be a mock

DA interview to be assessed by the participant as a practice. Some of the material used to

develop this training is adapted from previous ILR training and from training designed for DA

certification. The adapted material is modified for online training purpose.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 13

Development and Implementation

The training material will include material previously developed in addition to newly

developed or created ones.

List of Major Deliverables

The proposal for the training depends upon needs assessment collected from four sources.

The Survey Monkey questionnaire was created and delivered to the faculty after taking all the

official permissions. The deliverable will include readings for the ILR and also readings about

the Modes for text typology. In addition, using google doc, I will create multiple choice

questions to check the trainees knowledge about ILR level descriptors. Also, I will prepare

level-appropriate texts for the trainees to rate. Then I will create a text-rating matching activity

for both reading and listening skills. Finally, I will upload electronic form so that the trainees

will be able to download and use in the future. Moving from design to development, I should

consider the applicability of the activities in the delivery moodle and the issue of navigation

between the pages. Also, I should keep in mind how to the feedback will be accessed in the

development phase.

Development Plan

This is a requirement for the master's degree in the MIST program, so the time constraint

is the timeline given to us by the program and I will try to complete it on time. The only cost will

be the required programs needed for the development of this model. Developing this module will

require more knowledge that I am expecting to gain during the coming terms both theoretical and

technical in addition to advisory support.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 14

Implementation Plan

This training is planned to be completed online as self-instructed. To start with, it might

be appropriate to conduct Beta Test where the facilitators and the SMEs take the training in

order to test the contents and how they are connected to each other. In addition, they can give a

recommendation on applicability and timing, also, any technical issue will be addressed. This is

followed by Pilot Test where a number of targeted audience will take the training in a

classroom or conference room under the supervision of the trainer/facilitator. The facilitators,

preferably certified DA specialist and those undertook the training during the Beta Test, with

knowledge in the field in order to answer any question that the trainees may raise related to DA

or to a technical issue related to the training module.

Evaluation

Formative Evaluation

During the training, the trainees will have different forms of activities that they require to

submit in order to get feedback. These activities are constructed on Google forms, and thus when

submitting the form, results will appear to the trainer or facilitator and will give an idea about the

completeness of the training and how accurate was the performance of the trainee. In order to

have the best out of this, the designer will share Google forms with the trainees facilitators and

supervisor.

The design of the training is to be used by different trainees and at different occasions, so

to evaluate the usability of the training module, the trainee will answer evaluation questionnaire

(see Appendix B for the questionnaire). Also, looking at the training records will give an idea

about the compliance, that is whether the training material has been implemented as designed or

not.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 15

Summative Evaluation

The trainee will answer a Kirkpatrick Level 1 (reaction ) evaluation questionnaire (see

Appendix C for the questionnaire). This evaluation will give the opportunity to have a feedback

on certain areas that require revision in the future. Level 2 (learning) evaluation will include test

instrument like matching ILR activities, assigning a rate to a reading or listening passage, and

filling a DA report after watching a mock DA interview. The facilitator will follow up on the

learners performance and whether the knowledge has been transformed and retained by the

learner. In Level 3 (application) evaluation, the trainees will apply the information they gained

in the classroom and in their job. This evaluation can be recognized by their direct supervisors

through reflection at their counseling reports and on students performance. As for evaluation

level 4 and 5, they are beyond the scope of this project.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 16

References

Alderson, J. C. (2004). Diagnosing Foreign Language Proficiency. pp.11-12 London:

Continuum.

Blood, I. (2011).Diagnostic second language assessment in the classroom. Columbia

University academic commons. Working Papers in TESOL & Applied Linguistics, 11(1),

57-58. Retrieved from https://doi.org/10.7916/D8ZG6RTP.

Defense Language Institute [Website]. Retrieved from

http://www.dliflc.edu/about/mission-vision/

Dunn, K. and Mulvenon, S. (2009). A critical review of research on formative assessment: the

limited scientific evidence of the impact of formative assessment in education. Practical

Assessment Research & Evaluation,14(7). Retrieved from

http://pareonline.net/getvn.asp?v=14&n=7

Hanna, G. S., & Dettmer, P. A. (2004). Assessment for effective teaching: Using context-

adaptive planning. Boston, MA: Pearson A&B.

King, A. (1993). From Sage on the Stage to Guide on the Side. College Teaching, 41(1),

30-35.

Mendoza, A. & Arandia, R. (2009). Language testing in Colombia: a call for more teacher

education and teacher training in language assessment. Profile Issues in Teachers`

Professional Development, 11(2), 55-70. Retrieved from

http://www.scielo.org.co/scielo.php?script=sci_arttext&pid=S1657-

07902009000200005&lng=en&tlng=en.

Mertler, C. (2003). Preservice versus inservice teachers' assessment literacy: Does classroom

experience make a difference?. ERIC. Retrieved from https://eric.ed.gov/?id=ED482277

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 17

Piskurich, G., M. (2015).Rapid instructional design: Learning ID fast and right, Hoboken, NJ:

John Wiley & Sons, Inc.

Tsagari, D.(2016). Assessment orientations of state primary EFL teachers in two

Mediterranean Countries. Center for Educational Policy Studies Journal / CEPS Journal,

6(1), 9-30. Retrieved from https://ojs.cepsj.si/index.php/cepsj/article/view/102

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 18

Appendix A

Survey Monkey Questionnaire

1 little 3 moderate 5 a lot

1. How much information do you know about DA? 1 3 5

2. How often do you conduct DA? 1 3 5

3. How often do you read a DA report? 1 3 5

4. How familiar are you with ILR levels? 1 3 5

5. How confident are you in selecting level appropriate reading

1 3 5
passages?
6. How confident are you in selecting level appropriate listening
1 3 5
passages?
7. How confident are you in constructing level-appropriate questions
1 3 5
for the selected passages?
8. How familiar are you with the principles and structure of OPI (oral
1 3 5
proficiency interview)?

9. How confident are you in using computers? 1 3 5

10. How familiar are you with using different software to record and edit
1 3 5
audios?

https://www.surveymonkey.com/r/CBB9SR2
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 19

Appendix B
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 20

Appendix C

Level 1 Evaluation