Running head: DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

Analysis of the Defense Language Institute ICPT 101 & 102

Listening Comprehension Level 1+ for Spanish

Team Short Circuit:
Alex Vlez, David Collins, Sungwoong Kim, and Kelley Buell
MIST California State University, Monterey Bay







IST 522 Assessment and Evaluation
Professor Bude Su, Ph.D.
July 8, 2014
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

2
Table of Contents

Introduction ............................................................................................................................................ 3
Purpose ..................................................................................................................................................... 5
Methods .................................................................................................................................................... 6
Results ....................................................................................................................................................... 7
Analysis ..................................................................................................................................................... 8
Discussion ................................................................................................................................................ 9
Recommendations .............................................................................................................................. 10
REFERENCES .......................................................................................................................................... 12
Appendix A ............................................................................................................................................ 13

DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

3
Introduction
Students learning Spanish at the Defense Language Institute Foreign Language Center
(DLIFLC) take five standardized tests, including a final test in their target language. DLIFLC
follows the Interagency Language Roundtable (ILR) standards, which range from level 0 to level
5, 0 being no proficiency and 5 being functionally native proficiency (See Appendix A). ILR is
...an unfunded Federal interagency organization established for the coordination and sharing of
information about language-related activities at the Federal level
(www.govtilr.org/IRL%20History.htm). According to the DLIFLC webpage (2014), DLIFLC
...develops standardized in-course language tests and provides related reporting, test
administration support, analysis, and consultation to promote continuous improvement in
teaching and learning in DLIFLCs resident Basic Course language programs (Test
Management Division section). The In-Course Proficiency Tests (ICPTs), are developed by the
...Test Development Division of the Directorate of Evaluation and Standardization, DLIFLC is
the largest standardized foreign language test development organization in the United States
(Test Management Division section).
ICPT is a prochievement test which includes features of both proficiency and
achievement. ICPT was introduced to bridge the gap between unit tests and the final test, the
Defense Language Proficiency Test (DLPT). Unit tests are achievement tests designed to be
taken after completing each unit, consisting of three to four chapters of the students
textbook. The purpose of achievement testing is to determine an examinees acquisition of
various aspects of course content, usually after a specific part of the curriculum has been
covered. However, DLPT is a proficiency test that does not cover a specified curriculum. The
purpose of language proficiency testing is to assess how well the students use the target language
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

4
in real-life situations. In contrast to achievement testing, proficiency testing focuses on overall
language competence without regard to the place, length of time, or manner in which that
competence has been acquired. For this reason, the unit test results do not indicate student
language proficiency level sufficiently.
According to DLIFLCs website (2014):
The content domains of these ICPTs are representative samples of the domains of
expected student cumulative language mastery in Basic Course instruction. Performance
standards of the individual ICPTs are keyed to language mastery targets of specific
subsets of the Basic Course that are identified by course titles and numbers within the
Basic Course Transcript. Timing for administration of these tests is likewise keyed to the
scheduled completion of each course subset. ICPTs test either global language listening
and reading proficiency development or Final Learning Objective (FLO) job-related
adjunct language performance skills.
The ICPT is used to measure the students achievement in reading and listening
comprehension prior to completing the DLPT, which is the final standardized test that students
have to pass to successfully graduate from DLIFLC. The ICPT mimics the DLPT in all areas and
aims to scaffold students from the lowest level of achievement, ILR level 0, to the highest
possible level believed to be attainable during their stay in DLIFLC, ILR level 3. Prior to taking
the DLPT, the ICPT is conducted five times and tests are identified as 101, 102, 201, 202, and
301. Those who achieve ILR levels 2, 2, 1+ in listening, reading, speaking respectively on the
DLPT, can graduate and become a military linguist. It is therefore imperative to monitor
students achievement through the ICPT series in order to ensure they are making adequate
progress toward their learning goals.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

5
Purpose
The purpose of this study is to analyze a group of fourteen students listening
comprehension performance at the ILR Level 1+ for ICPT 101 and 102. By comparing two
succeeding ICPT results, teaching teams can determine whether students language proficiency is
improving or not. They can then evaluate their instruction and curriculum according to the ICPT
results. In addition, students can find out their language proficiency level and determine the
areas they need to improve in. A comparison of the percent correct answers on the level 1+
questions between ICPT 101 and 102 listening tests will show whether or not those students
improved their target language listening proficiency over the month. Those students who receive
high scores on the ICPT can become more confident they will be successful on the DLPT.
Nine out of ten cases among those who fail to pass DLPT is because they fail to achieve
a level 2 in DLPT listening, although they pass level 2 and 1+ in DLPT reading and speaking. In
other words, it is more difficult to achieve a level 2 in listening than achieving a level 2 in
reading and 1+ in speaking. Students who are strong in listening but weak in reading and
speaking will have a better chance to pass DLPT than those who are strong in reading and
speaking but weak in listening. Considering level 2 in listening is the passing DLPT score,
measuring the number of the correct answers of level 1+ questions is important because it will
allow instructors to intervene early on if further support and study is needed.
Compared to level 0 and level 1 questions, level 1+ consists of high proficiency related
questions. Students who correctly answer more level 1+ questions indicate higher language
proficiency. Level 1+ is not level 1.5. Level 1+ describes language characteristic of the next
higher level but not sustained at that level. In other words, level 1+ is closer to level 1.9 which is
very close to level 2. For this reason, it is important to compare the number of the correct
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

6
answers of level 1+ questions between ICPT 101 and 102 listening tests to determine of if
student proficiency is adequately improving.

Methods
ICPTs typically consist of 30 to 45 test items which are administered within a two hour
time period. Due to confidentiality, the actual test items are classified and can not be disclosed.
However, it can be revealed that the questions consist of multiple-choice, short answers,
translation/transcription, and paragraph summaries. ICPT testing material topics cover broad
areas such as economy, art, medical, politics, and so on. The questions are designed to measure
student language proficiency in real life situations such as understanding conversations,
answering questions, and finding directions. The tests are produced in paper-and-pencil and
computer-delivered versions, with computer-delivered versions superseding the paper-and-pencil
versions when test lab scheduling permits. This way, the answers are electronically recorded to
ensure accuracy.
The study group of fourteen students took the Spanish ICPT 101 for listening
comprehension on February 28, 2013. Of the 40 items on this test, 17 assessed level 0+
proficiency, 18 assessed level 1 proficiency, and 5 questions assessed the targeted level 1+
proficiency. After a month of training and instruction, the same students took ICPT 102 for
listening comprehension on March 28, 2013. This test was composed of 30 total test items, 5
assessing level 0+ proficiency, 16 assessing level 1 proficiency, 8 assessing the target level 1+
proficiency, and 1 assessing level 2 proficiency. Therefore, the target level 1+ for listening
comprehension consisted of 5 test items on ICPT 101 and 8 test items on ICPT 102.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

7
The tests are scored and computed by the DLIFLC testing department. The results are
then sent back to the teaching team leader electronically within a week of the original test date.
The score is calculated as both a numerical number and a percentage of the items that were
correctly completed by the examinees. Due to the differing number of test items for level 1+
between ICPT 101 and 102, only the percent correct scores are used for this analysis to ensure
equity in comparison. For this group of students, the mean for level 1+ corrected questions is
56% on ICPT 101 listening comprehension and 61% on ICPT 102 listening comprehension.

Results
ICPT Spanish Listening Test (ILR Level 1+)
ICPT 101C Test Date (2/28/2013) ICPT 102C Test Date (3/28/2013)
Student ID Percent Correct Percent Correct
1 60 50
2 60 50
3 80 50
4 60 38
5 60 63
6 40 25
7 60 63
8 40 63
9 60 75
10 60 50
11 80 75
12 60 100
13 60 88
14 0 63
Table 1: ICPT Spanish Listening Percentages


DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

8
t-Test: Paired Two Sample for Means


Variable 1 Variable 2
Mean 55.71428571 60.92857143
Variance 380.2197802 382.0714286
Observations 14 14
Pearson Correlation 0.124264262
Hypothesized Mean Difference 0
df 13

t Stat
-
0.755112673
P(T<=t) one-tail 0.231821552
t Critical one-tail 1.770933396
P(T<=t) two-tail 0.463643104

t Critical two-tail 2.160368656


Table 2: ICPT Spanish Listening t-Test Results

The hypotheses are as follows; H
0
:
1
>=
2
and H
1
:
1
<
2
, where
1
and
2
denote
respectively the population means for the 101 and 102 test score percentages for ILR level 1+ in
listening comprehension. The test subjects are paired; therefore the t-Test for paired two sample
means is used. The null hypothesis is directional, so the one-tailed t-Test value calculation is
observed. The conventional alpha level is set at 0.05 and the degree of freedom (df) is 13.

Analysis
In comparing the absolute value of the t-stat with the one-tail t-critical value, it is
apparent that the t-Stat result of |-0.76| is much smaller than the one tailed t-critical value of 1.77
(0.76 < 1.77). Therefore, the null hypothesis cannot be rejected. Furthermore, the calculated
one-tailed p-value of 0.23 is much larger than the conventional alpha level of 0.05 (0.23>0.05).
This observation provides further evidence that the null hypothesis cannot be rejected.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

9
After review of the statistical analysis, it can be concluded that
2
is not statistically
greater than
1
; therefore, the data does not support the research hypothesis of
1
<
2
. Based on
this result, it is concluded that students did not achieve higher listening proficiency on the level
1+ questions between ICPT 102 than ICPT 101. In other words, after one month language
instruction, students Spanish listening proficiency has not improved significantly.

Discussion
Measuring the language proficiency of a group of individuals is a challenging process,
even though ICPT is designed to produce two specific data sets, which are achievement and
proficiency. Considering the number of questions (30 to 45 questions depending on which test)
within an ICPT listening test, the number of questions may hold limitations that can possibly
produce inaccurate data regarding learning results for real world situations. In other words, the
reliability of ICPT may not be consistent and may not accurately reflect the actual student
language proficiency.
For this study, only level 1+ proficiency was analyzed due to the fact that this particular
level is the precursor of level 2, which is the minimum needed to pass the DLPT. IPCT 101 only
had five test items related to level 1+ and ICPT 102 only had eight. This is a very small number
of questions from which to base an assessment on whether or not someone is proficient in
Spanish listening comprehension. However, because of the rigor and standardization used in the
development of these tests by DLIFLC, it may be assumed that the number of test items used to
assess level 1+ are an accurate measure of proficiency. At the end of the program, students are
required to enhance their skill level and acquire the target language proficiency through new
instructional materials that have been proven to reflect these expectations.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

10
Another issue with the assessment items used is that DLIFLC has strict confidentiality
measures and the actual test items can not be disclosed. Ultimately, there is uncertainty as to
which types of questions were measured according to the data reviewed. Certainly, it is easier to
guess a correct answer on a multiple-choice question than to produce a correct answer on a short
answer or paragraph summary when ones comprehension is not at the level of the subject
matter. It is unknown what type of questions were on the test, which could make it difficult to
measure the results of one test to the following test. For example, if ICPT 101 had five multiple-
choice and ICPT 102 had five multiple choice, two short answer, and a paragraph summary, it
may be more difficult to determine the level of correlation between ICPT 102 and ICPT
101. This concept could potentially affect the ability for students to show improvement from
ICPT 101 to ICPT 102 in level 1+. However, DLIFLCs tests are standardized and have been
created by the Test Review and Education Division of the Directorate of Evaluation and
Standardization. It can therefore be assumed that the testing items have been evaluated
thoroughly and accurately reflect the students listening comprehension proficiency.

Recommendations
After careful review and consideration, it is recommended to use a holistic approach
when evaluating the early stages of language proficiency related to classroom learning. The
teaching team should use caution in their interpretation of ICPT 101 and 102 results because
student language proficiency may have not been accurately measured by the limited test
questions or the data analysis tool (measurement). In regards to the target audience and the early
stages of their learning, language test results such as ICPT 101 and 102 should not be used to
predict the final outcomes of DLPT. In other words, teachers need to remember that
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

11
students will increase their skill level and language proficiency with time and their ability to
complete assigned tasks such as study guides etc. However, when it is noted that students have
not improved in the early stages, it may be necessary to provide interventions that will enhance
retention and transfer for later tests.
ICPT results can be used to inform the content and materials used in the teaching teams
daily lesson plans, especially in the early stages. For example, teaching teams may incorporate
an increased amount of authentic materials such as news, drama, and radio in their classroom
instructions. Also, the test results can be used to prepare and provide tailored instructions for
those who received lower scores on ICPTs. Low proficiency students will be focused by one-on-
one instruction in order to increase their language proficiency. Results can also be used to
reshuffle students between classes in order to create a balance in the student proficiency level
ratio. Teaching teams should avoid placing low proficiency students together in same classroom
because peer role modeling is important in language learning. In addition to these strategies,
Voluntary Special Assistance is a form of extra help that is provided at the last hour hour of
classes every day between 3:00pm and 3:50pm. Those students can benefit from that extra help
but the teaching team would have to recommend it by an official counseling.
Although, the mean scores showed a small increase from ICPT 101 to ICPT 102, the
concluding evidence shows that there is no statistical increase between these tests. It is therefore
recommended that the data from this analysis is used to inform the teaching team that this group
of students will require further support and intervention strategies before they can achieve level 2
proficiency in listening comprehension. Early intervention will help to ensure success for these
students so that they may graduate from DLIFLC as military Spanish linguists.

DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

12
REFERENCES

Defense Language Institute Foreign Language Center (2014). In-Course Proficiency Test Development
Division. Retrieved on June 21, 2014 from: http://www.dliflc.edu/in-courseprofici.html

Interagency Language Roundtable (2014). Interagency Language Roundtable Language Skill Level
Descriptions Listening. Retrieved on June 21, 2014 from:
http://www.govtilr.org/Skills/ILRscale3.htm


DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

13
Appendix A

The following are the definitions of the Listening Comprehension levels according to the ILR
webpage (http://www.govtilr.org/skills/ILRscale3.htm, 2014).
Listening 0 (No Proficiency) No practical understanding of the spoken language. Understanding
is limited to occasional isolated words with essentially no ability to comprehend communication.
Listening 0+ (Memorized Proficiency) Sufficient comprehension to understand a number of
memorized utterances in areas of immediate needs. Slight increase in utterance length
understood but requires frequent long pauses between understood phrases and repeated requests
on the listener's part for repetition. Understands with reasonable accuracy only when this
involves short memorized utterances or formulae. Utterances understood are relatively short in
length. Misunderstandings arise due to ignoring or inaccurately hearing sounds or word endings
(both inflectional and non-inflectional), distorting the original meaning. Can understand only
with difficulty even such people as teachers who are used to speaking with non-native speakers.
Can understand best those statements where context strongly supports the utterance's meaning.
Gets some main ideas.
Listening 1 (Elementary Proficiency) Sufficient comprehension to understand utterances about
basic survival needs and minimum courtesy and travel requirements in areas of immediate need
or on very familiar topics, can understand simple questions and answers, simple statements and
very simple face-to-face conversations in a standard dialect. These must often be delivered more
clearly than normal at a rate slower than normal with frequent repetitions or paraphrase (that is,
by a native used to dealing with foreigners). Once learned, these sentences can be varied for
similar level vocabulary and grammar and still be understood. In the majority of utterances,
misunderstandings arise due to overlooked or misunderstood syntax and other grammatical clues.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

14
Comprehension vocabulary inadequate to understand anything but the most elementary needs.
Strong interference from the candidate's native language occurs. Little precision in the
information understood owing to the tentative state of passive grammar and lack of vocabulary.
Comprehension areas include basic needs such as: meals, lodging, transportation, time and
simple directions (including both route instructions and orders from customs officials,
policemen, etc.). Understands main ideas.
Listening 1+ (Elementary Proficiency, Plus) Sufficient comprehension to understand short
conversations about all survival needs and limited social demands. Developing flexibility evident
in understanding a range of circumstances beyond immediate survival needs. Shows spontaneity
in understanding by speed, although consistency of understanding is uneven. Limited vocabulary
range necessitates repetition for understanding. Understands more common time forms and most
question forms, some word order patterns, but miscommunication still occurs with more complex
patterns. Cannot sustain understanding of coherent structures in longer utterances or in
unfamiliar situations. Understanding of descriptions and the giving of precise information is
limited. Aware of basic cohesive features (e.g., pronouns, verb inflections) but many are
unreliably understood, especially if less immediate in reference. Understanding is largely limited
to a series of short, discrete utterances. Still has to ask for utterances to be repeated. Some ability
to understand facts.
Listening 2 (Limited Working Proficiency) Sufficient comprehension to understand
conversations on routine social demands and limited job requirements. Able to understand face-
to-face speech in a standard dialect, delivered at a normal rate with some repetition and
rewording, by a native speaker not used to dealing with foreigners, about everyday topics,
common personal and family news, well-known current events and routine office matters
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

15
through descriptions and narration about current, past and future events; can follow essential
points of discussion or speech at an elementary level on topics in his/her special professional
field. Only understands occasional words and phrases of statements made in unfavorable
conditions, for example through loudspeakers outdoors. Understands factual content. Native
language causes less interference in listening comprehension. Able to understand facts; i.e., the
lines but not between or beyond the lines.
Listening 2+ (Limited Working Proficiency, Plus) Sufficient comprehension to understand
most routine social demands and most conversations on work requirements as well as some
discussions on concrete topics related to particular interests and special fields of competence.
Often shows remarkable ability and ease of understanding, but under tension or pressure may
break down. Candidate may display weakness or deficiency due to inadequate vocabulary base
or less than secure knowledge of grammar and syntax. Normally understands general vocabulary
with some hesitant understanding of everyday vocabulary still evident. Can sometimes detect
emotional overtones. Some ability to understand implications.
Listening 3 (General Professional Proficiency) Able to understand the essentials of all speech
in a standard dialect including technical discussions within a special field. Has effective
understanding of face-to-face speech, delivered with normal clarity and speed in a standard
dialect on general topics and areas of special interest; understands hypothesizing and supported
opinions. Has broad enough vocabulary that rarely has to ask for paraphrasing or explanation.
Can follow accurately the essentials of conversations between educated native speakers,
reasonably clear telephone calls, radio broadcasts, news stories similar to wire service reports,
oral reports, some oral technical reports and public addresses on non-technical subjects; can
understand without difficulty all forms of standard speech concerning a special professional
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

16
field. Does not understand native speakers if they speak very quickly or use some slang or
dialect. Can often detect emotional overtones. Can understand implications. (Has been coded L-3
in some nonautomated applications.) [Data Code 30]
Listening 3+ (General Professional Proficiency, Plus) Comprehends most of the content and
intent of a variety of forms and styles of speech pertinent to professional needs, as well as
general topics and social conversation. Ability to comprehend many sociolinguistic and cultural
references. However, may miss some subtleties and nuances. Increased ability to comprehend
unusually complex structures in lengthy utterances and to comprehend many distinctions in
language tailored for different audiences. Increased ability to understand native speakers talking
quickly, using nonstandard dialect or slang; however, comprehension is not complete. Can
discern some relationships among sophisticated listening materials in the context of broad
experience. Can follow some unpredictable turns of thought readily, for example, in informal and
formal speeches covering editorial, conjectural and literary material in subject matter areas
directed to the general listener.
Listening 4 (Advanced Professional Proficiency) Able to understand all forms and styles of
speech pertinent to professional needs. Able to understand fully all speech with extensive and
precise vocabulary, subtleties and nuances in all standard dialects on any subject relevant to
professional needs within the range of his/her experience, including social conversations; all
intelligible broadcasts and telephone calls; and many kinds of technical discussions and
discourse. Understands language specifically tailored (including persuasion, representation,
counseling and negotiating) to different audiences. Able to understand the essentials of speech in
some non-standard dialects. Has difficulty in understanding extreme dialect and slang, also in
understanding speech in unfavorable conditions, for example through bad loudspeakers outdoors.
DATA ANALYSIS DLIFLC LC 1+ ICPT 101 & ICPT 102

17
Can discern relationships among sophisticated listening materials in the context of broad
experience. Can follow unpredictable turns of thought readily, for example, in informal and
formal speeches covering editorial, conjectural and literary material in any subject matter
directed to the general listener.
Listening 4+ (Advanced Professional Proficiency, Plus) Increased ability to understand
extremely difficult and abstract speech as well as ability to understand all forms and styles of
speech pertinent to professional needs, including social conversations. Increased ability to
comprehend native speakers using extreme nonstandard dialects and slang, as well as to
understand speech in unfavorable conditions. Strong sensitivity to sociolinguistic and cultural
references. Accuracy is close to that of the well-educated native listener but still not equivalent.
(Has been coded L-4+ in some nonautomated applications.) [Data Code 46]
Listening 5 (Functionally Native Proficiency) Comprehension equivalent to that of the well-
educated native listener. Able to understand fully all forms and styles of speech intelligible to the
well-educated native listener, including a number of regional and illiterate dialects, highly
colloquial speech and conversations and discourse distorted by marked interference from other
noise. Able to understand how natives think as they create discourse. Able to understand
extremely difficult and abstract speech.