TECHNICAL REPORT

ALGEBRA I, BIOLOGY, AND LITERATURE

2013

Provided by Data Recognition Corporation

 Table of Contents

TABLE OF CONTENTS
Glossary of Common Terms ....................................................................................................................... i
Preface: An Overview of the Assessments ............................................................................................... vii
The Keystone Exams from 2008 to Present ............................................................................................... vii
Assessment Activities Occurring from 2010 to Present............................................................................ viii
Chapter One: Background of the Keystone Exams ..................................................................................... 1
Assessment History in Pennsylvania ............................................................................................................1
The Keystone Exams ....................................................................................................................................1
Chapter Two: Test Development Overview of the Keystone Exams ........................................................... 5
Keystone Blueprint/Assessment Anchors and Eligible Content ..................................................................5
High-Level Test Design Considerations ........................................................................................................7
Online Testing Design Considerations .........................................................................................................8
Algebra I .......................................................................................................................................................9
Biology....................................................................................................................................................... 11
Literature .................................................................................................................................................. 13
Literature Passages ................................................................................................................................... 14
Chapter Three: Item and Test Development Processes ............................................................................ 17
General Keystone Test Development Processes ...................................................................................... 17
General Test Definition ............................................................................................................................. 18
Algebra I Test Definitions .......................................................................................................................... 18
Biology Test Definitions ............................................................................................................................ 20
Literature Test Definitions ........................................................................................................................ 23
Item Development Considerations ........................................................................................................... 25
Item and Test Development Cycle ............................................................................................................ 27
General Item and Test Development Process .......................................................................................... 30
Chapter Four: Universal Design Procedures Applied to the Keystone Exams Test Development Process . 35
Universal Design........................................................................................................................................ 35
Elements of Universally Designed Assessments ....................................................................................... 35
Guidelines for Universally Designed Items ............................................................................................... 37
Item Development .................................................................................................................................... 38
Item Format .............................................................................................................................................. 39
Assessment Accommodations .................................................................................................................. 40

2013 Keystone Exams Technical Report

 Table of Contents

Chapter Five: Field Test Leading to the Spring 2013 Core ......................................................................... 41
Field Test Overview ................................................................................................................................... 41
Spring 2011 Keystone Exams Embedded Field Test ................................................................................. 41
Statistical Analyses and Results ................................................................................................................ 45
Review of Items with Data ........................................................................................................................ 49
Chapter Six: Operational Forms Construction for 2013 Administrations .................................................. 51
Final Selection of Items and Keystone Forms Construction ..................................................................... 51
Special Forms Used with the Operational 2013 Keystone Exams ............................................................ 52
Chapter Seven: Test Administration Procedures ...................................................................................... 57
Sections, Sessions, Timing, and Layout of the Keystone Exams ............................................................... 57
Sections and Sessions ............................................................................................................................... 57
Timing ....................................................................................................................................................... 58
Layout........................................................................................................................................................ 60
Shipping, Packaging, and Delivery of Materials ........................................................................................ 61
Chapter Eight: Processing and Scoring ..................................................................................................... 63
Receipt of Materials .................................................................................................................................. 63
Scanning of Materials ............................................................................................................................... 64
Materials Storage ...................................................................................................................................... 66
Scoring Multiple-Choice Items .................................................................................................................. 67
Rangefinding ............................................................................................................................................. 67
Scorer Recruitment and Qualifications ..................................................................................................... 68
Leadership Recruitment and Qualifications.............................................................................................. 68
Training ..................................................................................................................................................... 69
Handscoring Process ................................................................................................................................. 70
Handscoring Validity Process .................................................................................................................... 70
Quality Control .......................................................................................................................................... 72
Chapter Nine: Description of Data Sources .............................................................................................. 79
Student Filtering Criteria ........................................................................................................................... 79
Key Verification Data ................................................................................................................................ 80
Calibration of Operational Test Data ........................................................................................................ 80
Final Data .................................................................................................................................................. 80
Spiraling of Forms ..................................................................................................................................... 81

2013 Keystone Exams Technical Report

 Table of Contents

Chapter Ten: Summary Demographic and Accommodation Data for Spring 2013 Keystone Exams .......... 83
Assessed Students..................................................................................................................................... 83
Reasons for Student Non-Assessment ...................................................................................................... 85
Demographic Characteristics of Students Receiving Test Scores ............................................................. 87
Test Accommodations Provided ............................................................................................................... 94
Glossary of Accommodation Terms ........................................................................................................ 112
Chapter Eleven: Classical Item Statistics ................................................................................................ 117
Item-Level Statistics ................................................................................................................................ 117
Item Difficulty ......................................................................................................................................... 117
Item Discrimination................................................................................................................................. 118
Scatter Plots of Item Discrimination and Difficulty ................................................................................. 119
Observations and Interpretations ........................................................................................................... 124
Chapter Twelve: Rasch Item Calibration ................................................................................................ 127
Description of the Rasch Model .............................................................................................................. 127
Checking Rasch Assumptions .................................................................................................................. 128
Rasch Item Statistics ............................................................................................................................... 132
Chapter Thirteen: Standard Setting ....................................................................................................... 145
Standard Setting and Performance Level Descriptors ............................................................................ 145
Development Overview for the Performance Level Descriptors ............................................................ 145
Performance Level Descriptors Meeting 1 ............................................................................................. 146
Performance Level Descriptors Meeting 2 ............................................................................................. 149
Standard Setting ..................................................................................................................................... 152
Chapter Fourteen: Scaling ..................................................................................................................... 167
Raw Scores to Rasch Ability Estimates.................................................................................................... 167
Rasch Ability Estimates to Scaled Scores ................................................................................................ 168
Raw-to-Scaled Score Tables .................................................................................................................... 170
Chapter Fifteen: Equating ..................................................................................................................... 171
Pre- vs. Post-Equating ............................................................................................................................. 171
Equating Design for Keystone Exams ...................................................................................................... 172
Post-Equating Check Analyses ................................................................................................................ 172
Equating for the Embedded Field Test Items.......................................................................................... 178

2013 Keystone Exams Technical Report