Presentation to the Delaware State Board of Education

Brian M. Touchette
Office of Assessment
December 18, 2014

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common Core

State Standards. They are designed to let teachers and parents know whether
students are on track to be college- and career-ready by the time they
graduate.
Because the new standards set higher expectations for students--and the new

tests are designed to assess student performance against these higher

expectations--our definition of grade level performance is higher than it used to
be.
As a result, it means that fewer students will meet grade level standards,

especially for the first few years. Results should improve as students have
additional years of instruction aligned to the new standards and become better
equipped to meet the challenges they present.
This does not mean that our students are doing worse than they did last year.

Rather, the scores represent a new baseline that provides a more accurate
indicator for educators, students, and parents as they work to meet the rigorous
demands of college and career readiness.

Achievement Level Timeline (2014)

April 30

October 6-18
October 13-19

States Approved Achievement Level Setting

Plan
Online Panel
In-Person Panel (Dallas)

October 20

Cross-Grade Review Committee (Dallas)

October 30

Technical Advisory Committee Review

November 14
Fall/Winter

States vote to approve Achievement Level

Recommendations
State Adoption of Achievement Levels

Achievement Level Timeline (2014)

April 30

October 6-18
October 13-19

States Approved Achievement Level Setting

Plan
Online Panel
In-Person Panel (Dallas)

October 20

Cross-Grade Review Committee (Dallas)

October 30

Technical Advisory Committee Review

November 14
Fall/Winter

States vote to approve Achievement Level

Recommendations
State Adoption of Achievement Levels

Online Panel
Crowd sourcing thousands of participants
Recommendations broken down by subgroup (teachers,

higher education, etc.)

The process involved:
Online orientation
Review of test questions
Recommended a cut score for the threshold of Performance

Level 3
142 Delawareans participated (exceeded SBACs target)

Three Cut Scores Defines Four

Performance Levels

Reviewing Items in the Ordered Item Booklet

(OIB) for Bookmarking
1. What do you know about a
student who responds
successfully to this item; that is,
what skills must a student have in
order to know the correct answer?

2. What makes this item

more difficult than
preceding items?

This activity helps

you acquire the
knowledge to make
content-based cut
score
recommendations
6

Selection of Placement of Bookmark

A threshold Level 3 student would have about a 50% chance

of success on the item just before the bookmark

A threshold Level 3 student would have less than a 50%
likelihood of success on the items from the bookmark on

Achievement Level Timeline (2014)

April 30

October 6-18
October 13-19

States Approved Achievement Level Setting

Plan
Online Panel
In-Person Panel (Dallas)

October 20

Cross-Grade Review Committee (Dallas)

October 30

Technical Advisory Committee Review

November 14
Fall/Winter

States vote to approve Achievement Level

Recommendations
State Adoption of Achievement Levels

In-Person Panel
K-12 Educators, Higher Education, parents, and community

advocates brought together

Recommended cut scores for all Performance Levels
500 participants in grade/subject panels
Nominated by states
25 Delawareans participated
Used Bookmark procedure
Three Rounds in the process

Reviewing Items in the Ordered Item Booklet

(OIB) for Bookmarking
1. What do you know about a
student who responds
successfully to this item; that is,
what skills must a student have in
order to know the correct answer?

2. What makes this item

more difficult than
preceding items?

This activity helps

you acquire the
knowledge to make
content-based cut
score
recommendations
10

Three Rounds to Determine Three Cut

Scores

11

In-Person Panel Agenda

Day 1
Morning: Study Common Core State Standards and
Achievement Level Descriptors
Afternoon: Study the Ordered Item Booklet
Day 2
Morning: Bookmark training and Round 1
Make bookmark recommendations individually
Afternoon: Round 2
Discuss differences in Round 1 bookmark placements
at each table
Make Round 2 bookmarks individually
12

In-Person Panel Agenda

Day 3
Morning: Round 3
Discuss Round 2 bookmark placements for the entire
room (all panelists at all tables)
View supporting data based on Round 2 bookmarks
(e.g., impact data, online panel results)
Place Round 3 bookmarks individually
Review final recommendations
Evaluate the process

13

Achievement Level Timeline (2014)

April 30

October 6-18
October 13-19

States Approved Achievement Level Setting

Plan
Online Panel
In-Person Panel (Dallas)

October 20

Cross-Grade Review Committee (Dallas)

October 30

Technical Advisory Committee Review

November 14
Fall/Winter

States vote to approve Achievement Level

Recommendations
State Adoption of Achievement Levels

Cross-Grade Review Committee

Also referred to as Vertical Articulation
60 panelists selected from In-Person Panel
One Delawarean participated

Reviewed Online Panel and In-Person Panel results

Recommend changes to smooth the data:
Motion
Second
Discussion
Vote: 2/3 majority required
Aligned cut scores across grades
Eliminated scaled score reversals

Vertical Articulation

This is an
example to
illustrate Vertical
Articulation. It is
NOT actual data
from Achievement
Level Setting.

16

Achievement Level Timeline (2014)

April 30

October 6-18
October 13-19

States Approved Achievement Level Setting

Plan
Online Panel
In-Person Panel (Dallas)

October 20

Cross-Grade Review Committee (Dallas)

October 30

Technical Advisory Committee Review

November 14
Fall/Winter

States vote to approve Achievement Level

Recommendations
State Adoption of Achievement Levels

Setting Achievement Levels

(Continued)
States Chiefs Review
Reviewed final recommendations from cross-grade Review

Committee
Approval by State Chiefs

State Board Review to Seek Approval

English Language Arts/Literacy:

Threshold Scale Scores
2800

Threshold Scale Score

2750
2700
2650

Level 4

2600

Level 3

2550

Level 2

2500
2450

2400
2350
2300

7
Grade

10

State Agreement on November 14, 2014

11

ELA/Literacy: Estimated Percentage of

Students at each Achievement Level
100%
90%

18%

18%

15%

11%

8%

30%

30%

Est. % in Each Level

80%
70%

20%

23%

29%

9%
32%

11%
30%

Level 3

60%
50%

27%

22%

40%

23%

29%

28%

Level 2
31%

31%

Level 1

30%
20%

Level 4

35%

37%

33%

10%

30%

34%

28%

28%

0%

6
7
8
11
Grade
Estimates based on Spring 2014 Smarter Balanced field test conducted
in 21 states and USVI.

Proposed Scale Score Cut Points

ELA/Literacy Scale Score Thresholds (scale 2000 3000)
Grade

Level 1-to-2

Level 2-to-3

Level 3-to-4

2367

2432

2490

2416

2473

2533

2442

2502

2582

2457

2531

2618

2479

2552

2649

2487

2567

2668

11

2493

2583

2682

For Board Action in January 2015

22

Mathematics: Threshold Scores

2800

Threshold Scale Score

2750

Level 4

2700

Level 3

2650
2600

Level 2

2550
2500
2450

2400
2350
2300

7
Grade

10

11

State Agreement on November 14, 2014

Mathematics: Estimated Percentage of

Students at each Achievement Level
100%
90%

Est. % at Each Level

80%

70%

12%

13%

15%

14%

13%

13%

11%

27%

24%

18%

19%

20%

19%

22%

60%

50%
40%

29%

36%

32%

32%

31%

30%

27%

10%

32%

27%

Level 3
Level 2

30%
20%

Level 4

35%

35%

36%

38%

40%

6
Grade

11

Level 1

0%

Estimates based on Spring 2014 Smarter Balanced field test conducted

in 21 states and USVI.

Proposed Scale Score Cut Points

Math Scale Score Thresholds (scale 2000 3000)
Grade

Level 1-to-2

Level 2-to-3

Level 3-to-4

2381

2436

2501

2411

2485

2549

2455

2528

2579

2473

2552

2610

2484

2567

2635

2504

2586

2653

11

2543

2628

2718

For Board Action in January 2015

25

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common Core

State Standards. They are designed to let teachers and parents know whether
students are on track to be college- and career-ready by the time they
graduate.
Because the new standards set higher expectations for students--and the new

tests are designed to assess student performance against these higher

expectations--our definition of grade level performance is higher than it used to
be.
As a result, its means that fewer students will meet grade level standards,

especially for the first few years. Results should improve as students have
additional years of instruction aligned to the new standards and become better
equipped to meet the challenges they present.
This does not mean that our students are doing worse than they did last year.

Rather, the scores represent a new baseline that provides a more accurate
indicator for educators, students, and parents as they work to meet the rigorous
demands of college and career readiness.

Learn More About Achievement Level

Setting
Video explanation of process
https://www.youtube.com/watch?v=bW_yGf4BB1E

Interpretation and Use of Scores

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-andUse-of-Scores.pdf

Easy to follow flow chart

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-BalancedAchievement-Level-Setting-Overview-.pdf

Q&A Document
http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/SmarterBalanced-Achievement-Levels-QA.pdf

www.DelExcels.org
27