1

NATIONAL MATHEMATICS ADVISORY PANEL

STRENGTHENING MATH EDUCATION THROUGH RESEARCH
+ + + + +

MEETING
+ + + + +

Friday,
April 20, 2007
+ + + + +

Illinois Mathematics and Science Academy

1500 W. Sullivan Road
Aurora, Illinois 60506
+ + + + +

8:50 a.m.

PANEL MEMBERS:
DR. LARRY FAULKNER, CHAIR
DR. CAMILLA PERSSON BENBOW, VICE CHAIR
DR. DEBORAH LOEWENBERG BALL
DR. A. WADE BOYKIN
DR. DOUG CLEMENTS
DR. SUSAN EMBRETSON
DR. FRANCIS (SKIP) FENNELL
DR. BERT FRISTEDT
DR. DAVID C. GEARY
DR. RUSSELL M. GERSTEN
MS. NANCY ICHINAGA (NOT PRESENT)
DR. TOM LOVELESS
DR. LIPING MA
DR. VALERIE F. REYNA
DR. WILFRIED SCHMID
DR. ROBERT S. SIEGLER
DR. JAMES SIMONS (NOT PRESENT)
DR. SANDRA STOTSKY
MR. VERN WILLIAMS
DR. HUNG-HSI WU

EX OFFICIO MEMBERS:
DR. DAN BERCH
DR. JOAN FERRINI-MUNDY
MS. DIANE JONES
MR. RAY SIMON (NOT PRESENT)
DR. GROVER J. (RUSS) WHITEHURST

STAFF:
MS. TYRRELL FLAWN, EXECUTIVE DIRECTOR
MS. IDA EBLINGER KELLEY
MS. MARIAN BANFIELD
MS. JENNIFER GRABAN
MR. KENNETH THOMSON
MR. ROBERT GOMEZ
 2
I N D E X

Welcome and Opening Remarks

Larry Faulkner, Chair . . . . . . . . . . . . . . 3

Janice Krouse, Illinois

Mathematics and Science Academy . . . . . . . . . 4

Public Comments

Dr. Henry Borenson . . . . . . . . . . . . . . . .14

Dr. Patrick Thompson . . . . . . . . . . . . . . .18

Mr. Kevin Killion . . . . . . . . . . . . . . . . 23

Mr. Jack Rotman . . . . . . . . . . . . . . . . . 28

Mr. Ken Indeck . . . . . . . . . . . . . . . . . .32

Dr. Sara Delano Moore . . . . . . . . . . . . . . 38

Dr. Barbara Wilmot . . . . . . . . . . . . . . . .44

Ms. Janie Zimmer . . . . . . . . . . . . . . . . .49

Ms. Cindy Jones . . . . . . . . . . . . . . . . . 55

Task Groups Progress Reports to Panel

Conceptual Knowledge and Skills . . . . . . . . . 62

Learning Processes . . . . . . . . . . . . . . . .65

Instructional Practices . . . . . . . . . . . . . 67

Teachers. . . . . . . . . . . . . . . . . . . . . 85

Assessment. . . . . . . . . . . . . . . . . . . . 105

Adjournment . . . . . . . . . . . . . . . . . . . 129
 3
1 P-R-O-C-E-E-D-I-N-G-S

2 CHAIRMAN FAULKNER: I ask the person who's

3 running the audio if it is true that I have to hold

4 this button down while I'm speaking?

5 AUDIO TECHNICIAN: Yes.

6 CHAIRMAN FAULKNER: It is true, okay, and

7 that means, I believe, that all of the speakers and

8 everyone here at the panel will, when you do speak,

9 have to hold the button down. Okay.

10 Okay, I'm Larry Faulkner. I'm chair of

11 the National Math Panel. Vice-Chair, Camilla Benbow

12 is next to me here. And we want to welcome everyone

13 in the audience to this open session of the National

14 Math Panel.

15 I'd like to begin by thanking the Illinois

16 Math and Science Academy for hosting this open

17 session. This is the sixth meeting of the National

18 Math Panel. We are holding these meetings across the

19 country in various locations, geographically

20 distributed, but we've tried pretty consistently to

21 associate the meetings of the Math Panel with

22 institutions in locales that symbolize high

23 achievement in the academic enterprise. The Illinois

24 Math and Science Academy certainly does symbolize

25 that.

26 Let me indicate that we have signing

 4
1 services available. They are active right now. We

2 can continue signing services through the entire

3 meeting if there is a use for them. If no one is

4 making use of them, we will discontinue. So I'd like

5 to indicate whether this is a desire to continue

6 signing services. Seeing no such indication, we will

7 discontinue the services. If there is a need to

8 institute them as the meeting goes along we can do

9 that. Thank you.

10 Now, let me introduce Dr. Janice Krouse.

11 Dr. Krouse currently serves as curriculum and

12 assessment leader at IMSA. She is instructor of

13 mathematical investigations II, III, IV, pre-calculus,

14 and advanced-placement calculus. She has a Bachelor's

15 Degree in secondary education in mathematics from

16 Clarion University of Pennsylvania, a Master's in

17 mathematical sciences from Clemson, and a Doctorate in

18 mathematics education from the University of

19 Pittsburgh. She is a member of the National and

20 Illinois Council of Teachers of Mathematics, National

21 Council of Supervisors of Mathematics. Dr. Janice

22 Krouse will be representing IMSA.

23 DR. KROUSE: Thank you, Dr. Faulkner.

24 Good morning. On behalf of the faculty and staff of

25 the Illinois Mathematics and Science Academy, I

26 welcome all of you here today.

 5
1 I am honored to have this opportunity to

2 greet such a distinguished group as you meet again to

3 engage in this important work. I join you in

4 recognizing the significant consequences of a quality

5 mathematics education for the children of this

6 country, as mathematics and critical thinking skills

7 so profoundly affect their lives and their ability as

8 responsible citizens to shape the human future. In my

9 brief comments today I hope to share with you our

10 vision of mathematics education and its power in

11 shaping minds.

12 My colleagues and I take our role in

13 influencing tomorrow's leaders very seriously. The

14 quality of the engagement between teacher and student

15 and between the student and the mathematics cannot be

16 underestimated. It was for these reasons that the

17 charter mathematics faculty and Presidential Awardees

18 of the Illinois Mathematics and Science Academy

19 invested their time, talents and energy into

20 authorizing a pre-calculus curriculum named

21 Mathematical Investigations for their students. With

22 ongoing revisions and updates Mathematical

23 Investigations, known affectionately as 'MI", is still

24 taught here today. And I am proud to say that it is

25 one of my primary duties to ensure the consistency and

26 coherence of this curriculum and its delivery.

 6
1 Charter math faculty and author of MI,

2 Chuck Hamberg, often said, “If you stop when you get

3 the answer to a problem, you miss half of the

4 mathematics." It has been noted that one of the

5 strengths of our program is the space we give students

6 to solve a problem “85 different ways". It is that

7 very notion of curiosity that drives learners to their

8 full potential. It is our job as educators to believe

9 in that potential and to create conditions in which it

10 can be realized. What, then, is the role of the

11 teacher in the MI classroom?

12 Largely, I am a guide. I imagine the

13 impression of an observer to my MI classroom. It

14 looks like a teacher's dream job. There is very

15 little at-the-board lecture on some days and, instead,

16 the observer sees me milling about the room,

17 intermittently asking or answering questions of

18 students who are working in small groups. Even

19 first-time IMSA students sometimes wonder, when is she

20 going to teach?

21 But next, I step back into my shoes as the

22 teacher and the facade of a simple job shatters.

23 Teaching is now far more exhausting than preparing

24 lessons and lecturing. In the traditional format the

25 teacher is almost always in control of what happens

26 next. Everything is predictable, planned and

 7
1 polished. There is often a sense of, I taught it, so

2 they now know it. Unfortunately, there is little way

3 to actually validate that sense until a formal

4 assessment is given, and by then, it's way too late

5 for some students.

6 Mathematical Investigations (MI) invites

7 learners into the science of mathematics through

8 carefully crafted questions and problems. Students

9 observe patterns and phenomena, make conjectures, test

10 their hypotheses on new problems, and analyze their

11 results. All the while students are engaged in

12 conversations with peers and teachers about

13 mathematics.

14 Ideas, probing questions, insights, and

15 supporting arguments emerge daily. Through these

16 conversations, students forge connections within and

17 among mathematical concepts in ways that make sense to

18 them. They utilize various forms of technology to

19 explore and test their conjectures. Most importantly,

20 they are not forced to merely absorb a neatly packaged

21 explanation given by the teacher.

22 In fact, the word “teaching” takes on a

23 whole new meaning in the MI classroom. It goes well

24 beyond standing at the board and dispensing content,

25 methodologies and algorithms organized in a manner

26 that makes perfect sense to the well-educated and

 8
1 well-meaning teacher. It now means letting go,

2 listening, assessing, reacting, questioning, probing,

3 listening, reacting, clarifying, watching, listening,

4 guiding, but not just telling, and again assessing

5 every student in the room, every day.

6 There is a delicate balance of timing that

7 must be maintained of when to let the students grapple

8 with a new or difficult idea, and when to intervene,

9 help them make necessary connections and to see the

10 big picture. There is a constant need to think on

11 your feet as students ask questions that even the

12 seasoned teacher does not anticipate. There is a need

13 for enough self-confidence and mathematical prowess to

14 let the students watch you grapple with a challenging

15 problem so that they can see you as a model problem

16 solver, even if that means you make a mistake in front

17 of them. This is something that the traditional

18 teacher wouldn't dream of. There is a need to be able

19 to answer students' questions with questions that lead

20 them to the answers they thought they couldn't get.

21 There is a need to hear the misconception that truly

22 underlies the superficial I don't know how to do this

23 one type of question.

24 Then, somewhere in the midst of the

25 grappling and questioning, the synthesis begins.

26 Students respond to the teacher's probing and

 9
1 challenging questions by refining their understandings

2 of complex ideas. Ultimately, the forging of

3 connections consummates in closure of sound

4 mathematical ideas that students can transfer and

5 apply to tomorrow's questions.

6 And once you think you've mastered all of

7 that, you get a new class of students and you have the

8 grand opportunity to start all over again. You find

9 the balance again, perhaps in a different place,

10 because all students are different, and teaching MI

11 actually lets you see that and react to it. The MI

12 teacher has the luxury of hearing students talk about

13 the mathematics in their language, using their

14 constructs. You learn to read how each student in

15 your class thinks about mathematics, and you have the

16 privilege of adjusting your instruction to suit all of

17 those needs. That is simply impossible in a

18 traditional classroom. Results on formal assessments

19 are rarely surprises. Such tests are merely

20 opportunities for students to demonstrate their

21 knowledge in a more formal manner.

22 With Mathematical Investigations, we give

23 our learners an opportunity to engage in the learning

24 process in ways unlike any they have previously

25 experienced. The mathematics environment here is

26 clearly one of collaboration, making connections, and

 10
1 solving problems. These skills are absolutely

2 fundamental to tomorrow's leaders.

3 These explorations are often aided by

4 various forms of technology, including the TI-89

5 Titanium/CAS graphing calculator, Mathematica,

6 Geometer's Sketchpad, Fathom, and the Internet.

7 Technology enables students to actively pursue

8 questions about mathematical constructs that otherwise

9 would be unattainable. Further, today's students are

10 engaged with technology so frequently, that to deny

11 them this resource in their learning is asking them to

12 divorce their natural environment from their

13 schooling. Fluency with emerging technologies in

14 problem solving will continue to be a critical,

15 necessary and expected skill for our students.

16 Results? With over 850 students taking

17 the Advanced Placement BC Calculus exam over the last

18 seven years, we enjoy a collective average of over 4.6

19 on a five-point scale. Intel Finalists, Siemens

20 winners, and inventors of Papal, Mosaic and YouTube

21 are among our alum.

22 Certainly, we can lay claim to working

23 with some remarkable students, but what we do in the

24 mathematics classroom is applicable to a much wider

25 audience. In 2003, IMSA mathematics faculty were

26 called on as pedagogical experts to help a neighboring

 11
1 district to determine criteria by which a mathematics

2 program would be selected for their high school that

3 would invite a student-centered, problem-based,

4 integrative and collaborative environment such as

5 ours. After a complex and thorough process led this

6 district to a selection, the process was repeated for

7 finding an appropriate program for their honors

8 students. After careful evaluation and critique,

9 IMSA's Mathematical Investigations was chosen, and is

10 now in its second year of implementation. The huge

11 paradigm shift was not without its bumps in the road.

12 But the benefits were evident to the teachers, even by

13 the end of the first year.

14 One of their teachers stated: The most

15 surprising thing that I encountered was how difficult

16 it was to get students to try all the way through a

17 problem. So many students waited for answers at the

18 beginning of the year, and now they'd rather find out

19 themselves than hear it from me. The challenges the

20 students face make them think about their thinking.

21 The students are willing to attempt any problem handed

22 to them and understand many of the processes instead

23 of just memorizing formulas.

24 Several years ago, an IMSA graduate

25 recounted her experience as a first year physics

26 student in a prestigious university's honors program.

 12
1 The young, yet highly accomplished professor, who

2 believed in the power of intimidation, began his

3 lesson by asking the class if anyone remembered a very

4 esoteric formula. As the IMSA graduate recalls, no

5 one did, and much to the delight of her professor, the

6 class nearly froze. Then, in the uncomfortable

7 silence, she raised her hand and gave the formula.

8 The professor, in amazement, asked her how she

9 remembered it, and she said, “I didn't. In my school,

10 we learned how to derive it.”

11 Ah, there is such power in giving students

12 the space to solve a problem in a multitude of ways;

13 in asking, “what if,” after the first answer is found

14 so as not to miss half of the mathematics; of engaging

15 students in deep, meaningful learning so that when the

16 formulas fade, the understanding endures.

17 I want to thank Dr. Marshall, Dr.

18 Faulkner, and the members of the Panel for this

19 opportunity to speak with you today. I look forward

20 to the recommendations that your research and wealth

21 of experience bring to your final report.

22 CHAIRMAN FAULKNER: Thank you, Dr. Krouse.

23 We appreciate the opportunity to be here and to hear

24 from you today. Let's proceed now into the open

25 session, which is going to be dedicated to public

26 comment. I would like to begin by introducing the new

 13
1 members of the National Math Panel. There are three

2 persons who have joined the Panel with this meeting

3 here in the Chicago area.

4 Let me start by introducing Doug Clements,

5 Professor of Early Childhood Mathematics and Computer

6 Education at the University of Buffalo, State

7 University of New York. Welcome Dr. Clements. Dr.

8 Susan Embretson, Professor of Psychology at Georgia

9 Institute of Technology, who is, where? Okay.

10 Welcome, Dr. Embretson. And Dr. Bert Fristedt,

11 Professor of Mathematics at the University of

12 Minnesota. Welcome, Dr. Fristedt.

13 We have had comments from the public on an

14 open basis consistently around the country. The

15 comments that we have received have been done on a

16 first come, first-served basis with the time

17 available. We have found these comments to be quite

18 useful, as we have received them over the period of

19 the Math Panel's meetings during the last year

20 approximately.

21 The nine speakers who will be speaking

22 today were registered for public comments. The list is

23 available to the Panel members in the notebooks under

24 tab six. There is one person on the waiting list.

25 Each speaker is limited to five minutes. There's a

26 timer right there at the table where testimony will be

 14
1 made. Panelists will have the opportunity to ask

2 questions of the speaker after the remarks are

3 concluded.

4 Let me go ahead and open this testimony.

5 The first presenter will be Henry Borenson.

6 DR. BORENSON: Mr. Chairman and members of

7 the Panel, I thank you for this opportunity. My name

8 is Dr. Henry Borenson, President of Borenson &

9 Associates, Incorporated. Some twenty years ago, as a

10 middle school math teacher, I was concerned with the

11 difficulty students were having learning algebra

12 abstractly. I determined to find a way to simplify

13 the concepts and make them concrete and visual and to

14 make them accessible to all grade school students.

15 After two years of experimentation working

16 with children, including children with learning

17 disabilities, I developed a system known as Hands On

18 Equations. This is a system, which uses game pieces,

19 such as those you see here, a flat laminated balance,

20 and a specific sequence of ideas to enable students as

21 early as the third grade to physically represent and

22 solve algebraic linear equations. The types of

23 equations, until then, were typically taught in the

24 eighth or the ninth grade.

25 Since 1995, Borenson & Associates has

26 conducted more than 1,500 Making Algebra Child's Play

 15
1 workshops throughout the United States. In these

2 workshops, teachers of grades three to eight learn how

3 to introduce the concept of a variable, the concept of

4 an equation, the subtraction and addition property of

5 equalities, and other key algebraic principles.

6 A key part of these workshops is a student

7 demonstration with local fourth and fifth grade

8 students. More than 1,500 times since 1995 the

9 teachers attending our seminars have seen how, in

10 three lessons, fourth and fifth grade students,

11 including so-called low ability students, can learn to

12 solve an algebraic linear equation such as 4x + 3 = 3x

13 + 9.

14 In a study to determine teachers’

15 confidence level to teach algebraic linear equations

16 to their lowest achieving students, Barbara N.

17 Borenson (2006) discovered that only 17 percent of 751

18 teachers, from grades three to eight attending a

19 Making Algebra Child's Play workshop, felt they would

20 be successful using the traditional abstract teaching

21 methods, while 98 percent expressed confidence of

22 success if they were to use the Hands On Equations and

23 materials. The study is shown in Appendix A of the

24 handout.

25 In an ongoing series of studies involving

26 multiple student characteristics and multi-site

 16
1 replications, supervised by Dr. Larry Barber, formerly

2 director of research at Phi Delta Kappa, we have found

3 significant pre to post-test gains for second grade

4 gifted students, sixth grade regular students and

5 ninth and tenth grade low achieving students.

6 Recently we completed a study involving

7 four fifth grade inner city classes comprising a total

8 of 111 students. The pre-test to post-test results

9 showed a large and highly significant increase in

10 scores. The combined mean increased in percentage

11 terms from 44.8 percent on the pre-test to 85.3

12 percent on the post- test. On the three-week

13 retention test, with no instruction in the interim,

14 the mean was 78.6 percent. When compared with a

15 pre-test score of 44.8 percent this increase was found

16 to be statistically significant with a T-value of

17 13.71. We are talking about fifth grade inner city

18 students succeeding with important algebraic concepts.

19 This study will be found in the Appendix B.

20 We believe we have provided evidence that

21 Hands On Equations learning system of instruction

22 significantly and positively impacts teachers’

23 self-confidence in their ability to introduce

24 algebraic equations to their students. We have

25 provided evidence that the program makes a measurable

26 difference in student learning. We believe it is

 17
1 possible and it is important, as the previous speaker

2 alluded to, for all students to gain the perception

3 that mathematics is a subject that they can understand

4 and a subject at which they can excel.

5 In Hands On Equations the students do not

6 memorize a set of procedures in order to obtain an

7 answer. They can use their creativity to apply

8 general algebraic principles in the manner that best

9 suits them. We ask the Panel to consider recommending

10 Hands On Equations as a supplementary program that is

11 effective in introducing grade school students to

12 basic algebra. Thank you very much.

13 CHAIRMAN FAULKNER: Thank you, Dr.

14 Borenson. Are there questions from the Panel? If

15 not, thank you. I will now turn to the second --

16 DR. FENNELL: Mr. Chairman?

17 CHAIRMAN FAULKNER: Yes.

18 DR. FENNELL: Just one question. Dr.

19 Borenson, the paper that you referenced --

20 CHAIRMAN FAULKNER: Push your button.

21 DR. FENNELL: I am. Thank you, Mr.

22 Chairman. The paper that you referenced, will we have

23 copies of that?

24 DR. BORENSON: Yes, in the handout. They

25 will be available in the handout.

26 CHAIRMAN FAULKNER: Thank you. Let's go

 18
1 now to the second presenter. It's Andy Isaacs. Is

2 Andy Isaacs here? Apparently not. We will go to the

3 third presenter, Cindy Jones. Is Cindy Jones here?

4 All right, I will go to the fourth presenter, Patrick

5 Thompson.

6 DR. THOMPSON: Does everyone have a copy

7 of my testimony? Okay, there are some things in

8 there. Mr. Chairman, Madam Vice-Chairman, Panel

9 members, thank you very much for this opportunity to

10 speak with you about the Panel's work. I will speak

11 to five of the Panel's charges.

12 My first remark addresses charges one and

13 seven; critical skills and skill progressions and

14 research in support of math education. But it

15 actually cuts across all of the charges that I listed,

16 that I'm going to address.

17 The Panel has a significant task of

18 responding to a list of charges that take skills as

19 the primary component in mathematics learning when the

20 notion of skill itself is hardly well-defined. Do you

21 take skill to mean a child's ability to perform

22 reliably a procedure when told to perform the

23 procedure? Or do you take skill to mean a child's

24 ability to have developed sufficient knowledge and

25 appropriate flexibility of thought to solve most

26 problems of a particular genre of problems, even those

 19
1 that might have subtle and nuance differences from any

2 the students might have seen.

3 I am noting that I'm going to have to skip

4 through some of what's in the prepared remarks because

5 when I actually timed myself, looking in the eyes of

6 people, I couldn't read as fast as when I was alone in

7 my office.

8 Thus, it is incumbent upon the Panel to

9 make clear where it stands with regard to what

10 students should learn, and to justify that stance

11 according to the pragmatic consequences that relative

12 stances have regarding students' learning and

13 preparation for future learning.

14 In regard to charges three and four,

15 processes of learning and affective instructional

16 practices, I offer an example from a current research

17 project on affective models for secondary mathematics

18 instruction.

19 We created an implementation of Algebra I

20 in collaboration with one of the participating

21 teachers in order to develop artifacts that would make

22 concrete to the teachers what it was that we had in

23 mind, that they had difficulty envisioning.

24 We also hope that these students would

25 display proficiency in the algebra the teachers were

26 accustomed to assessing, but display it as a

 20
1 consequence of understanding ideas well and not

2 because of having memorized the prescribed procedure.

3 The students we taught were not in an

4 honors program, thus they were taking Algebra I at

5 ninth grade. Their computation skills were atrocious.

6 They had no understanding of fractions. Their

7 experience in mathematics was that teachers showed

8 them procedures they were supposed to remember until

9 the next test. Their feelings about mathematics were

10 that it was a dehumanizing experience that no one in

11 their right mind would choose to experience having had

12 the option not to.

13 So our immediate question was what to do

14 about their lack of skills given that our goal was to

15 have them eventually engage with significantly

16 mathematical ideas. Do we re-teach what they've

17 already not learned? Well, we decided that we

18 wouldn't, that we would move on. We began the year

19 with no review and we designed the instruction by the

20 seat of our pants, always guided by our goal of having

21 them engage meaningfully with significant mathematical

22 ideas and at the same time be able to pass their

23 mandated tests.

24 We focused on central ideas prior to

25 calculus curriculum like variation, covariation, rate

26 of change and functional relationship. The

 21
1 appendices, by the appendices, I'm referring to files

2 that are on the CD that I turned in. Those aren't

3 printed in the materials that I gave you. The

4 appendices contain examples of the kind of work we

5 need to expect from the students.

6 Here, I'll give one example, to make a

7 point. Actually, I'm running out of time so I'll let

8 you read the example. But if you notice, it has to do

9 with having students construct the sum of two

10 functions, which are not defined by a formula, in

11 their experience, but nevertheless, focuses on the

12 idea that, in fact the sum of two functions is a

13 function. Then we shared the definition and they

14 became excited that they had dealt with such

15 complicated functions and wanted a printout to take

16 home to show their friends and parents.

17 Another point of what I say is that, in my

18 opinion, this nation suffers not from a lack of

19 research, but from a lack of imagination. It suffers

20 from lack of imagination at all levels especially at

21 the levels of policy and politics.

22 With regard to teacher training, at ASU

23 our biggest problem is recruitment and retention. I

24 give statistics about that in my testimony. One of

25 them has to do with the fact that less than 30 percent

26 of secondary math students who are required to take

 22
1 three semesters of calculus actually complete three

2 semesters of calculus. In other words, we lose them.

3 And we actually lose many of them from ASU, not just

4 to other majors.

5 My time is up. I'm over-time. So I'll

6 let you read the rest of my testimony.

7 CHAIRMAN FAULKNER: Thank you Dr.

8 Thompson. Questions from the panel?

9 DR. BENBOW: Can you tell us what you

10 actually did in the classroom to engage the students?

11 DR. THOMPSON: Can you be more specific?

12 DR. BENBOW: Well, you had them work the

13 problems, but given that they didn't have the basic

14 skills, how did you engage them in significant math

15 without having had the basic skills already mastered?

16 DR. THOMPSON: Well, we focused on

17 beginning with phenomenon, having them use literal

18 symbols to represent phenomena. We focused on ideas

19 of variable and variation so that variable stood for

20 things that changed. The discussions were not about

21 how to compute but how to represent. Computations

22 flowed from that. Once you have a representation it's

23 about how you would compute something. But the

24 algebra that they wrote was algebra of representation,

25 not necessarily algebra of computation, except when we

26 looked at the mathematics of equivalence. Then we

 23
1 focused on the algebra of computation.

2 CHAIRMAN FAULKNER: Any other questions or

3 comments? Thank you, Dr. Thompson. We now go to

4 presenter number five, Kevin Killion.

5 MR. KILLION: Hi. I'm Kevin Killion. I

6 hold a degree in mathematics. I have been a research

7 VP in a marketing agency. I've written several

8 commercial/statistical analysis products. And I

9 operate a successful business in market and media

10 analysis.

11 I became involved with math reform when I

12 observed the difficulties my own son was having.

13 Today I serve as director of the Illinois Loop, a 12-

14 year-old organization of parents, teachers, school

15 board members and others. Our Illinoisloop.org

16 website is a valuable source about what is going on in

17 schools and we have logged over 600,000 visitors.

18 First, I have a comment on standards.

19 Calling one category of math programs standards-based

20 is a ploy that tarnishes other programs as somehow

21 being rudderless and adrift. I left over there my

22 beloved American College dictionary. I looked it up.

23 The word, standard, has 19 definitions. Similarly,

24 there is no single standard for math.

25 Another weapon is to blame lousy math

26 performance on attractable, dusty old methods.

 24
1 Schools are constantly told to embrace change and

2 teachers are exhorted to be agents of change. But the

3 reality couldn't be more starkly different.

4 Everything has already changed.

5 On our Illinois Loop website we provide

6 extensive information about how math is taught in

7 Illinois school districts, from Addison to Zion. This

8 resource is well-used by parents in tracking what

9 districts are doing. And here's what we found:

10 In Chicago, some 290 schools use

11 constructivist math programs in early grades. On the

12 flip side we've been able to identify only five, count

13 them, five conventional Chicago public schools that

14 use practice and mastery math programs. Plus there

15 are another five schools that are charter schools

16 offering Saxon Math.

17 With regards to the suburbs, the Illinois

18 Loop has collected information on the math programs

19 used in 118 suburban K-8 districts in five collar

20 counties. We find that constructivist products form

21 the math foundation in 77 percent of those districts.

22 But even that only hints at what's going on.

23 On the north shore, or in Lake County or

24 in some other areas, it's almost impossible to find

25 any schools with anything but constructivist math.

26 And across the area, the Chicago/Suburban area, we've

 25
1 identified only six districts out of 118 that make any

2 use whatsoever of those math programs most recommended

3 by practice and mastery reformers, such as Singapore

4 math or Saxon Math. So much for the argument that

5 parents in the suburbs already have the schools they

6 want.

7 Now here's a twist. We've all heard of

8 the dance of the lemons. Well, there's also the dance

9 of the math lemons performed by districts unhappy with

10 their math programs. As an example, District 39 up in

11 Wilmette dumps Math Trailblazers and picks up Everyday

12 Math even as District 109 in Deerfield drops Everyday

13 Math to pick up and have a chance on Math

14 Trailblazers.

15 Like Lois Lane who couldn't see the truth

16 staring her in the face, these districts stick with

17 constructivist math and merely substitute one program

18 for another. We're sure not seeing any agents for

19 change there. These districts are firmly mired down

20 with a philosophy that they refuse to abandon.

21 In the course of our work at the Illinois

22 Loop we receive hundreds of messages from parents.

23 Many of them are concerned about constructivist math

24 programs in their schools and what these programs are

25 doing to their kids. I'll close by reading just a few

26 snips of what parents are saying. I implore you to

 26
1 listen to the passion and the concern expressed.

2 A Glencoe mom tells us that math problems

3 here are bad and getting worse. A Skokie math teacher

4 told us that this series has been a dismal failure in

5 teaching math. A Homewood parent tells us that the

6 math program there is the most confusing, ridiculous

7 method she’s ever seen. She couldn’t believe parents

8 are accepting this and how sad it is for their

9 children. A Glenview couple writes that the math

10 program there stinks. A Downers Grove parent wrote to

11 us, “It is beyond belief that so many parents can be

12 so upset at the situation and yet be paralyzed.”

13 A Hinsdale parent told us that more than

14 40 percent of parents pay tutors up to $50 a hour to

15 teach their kids properly. A Naperville mom fears

16 that when her daughter finishes in this school system,

17 she will be well experienced in arts and crafts, but

18 she will lack the ability to make change. A parent

19 laments that as the result of the math problem in her

20 Lake Forest school, “You can't get your kid into Kumon

21 classes around here. When will they learn?” By the

22 way, in Naperville there are nine Kumon centers in the

23 area. A Crystal Lake parent wrote to us, “Everyone I

24 have talked to thinks this program is horrible and

25 their kids are struggling.” A Batavia couple says,

26 “This trend needs to be stopped now before we have a

 27
1 complete train wreck.” A Plainfield parent says, “I

2 think it's the most absurd form of education that I've

3 ever seen.” And a Yorkville mom sums it all up by

4 saying, “Help, how can I save our children from this

5 blight?”

6 CHAIRMAN FAULKNER: Your time has just

7 expired.

8 MR. KILLION: I just did. A Yorkville mom

9 says, “Help, how can I save our children from this

10 blight?” Members of the Math Panel, thank you for

11 your concern.

12 CHAIRMAN FAULKNER: Thank you, Mr.

13 Killion. Questions or comments? Yes, we have one

14 here.

15 DR. SIEGLER: I grew up in this area so I

16 know most of the suburbs that you mentioned are quite

17 affluent suburbs and the parents aren't usually shy

18 about organizing if they have a strong opinion.

19 If these are representative of parental

20 views, what do you think is keeping school board

21 members from being elected who want to change the

22 current system?

23 MR. KILLION: I don't think there's

24 sufficient time to go into the problems of school

25 board elections here. Suffice to say that these are

26 real opinions representative of hundreds that we get

 28
1 at the Illinois Loop. They are suffering with what's

2 going on with their kids. If somebody else believes

3 in a different way of doing things and they want to

4 choose a program for their kids, that's fine, but

5 these are parents who are suffering.

6 CHAIRMAN FAULKNER: Are there any other

7 comments or questions? Let me go ahead and proceed to

8 the next presenter, Jack Rotman.

9 MR. ROTMAN: Let's see if I can master the

10 technology, is that okay? Is the microphone working?

11 No? Is that better, okay, thank you.

12 To briefly introduce myself, I am Jack

13 Rotman. I have been a professor at Lansing Community

14 College in Michigan for 34 years. I have been active

15 in American Mathematical Association of Two-Year

16 Colleges (AMATYC). I currently chair the

17 developmental mathematics committee of that group.

18 And I was a contributing writer for the 2006 standards

19 document, Beyond Crossroads.

20 I have three questions for the panel,

21 which are the basis for my remarks. One: Are

22 sufficient and necessary conditions present in the

23 schools to provide mathematics learning for all

24 students? Two: Are there barriers outside of the

25 education system that substantially limit the learning

26 of mathematics for some groups of students? Three: Do

 29
1 we plan for the system, which provides a second chance

2 for students who did not learn sufficient mathematics

3 in the schools?

4 On the first question, are there

5 sufficient and necessary conditions present? At the

6 most basic level, students must stay present and

7 attending in order to benefit from the curriculum. At

8 the secondary level we are all aware of the

9 substantial problem with drop-outs. However, there

10 are also a lot of absences in the schools. Studies

11 show that seven percent of the students were absent on

12 a given day and that was only for unexcused absences.

13 For students who are present we need to be concerned

14 about how much they are actually attending. An

15 optimistic study estimated that students were in

16 attendance and with material 65 to 75 percent of the

17 time.

18 In a different study of various methods of

19 teaching, the only method that increased student

20 attention was the debate/discussion method. The group

21 learning methods only increased attention a little

22 bit.

23 On the second question, are there barriers

24 outside of schools that limit opportunities? The

25 Panel has discussed the concept of stereotype threat,

26 which is one of those barriers. I would encourage the

 30
1 Panel to consider broader viewpoints of these issues.

2 One of these viewpoints is called critical race

3 theory.

4 Critical race theory begins with the

5 assumption that racism is embedded within the social

6 structure and analyzes information from that

7 viewpoint. Critical race theory suggests that the

8 achievement gap that we talked about is really an

9 opportunity gap. A more radical view sees

10 standardized testing as a means to justify

11 differences.

12 Also, some researchers have documented a

13 default trajectory towards dropping out in certain

14 types of communities. In addition, in some regions we

15 again have schools that are separate but not equal due

16 to policies such as schools of choice and other

17 issues. This segregation results in a situation where

18 the Lansing high schools are 70 percent minority,

19 while the Lansing area itself is only 35 percent

20 minorities.

21 I would also encourage the Panel to

22 consider other barriers that exist outside the

23 education system. For example, mathematics still

24 faces the barrier that it is acceptable or even

25 desirable to be "bad at math." Will we hear the

26 President say that qualitative, quantitative literacy

 31
1 is a personal value for me? How about our role models

2 in entertainments and sports? Are they going to say

3 my mathematical skills allowed me to accomplish what I

4 needed? Or will you see Ben Wallace helping middle

5 school students with their mathematics? Or do we see

6 these pupils say, math was always hard for me too?

7 On the third question, the back up system,

8 the second chance. Most countries don't have our

9 community college system. Even the community colleges

10 offer a second chance for many adults to learn the

11 mathematics they need. However, the country hardly

12 has a systematic plan for this approach. Outside of

13 the work of American Mathematical Association of Two-

14 Year Colleges (AMATYC) and a little bit of The

15 Mathematical Association’s work, nothing systematic is

16 done beyond the state level.

17 I will suggest the Panel consider

18 community colleges as part of the system and that we

19 be included in the dialogue. We provide a recruiting

20 ground for mathematics and science fields. Also, we

21 offer a response time measured in one to three years

22 instead of 12 years for K-12 schools. I would think

23 our involvement would be appropriate.

24 As we consider our work to strengthen

25 mathematics education I hope we can establish those

26 minimal conditions for learning, look at barriers to

 32
1 learning outside of the schools and include community

2 colleges in our discussions. Thank you for your

3 attention and the opportunity to address the Panel.

4 CHAIRMAN FAULKNER: Thank you, Professor

5 Rotman. Questions or comments from the Panel? None.

6 The seventh presenter is Ken Indeck.

7 MR. INDECK: My name is Ken Indeck. I am

8 a high school math teacher with nearly three decades

9 of experience and I'm speaking as a representative for

10 the Illinois Association for Gifted Children. My

11 remarks are primarily anecdotal because it is

12 important for me to communicate the realities as

13 viewed from within the school system. I am confident

14 that similarities exist in most educational settings.

15 One of the hallmarks of gifted education

16 is the notion that one size does not fit all. In

17 Illinois the same content benchmarks are used to

18 assess all students. For the bottom third of the

19 academic spectrum these benchmarks are a stretch,

20 often unrealistically so. For the top third, these

21 students have often surpassed them.

22 Last year I was talking through some

23 curricular improvements we could implement for bright

24 students in our building. Before I finished, the

25 administrator I was speaking with, stopped me and

26 said, you're not going to want to hear this, but

 33
1 that's not going to help us meet AYP, those kids will

2 be fine. We need to focus on raising the scores of

3 the students who will help us. Unless you think

4 otherwise, that administrator is an excellent

5 educator.

6 As a parent I was thrilled when our son's

7 third grade math teacher told us how proud she was the

8 entire class had completed both the third and fourth

9 grade material. Imagine my shock when we found his

10 fourth grade math teacher was teaching the fourth

11 grade curriculum, knowing full well the students had

12 been through and mastered that content, simply because

13 she was not able to teach the fifth grade material.

14 Half that class lost interest in math. By sixth grade

15 there were a handful of students who were still

16 excited about math and ready for algebra, but they

17 were not allowed to take the course because the junior

18 high didn't offer it. My son is now in tenth grade

19 and I say with mixed emotions, he is doing fine.

20 I envision four entwined approaches to

21 improving our current state of affairs in math

22 education. First, advocate for the use of best

23 practices. Acceleration is important but it is not

24 enough and absent coordinated sequence spanning years

25 it can even be detrimental.

26 Few high school math teachers are

 34
1 knowledgeable about differentiated instruction and

2 fewer still are skilled in its implementation. For

3 many high schools the gifted curriculum is synonymous

4 with AP course offerings. Well, this might be a

5 starting point. We know better. Yes, we want our

6 brightest and most able students exposed to

7 age-advanced concepts. However, those students thrive

8 when they are also in a rich environment that helps

9 them see connections to other topics in the

10 curriculum, and where they are allowed to explore how

11 those connections can be put to use making the world

12 better by improving people's lives.

13 Second, encourage and support the

14 educators who take reasonable professional risks. The

15 current practice of looking for significant

16 improvement over short stretches of time does not

17 realistically encourage a teacher to switch from one

18 set of techniques to another, even if the new set is

19 extremely promising, when it will likely take on the

20 order of five years to master those skills, and

21 another five to ten years to become expert in their

22 use.

23 Third, it is essential to provide

24 significant support for research. In education we

25 need research regarding instructional practices. We

26 need to know more about how grouping students and

 35
1 sequencing topics influence learning. It is important

2 to develop broader assessment practices, practices

3 that extend beyond recalling facts and solving one or

4 two step problems. In order to maintain our nation's

5 leadership in the areas of science and technology it

6 is essential to support math, the research in

7 mathematics, science and their applied fields both

8 through academia and industry.

9 Finally, it is crucial that we do a better

10 job educating the public about the educational

11 enterprise as a whole. The typical adult non-educator

12 does not fully understand how hard the work is and

13 unlikely has an realistic set of expectations both for

14 what our schools can provide and how the educational

15 growth of students can be documented.

16 We have before us opportunities for

17 establishing long-term leadership for the economic

18 strength and for improving the quality of life for our

19 nation and the world. That leadership is likely to

20 come from students at the top end of the academic

21 spectrum, who are well grounded in math and science

22 and who recognize the connections between those

23 subjects and the broader world around them. The

24 notion that we are doing fine is not good enough.

25 Strengthening the educational system

26 should prompt increased achievement for all. Closing

 36
1 the achievement gap should not translate to holding

2 hostage the education of our most able students. If

3 we compare students' performance to their own

4 capabilities, as the mission statements for most

5 schools suggest, it is the bright students who fall

6 short and are furthest from reaching their potential.

7 We must do better. Please help us. Thank you.

8 CHAIRMAN FAULKNER: Thank you, Mr. Indeck.

9 We appreciate your comments. Comments or questions

10 from the Panel? Wilfried?

11 DR. SCHMID: You introduced yourself as a

12 teacher of gifted children. At what kind of school do

13 you teach?

14 MR. INDECK: I teach at a regular high

15 school. I am not teaching gifted classes at this

16 point. I was at one time in my career the curriculum

17 and staff development coordinator for High School

18 District 214 for their Talent Development Program.

19 DR. SCHMID: So at that point you were

20 designated as a teacher of gifted children?

21 MR. INDECK: In our district we don't call

22 gifted children. It is a program for developing

23 talent.

24 DR. SCHMID: I see.

25 MR. INDECK: That's the closest we have in

26 our district.
 37
1 DR. SCHMID: In any case, but you were

2 designated as such?

3 MR. INDECK: Yes.

4 DR. SCHMID: And no longer are? Did the

5 policy change?

6 MR. INDECK: Yes, I don't currently hold

7 that position.

8 DR. SCHMID: Does anybody else?

9 MR. INDECK: No, the position was

10 disbanded because it doesn't help them meet AYP.

11 CHAIRMAN FAULKNER: Valerie?

12 DR. REYNA: Thank you. What do you think

13 the barriers are to really having two goals in mind at

14 the same time, the adequacy goal and excellence as a

15 goal as well? You make the argument yourself in your

16 own testimony that these two are not exclusive. If

17 we're focusing on one of the goals, why does that mean

18 the exclusion of the other? What do you think the

19 barrier is there?

20 MR. INDECK: There are multiple barriers,

21 but it seems to me that when we're in a system that

22 tries to get all students to a certain level and

23 doesn't look for growth on the part of all students,

24 that once students are to that particular level,

25 there's very little incentive within the structure

26 itself to move those students forward. The focus is

 38
1 more on moving those students who haven't reached that

2 benchmark, to the benchmark.

3 CHAIRMAN FAULKNER: Any other questions or

4 comments? Thank you, Mr. Indeck. We now turn to

5 presenter number eight, Sarah Delano Moore.

6 DR. DELANO MOORE: Good morning. My name

7 is Sara Delano Moore and I'm the Director of

8 Mathematics and Science at ETA/Cuisenaire.

9 ETA/Cuisenaire is a leading publisher of supplemental

10 instructional resources for mathematics, science and

11 literacy. For over 40 years our company has pioneered

12 the development and effective use of hands-on

13 materials or manipulatives to improve student learning

14 outcomes.

15 I am here this morning to share my

16 thoughts on the role of manipulative-based instruction

17 in mathematics, and I will begin by briefly sharing my

18 own background.

19 I am a fourth generation teacher, although

20 the first to teach mathematics. My undergraduate

21 education focused on molecular biology, so I am a

22 scientist by training. I taught mathematics and

23 science in middle grade schools and have worked in

24 higher education as well teaching both mathematics

25 methods courses and curriculum. My research in

26 writing has focused on the use of award winning and

 39
1 high quality literature, alongside hands-on

2 experiences, to teach rich mathematics and science at

3 all levels.

4 ETA's products and associated professional

5 development training have always been grounded in the

6 belief that children learn mathematics by doing it in

7 active, hands-on ways. We are fortunate that this

8 belief has a long research base to support it.

9 The three part learning cycle we use to

10 discuss instruction with manipulatives includes phases

11 called concrete, representational and abstract.

12 Jerome Burner's work talked about a similar cycle as

13 inactive, iconic and symbolic. Most recently Michael

14 Batista used the terms, action, reflection, and

15 abstraction. In all cases the basic idea is that

16 children must first have hands-on experiences with the

17 math and then use the representational phase as a

18 transition to the abstract more formal mathematics.

19 There is no question that children need to

20 be computationally fluent. These children must also

21 understand the mathematics behind the computational

22 procedures they use. I love mathematics. I earned

23 good grades in math class at school. I'm not sure,

24 however, that I genuinely understood mathematics until

25 I learned to use manipulatives to teach math.

26 I had my first “ah-hah" experience in

 40
1 mathematics in my early 20's when I learned to

2 multiply two digit numbers with base ten blocks. I

3 finally knew what was really going on when I wrote

4 down all those numbers years ago in fourth grade. I

5 saw the connection between multiplication of whole

6 numbers and binomial multiplication in algebra. Math

7 became a connected whole for me.

8 We don't know what problems our students

9 will need to solve as adults. We can be certain they

10 will need problem solving skills. They will also need

11 the confidence they can solve problems successfully.

12 Children learn by making connections between the

13 familiar and the unfamiliar. Our role as teachers is

14 to guide children toward the connections we want them

15 to make.

16 Manipulatives provide a bridge between the

17 concrete world of a child and the abstract concepts of

18 mathematics. They may also serve as an enticement to

19 learn math, which does not, on the surface, appear

20 engaging. By using the manipulatives, literature,

21 and other active instructional resources, children can

22 be drawn into the world of math and find success

23 there. Every child must find meaningful success in

24 mathematics, and we must use every resource we have to

25 ensure this happens.

26 Effective use of manipulatives is one

 41
1 resource to help children find success in mathematics.

2 Children must conduct structured investigations and

3 work towards an understanding of procedures and

4 strategies that can be generalized.

5 Unfortunately, manipulatives are too often

6 used as hands-on worksheets with teachers telling

7 students exactly which piece to touch and where to

8 place it as they act out the traditional algorithm.

9 Professional development is critical if teachers are

10 to use manipulatives as the powerful tool research

11 shows them to be.

12 For all of us as teachers it is a great

13 day when a student has an "ah-hah" moment of learning.

14 One of the joys of my job working in professional

15 development is to see that same "ah-hah" from adults

16 as they see mathematics taught with manipulatives and

17 understand, sometimes for the first time, what was

18 really going on back in elementary school when they

19 obediently memorized the sequence of squiggles on the

20 page which represented a mathematical idea.

21 Manipulatives are one of the most powerful

22 tools in a teacher's arsenal for helping students

23 learn mathematics well. I urge the Panel to ensure

24 these "ah-hah" moments continue in classrooms by

25 supporting the use of manipulatives in mathematics

26 instruction in their report. Thank you.

 42
1 CHAIRMAN FAULKNER: Thank you, Dr. Moore.

2 Are there questions? There obviously are. Tom?

3 DR. LOVELESS: You mentioned the research

4 that supports manipulatives? Could you describe a

5 couple of, or even just one piece of research that you

6 are convinced is persuasive in that regard?

7 DR. DELANO MOORE: I made an intentional

8 choice today not to do the foot-noted presentation and

9 to talk instead. There are a number of pieces of

10 research, for example, on the use of base ten blocks

11 and various models. Also there is the work that Karen

12 Fuson and her colleagues in John Bransford's group

13 about how students learn text. For example, they talk

14 about the role of working from a concrete model even

15 if it’s a sketch.

16 And as I say to teachers who say they

17 can't use manipulatives on most state tests (Wisconsin

18 I think, is an exception, but there aren't many), “No

19 state has banned scratch paper.” So when they learn

20 to make those sketches, they can then, as our opening

21 speaker said, derive the formulas. I'd be happy to

22 provide additional, more formal work to you if you'd

23 like.

24 DR. LOVELESS: Just one follow up, would

25 you agree that the goal would be for students not to

26 have to depend on manipulatives eventually?

 43
1 DR. DELANO MOORE: In my experience, as

2 children learn the math, the use of manipulatives

3 really self-extinguishes. They reach a point where

4 they can use most often what are common algorithms,

5 perhaps an algorithm of their own adaptation, but can

6 do that work independently. The manipulatives serve

7 as a tool to bridge between their concrete world and

8 concrete thinking and the more formal mathematics that

9 they will need in life.

10 CHAIRMAN FAULKNER: Other questions or

11 comments? Valerie?

12 DR. REYNA: Are you familiar with the

13 research of David Uttal on the use of manipulatives,

14 U-t-t-a-l?

15 DR. DELANO MOORE: I don't believe I am.

16 DR. REYNA: And by the way, I hate to put

17 you on the spot on this.

18 DR. DELANO MOORE: That's quite all right.

19 Saying “I don't know” is an okay thing to do.

20 DR. REYNA: It certainly is. If you do

21 take a look at the research and want to communicate

22 with the Panel about the research, there are ways to

23 do that. I would be interested in your reaction to

24 that. It may only be a question of at what age

25 manipulatives are appropriate to use.

26 DR. DELANO MOORE: All right, I will take

 44
1 a look.

2 CHAIRMAN FAULKNER: Anything else? All

3 right, thank you, Dr. Moore. Let me indicate the

4 procedure that we'll follow. Number nine, Barbara

5 Wilmot, is next. Let me ask her to come forward. She

6 was the last signed up member. Since we had some who

7 did not arrive I want to indicate that we will next

8 take Janie Zimmer who was the person on the waiting

9 list. And we will then proceed to pick up number

10 three, who has arrived, and that's Cindy Jones. So we

11 will go to Barbara Wilmot next.

12 DR. WILMOT: Thank you, good morning. My

13 name is Dr. Barbara Wilmot. I've worked in

14 mathematics education from the elementary to the

15 university level for 45 years now. I taught at

16 Illinois State and directed a state professional

17 development program there. Now I'm an independent

18 consultant and administrator for a grant that supports

19 and monitors central Illinois schools that don't make

20 AYP year after year.

21 I've worked with over 100 districts and

22 stopped counting when I'd given 1,200 professional

23 development workshops in almost every state. I'm

24 speaking this morning for myself and for Learning

25 Resources, which is a leading provider of hands-on

26 classroom materials. I often use their materials in

 45
1 my professional development sessions and have

2 partnered with them to create this mathematics

3 manipulatives handbook, which they hand out free by

4 the thousands of copies in order to help teachers.

5 Today I speak really on behalf of millions

6 of students with language barriers or special needs,

7 many of whom are in mainstream classes. And yet for

8 the most part No Child Left Behind holds these

9 students to the same level of expectation as other

10 students. How can we level the playing field for them

11 in learning mathematics?

12 I'd like to share three points supporting

13 the fact that hands-on learning tools and related

14 professional development help English language

15 learners and students of special needs deepen their

16 understanding of mathematics and increase achievement.

17 The first point is similar to hers that

18 manipulatives allow students to build, model and

19 create multiple representations of mathematical

20 concepts and, therefore, help them meet benchmarks.

21 Whether we use NCTM or state standards as a guideline,

22 “build,” “model” and “create” are verbs that appear at

23 almost every grade level.

24 Other verbs such as “describe,” “verify”

25 and “generalize” also happen if engaging tasks are

26 offered for students. Certainly it is difficult to

 46
1 meet these outcomes without using manipulatives.

2 Meeting benchmarks and developing a deep

3 understanding require that students explore multiple

4 representations of mathematical concepts. Students

5 aren't likely to fail if they only learn fraction

6 concepts, which is a pre-requisite to learning

7 algebra, in one representational format. Just imagine

8 if learning fractions meant only drawings of pizza

9 slices, and unfortunately that's the reality in many

10 classrooms.

11 But in schools like West School in

12 Glencoe, Illinois, teachers, like math specialist,

13 Laura Menonski are using multiple formats. Laura

14 recalls modeling the concept of two thirds to her

15 students, and she could tell by the glazed look on

16 their faces that her explanations, instead of drawings

17 of set models, weren't enough. Then she brought out

18 manipulatives like the fraction spheres and tower

19 tubes to show two thirds in multiple dimensions. And

20 when students modeled and saw the different formats

21 they literally said, oh, and explained to her what she

22 was trying to tell them all along.

23 My second point is that manipulatives

24 allow students with limited language abilities and/or

25 special needs to understand simple and complex

26 mathematical concepts and to actually demonstrate

 47
1 their knowledge. Manipulatives enable English

2 language learners and students with special needs to

3 see concepts being modeled even when the students are

4 unable to understand the teacher's words.

5 Physical models also allow for assessment.

6 Students can build the representation and demonstrate

7 knowledge of ideas when they aren't ready to

8 communicate via symbols or words.

9 Chris Triola, a sixth grade teacher from

10 General McLean School District in Edinboro,

11 Pennsylvania says manipulatives allow his students

12 with special needs to develop “insights and

13 connections not available through paper, pencil or

14 lecture.”

15 My third and final point is that high

16 quality professional development is absolutely

17 essential to learn how to integrate manipulatives and

18 a variety of strategies and techniques into the

19 curriculum to differentiate the instruction for each

20 student. Teachers believe, in general, manipulatives

21 are highly effective, yet few actually use them and

22 fewer yet know how to use them correctly.

23 Manipulatives are most effective when the

24 students use them to probe and make conjectures and

25 generalize about a mathematical problem.

26 CHAIRMAN FAULKNER: Please wrap up, your

 48
1 time's expired.

2 DR. WILMOT: Oh, I'm sorry. I have found

3 that at least 100 hours of professional development

4 are necessary to make teachers comfortable with this.

5 So in order to meet the various needs, learning styles

6 and abilities, I hope that you'll really think about

7 the manipulatives as well as the professional

8 development piece in your recommendations. Thank you.

9 CHAIRMAN FAULKNER: Thank you, Dr. Wilmot.

10 Questions or comments from the Panel? Diane?

11 MS. JONES: I have a question. You

12 obviously have a lot of experience in teacher

13 professional development, and you mentioned the term

14 high quality. You know, the U.S. government spends

15 millions of dollars every year on teacher professional

16 development and yet it's very hard to distinguish high

17 quality from low quality. Could you give us some

18 guidance? In what way is professional development

19 best delivered and how should we be assessing

20 federally supported teacher professional development

21 to distinguish high quality from moderate or low

22 quality opportunities?

23 DR. WILMOT: Wow, that's a good question.

24 First of all, I really think that we’ve given up on

25 the one-shot professional development. However, it’s

26 useful for awareness and for disseminating information

 49
1 for an introduction to something.

2 But I really think that it has to be

3 district based and/or school based. It has to be long

4 term. There has to be support and an administrator.

5 The best situation that I've ever had is when the

6 administrator is there for every class or session that

7 I have and then goes in and says, okay, would Tuesday

8 or Wednesday be better for me to see how you're using

9 this. So I think that the use of it is really good.

10 And I think the keeping of data, both on student

11 achievement and teacher opinion. Having teachers

12 journal and reflect is a vital part of it too. So

13 there are just a lot of phases. But just the coming

14 in and going out doesn't help, you know. Less than

15 ten percent of change actually happens in the

16 classroom with that.

17 CHAIRMAN FAULKNER: Other questions or

18 comments? Thank you. We'll go to Janie Zimmer.

19 MS. ZIMMER: I'm glad I'm not too

20 technology challenged. This is interesting. Thank

21 you for the opportunity to speak to you today. I am

22 Janie Zimmer from Research Based Education speaking on

23 behalf of National Council of Supervisors of

24 Mathematics (NCSM). I serve on their board.

25 This morning I would like to discuss an

26 issue that is critical in mathematics education. The

 50
1 critical issue is equity, the opportunity for and the

2 expectation that every child will be successful in

3 mathematics and will have the opportunity to reach

4 high levels of mathematical content.

5 Schools and teachers do have that

6 expectation for a lot of our children. And we think

7 that we have this expectation for all children when we

8 profess to permit children into higher levels of math

9 classes beginning with Algebra I, if they are prepared

10 and ready for that rigorous work. We profess we want

11 every child to be successful, that is to get good

12 grades.

13 In the meantime, we continue to sort and

14 select which students will go into which high level

15 classes and which students will go into the low level

16 or remedial Algebra I A/B classes. In many schools

17 educators create classes into which they place

18 students according to their performance on state

19 assessments. Or they create inclusion classes that

20 contain both general education and special education

21 students, frequently without support. But does that

22 act in itself create equity?

23 In the words of a Pennsylvania teacher, I

24 expect very different things from the lower level or

25 inclusion class than I do from other classes.

26 Individual Education Plans (IEP’s) send the message

 51
1 that a student does not have to perform in the same

2 way as my other students. Isn't that holding a

3 different expectation? What I am communicating is

4 that some of my students are not smart enough to do

5 the same high-level work. Yet how are students who

6 enter the ninth grade with fourth grade mathematics

7 skills able to do the ninth grade high algebra

8 content? How are they able to do the same high-level

9 work of on grade level students who are entering that

10 same algebra class?

11 A school district of about 50,000 students

12 in Maryland has grappled with this issue. Today, all

13 students in the middle school are placed in on-grade

14 level classes with added support for struggling

15 students. In all 12 of their high schools, all

16 incoming students take Algebra I as the minimum class.

17 Students with IEP's or 504 plans are included in these

18 regular classes.

19 In addition, high schools provide an extra

20 support seminar as part of the schedule of students

21 who need extra help. These classes are assigned two

22 teachers; a math certified teacher and a special

23 education teacher. The classes have a student/teacher

24 ratio of 10:1, and they are co-taught by both

25 teachers. An observer walking in would most likely

26 not be able to tell which is the special education

 52
1 teacher and which is the general education math

2 teacher.

3 They have had much success with this

4 program. All 12 high schools have achieved AYP in

5 mathematics for all populations. Overall in the

6 district, the special education students of the extra

7 seminar class had a pass rate on the state algebra

8 data analysis assessment that was 17 percent higher

9 than the general population for those algebra classes.

10 That is, the group of the special education students

11 actually outperformed the general population.

12 In addition, special education students

13 who were in the extra seminar class had a pass rate

14 that greatly exceeded the pass rate of peer special

15 education students who had not been placed in the

16 extra seminar class. They exceeded by 36 percent in

17 one school and by 33, 27, 25, and 21 percent in

18 similar schools.

19 As we look throughout the country we see

20 other successful programs regarding equity in place.

21 Most special education students are not intellectually

22 challenged but they are challenged in many other ways.

23 Equity is on the plate of most mathematics educators

24 yet they need to grow and expand their understanding

25 of the deep implications of this principle.

26 We realize that equity in itself is not

 53
1 the mission of this Panel. But we ask you to take to

2 heart our information and address equity in every

3 facet of your work. Address the equity not only for

4 students with special needs, but also for students who

5 are speakers of other languages, who are economically

6 challenged, who have families unable to provide

7 support, who seem unmotivated or who, in some other

8 way, do not fit the norm. NCMS used to consider this

9 and we invite you to call upon us to inform your work

10 and provide support in any way that we can.

11 CHAIRMAN FAULKNER: Thank you, Ms. Zimmer.

12 Questions? Right here, Vern?

13 MR. WILLIAMS: You said that all of the

14 students took algebra in ninth grade, but did some of

15 the students take algebra in eight grade and then

16 geometry in ninth grade?

17 MS. ZIMMER: That's correct. The school

18 system in question is Howard County Public Schools and

19 they do have a gifted program in place where a lot of

20 the students, or a number of the students in seventh

21 and eighth grade actually take algebra and geometry.

22 They may come into ninth grade taking geometry or they

23 may come into ninth grade taking Algebra II.

24 MR. WILLIAMS: Okay, so they have

25 basically sorted the population starting in seventh

26 and eighth grade?

 54
1 MS. ZIMMER: Yes, they have.

2 MR. WILLIAMS: And my other question is,

3 the test that they used as a comparison, was it the

4 Maryland State Algebra Test?

5 MS. ZIMMER: The state test in Maryland is

6 an Algebra/Data Analysis test and that is the test

7 that they used.

8 CHAIRMAN FAULKNER: Diane?

9 MS. JONES: I'm quite familiar with Howard

10 County and the growth of the number of Huntington and

11 Sylvan Learning Centers, tutoring centers that have

12 grown in Howard County in the past five to ten years.

13 Was there any collection of data in this study in

14 terms of the number of students involved in this study

15 who were also receiving supplemental tutoring by the

16 many Huntington and Sylvan centers that now exist in

17 Howard County?

18 MS. ZIMMER: I'm not aware that there was

19 that correlation made.

20 CHAIRMAN FAULKNER: Valerie?

21 DR. REYNA: Are the data that you just

22 presented here going to be made available to the

23 Panel?

24 MS. ZIMMER: I do not have this data in my

25 possession right at this time, but I can get them and

26 send the reports to the Panel.

 55
1 CHAIRMAN FAULKNER: Anyone else?

2 MS. ZIMMER: If I could just add one other

3 thing. The co-taught classes were classes where there

4 was a lot of professional development for the

5 teachers. So the special education teachers were

6 brought up to speed on the content in mathematics,

7 which we find to be a problem across the nation.

8 CHAIRMAN FAULKNER: Thank you. Okay, we

9 are now going back to pick up number three, Cindy

10 Jones.

11 MS. JONES: I come to you from Providence,

12 Rhode Island where I am a curriculum coordinator for

13 mathematics. I work in a largely urban community with

14 a large immigrant and Latino population. My purpose

15 for coming here is just to describe some aspects of

16 the professional development that we've engaged in as

17 teachers that I feel is very effective.

18 Since the beginning of my teaching career

19 I've always had a love for data. This interest started

20 in 1998 when, in my first year of teaching, my

21 principal informed me that a RIDE, Rhode Island

22 Department of Ed official was coming to observe my

23 class. The Rhode Island official that came to observe

24 me did not revoke my teaching certificate. Instead,

25 she invited me to join her workshop.

26 The next three years, working with the

 56
1 Rhode Island Department of Ed Office of Assessment

2 Accountability Teacher Committee, I learned so much.

3 I became sold on the idea of using rubrics to assess

4 student's work. I was also sold on the idea that our

5 assessments and what we teach should be closely

6 aligned to state standards. I became proficient at

7 looking at standardized test results to help form my

8 instruction.

9 The SIP model, the Standards in Practice,

10 which is part of my appendices, has become an

11 essential piece of professional development for

12 teachers, administrators and curriculum coordinators.

13 The SIP model encourages colleagues to come together

14 and discuss student work in terms of how the work

15 demonstrates proficiency, the math concepts or grade

16 level expectation and the Rhode Island standards being

17 targeted.

18 Colleagues are prohibited from discussing

19 the student, but rather discussing the work itself.

20 In the SIP model, at first everyone assesses a bunch

21 of student work on his or her own. Then in small

22 groups, colleagues have discussions regarding the

23 grades they have assigned to each piece of work. When

24 discrepancies arise, colleagues are asked to reexamine

25 the student work and the rubric to come to an

26 agreement. The process allows educators to share

 57
1 ideas and their perspective with one another.

2 A typical rubric, I'm sure you're

3 familiar, is usually a one through four. One is below

4 proficiency, two is partially proficient, three is

5 proficient, and four is proficient with distinction.

6 As you can see the use of rubrics has permeated every

7 aspect of our school community. It has been a

8 powerful tool for us as teachers to keep the main

9 thing, the main thing. And more and more we are

10 learning not to judge student work based on personal

11 biases or family history, but more on what the student

12 was actually able to produce.

13 Since then I've become a math coach.

14 Being a math coach allows me to integrate standards

15 and assessment into my practice. One of the things I

16 do often in team meetings is look at the New England

17 common assessment programs release items, which are

18 released by the Rhode Island Department of Education

19 annually. Twenty-five percent of that exam is

20 released annually.

21 And one of the things that we do with

22 these release items is we align them to specific grade

23 level distinctions and Norman Webb's depth of

24 knowledge levels. Then we compare what we have to the

25 release test answer page.

26 Norman Webb's depth of knowledge of

 58
1 mathematics consists of four levels of proficiency.

2 The fourth, which is level four, is the most rigorous

3 type of assessment item. It requires more high order

4 thinking skills than the other three.

5 The New England Common Assessment Program,

6 otherwise known as NECAP, does not assess at level

7 four. The first depth of knowledge level, assessment

8 items may consist of simple recall or recognition of

9 facts or math terms and application of a well-known

10 algorithm. The other levels require more and more

11 thinking skills, such as comparing/contrasting. Depth

12 of knowledge two is more of the comparing/contrasting.

13 Justifying and making conjectures is depth of

14 knowledge level three. You'll find reference to these

15 different levels in my appendices.

16 Integrating depth of knowledge into

17 assessment items makes room for rigorous instruction.

18 As a result teachers have to go beyond just hitting

19 the surface of math concepts. We have to build the

20 kind of understanding that allows kids to make

21 conjectures and draw conclusions. As a result we know

22 we have to spend more time on math concepts and we

23 have to introduce them in many different contexts.

24 I would love to see more of my colleagues

25 and myself receive professional development in the

26 ways I've described above. I believe it has helped

 59
1 raise our school and district student achievement

2 scores in mathematics and empowers us as teachers to

3 own what we teach.

4 CHAIRMAN FAULKNER: Thank you, Ms. Jones.

5 Questions or comments? There are none, thank you.

6 We will reassemble at I think 10:25 for

7 the session which will involve the reporting of task

8 groups. We are now concluding the task group.

9 (Whereupon, the above-entitled matter

10 briefly went off the record.)

11 CHAIRMAN FAULKNER: This letter has been

12 put in my hands and I want to convey it to you. This

13 is a letter from the Vice-President Eric McLarin at

14 IMSA.

15 A proclamation was issued by the Governor

16 asking for all citizens of Illinois to join in a

17 moment of mourning and ring bells in memory of those

18 who lost their lives earlier this week at Virginia

19 Tech. The moment will be observed today at 11:00 a.m.

20 Dr. Gebble, 1980 Virginia Tech graduate with a Ph.D.

21 in microbiology will lead the IMSA community via the

22 public address system. The Governor's proclamation is

23 printed below. I'll read the proclamation.

24 Whereas, the Commonwealth of Virginia and

25 the United States of America suffered a great tragedy

26 on April 16, 2007, when 32 people were murdered and

 60
1 dozens more were injured on the campus of Virginia

2 Tech in Blacksburg, Virginia. And whereas the State

3 of Illinois grieves with those who lost loved ones on

4 that day. And we pray that they and the entire

5 Virginia Tech community can someday find peace and

6 solace in the wake of this senseless act of violence.

7 And whereas in the words of Virginia Governor, Timothy

8 M. Kaine, “April 16, 2007 will be remembered in the

9 hearts and minds of Virginians and all Americans for

10 the rest of their lives". Indeed this is a tragedy

11 that our nation will never forget and we come together

12 as a people to mourn with the victims' families. And

13 whereas Governor Kaine will declare a day of mourning

14 in Virginia on April 20 highlighted by a bell-ringing

15 ceremony at noon Eastern time in honor of the victims

16 of the Virginia Tech tragedy and whereas Illinois is

17 humbled, yet saddened, to join in this solemn

18 observance and will hold a bell-ringing ceremony in

19 accordance with Governor Kaine's declaration.

20 Therefore, I, Rod R. Blagojevich, Governor of the

21 State of Illinois, do hereby proclaim April 20, 2007

22 as a day of mourning for the Virginia Tech victims in

23 Illinois, and I encourage all citizens to join in the

24 ringing of bells at 11:00 a.m. Central time in memory

25 of those who have lost their lives on that dreadful

26 day.
 61
1 That will occur at 11:00 o'clock and we

2 will stop what we're doing and simply be a part of it.

3 With that let me turn the program over to

4 Vice Chair Camilla Benbow who will preside in this

5 next section.

6 VICE CHAIRPERSON BENBOW: We now move to

7 the open session to hear progress reports from the

8 various task groups. For those of you who may not

9 have been following the National Math Panel's work too

10 closely, let me just give you a little bit of

11 background how we are conducting our work.

12 The Presidential charge asked us to

13 address several questions, and we decided that the

14 best way to organize our work and be most effective

15 would be to form first, four task groups to address

16 the questions in the presidential charge.

17 The first task group is the Conceptual

18 Knowledge and Skills Task Group. The second task

19 group is Learning Processes. The third task group is

20 Instructional Practices. The fourth one is Teachers.

21 Those began right away.

22 It was always the intent that we would

23 have an Assessment Task Group as well, but we wanted

24 to make some progress on the first four before we

25 formed the Assessment Task Group. The Assessment Task

26 Group was actually formed at this meeting and has had

 62
1 already some meetings. And today, now, we will report

2 out what is the progress of their work so far.

3 So I am going to ask each task group to

4 come forward, either the Chair or several individuals

5 in the task group are going to give a report of our

6 work so far. Approximately a third of our work has

7 been looked at. I'm not sure it's exactly a third.

8 And we hope to continue reporting out bits and pieces

9 at the next meeting and again in St. Louis.

10 So the first task group that I ask to come

11 forward is Conceptual Knowledge and Skills and the

12 chair of that task group is Skip Fennell.

13 DR. FENNELL: Good morning. I'd like to

14 acknowledge my task group and some others who have

15 contributed to our work along the way; particularly

16 task group members Dr. Sandra Stotsky, Dr. Larry

17 Faulkner, Dr. Wilfried Schmid, and Dr. Liping Ma.

18 Then we have other members of the Panel who have

19 assisted in assembling our report to date including

20 particularly Dr. Hung-Hsi Wu.

21 So we are essentially addressing three

22 questions, the first one being what are the major

23 topics of school-based algebra as we know it. Our

24 analysis includes a review of states with standards

25 for Algebra I and Algebra II courses, the relatively

26 recent grade 12 NAEP objectives, the two related

 63
1 initiatives from Achieve, the American Diploma Project

2 benchmarks, as well as their end of course test in

3 Algebra II, and Singapore Mathematics Curriculum for

4 grades seven through ten.

5 I'm hesitating here because I'm noticing

6 that several members of the Panel are getting cups of

7 coffee and one of you better grab one for me. Okay, I

8 lost my train, sorry about that.

9 We're also looking at additional

10 international comparisons and major textbook

11 comparisons, as well, to give us sort of a descriptive

12 analysis relative to what is algebra. That will be

13 fueled by the research that some of the other groups

14 are working on; particularly the Learning Processes

15 group as they move into algebra itself.

16 We have, and it's matter of public record,

17 created a listing of major topics of school algebra

18 that will be supported with not only the major topics,

19 but a discussion of those topics in prose, supported

20 by research, hopefully to be made available at our

21 next meeting in Miami. And then there will be an

22 appendix that will take that relatively brief

23 discussion of algebra and expand that to a full

24 elaboration of algebra.

25 The corollary to the question relative to

26 the definition of algebra is the question, “What are

 64
1 the essential, foundational concepts and skills that

2 lead to algebra?” Again, there's an analysis here.

3 Our analysis is looking at the mathematics taught in

4 grades K-8 in top performing Trends in International

5 Math and Science Study (TIMSS) countries.

6 We are also looking at the differences in

7 curriculum approaches in those top-performing

8 countries. We have looked at the NCTM Curriculum

9 Focal Points. We are also looking at the mathematics

10 skills and concepts in the six highest rated state

11 curriculum frameworks, and also a yet to be completed

12 survey of teachers of algebra in this country. The

13 survey is going to begin very soon.

14 So we will come out of that with a draft

15 of the foundations, the essentials that students ought

16 to have prior to experiences in algebra. This would

17 not be an entire full curriculum, but the elements,

18 the critical foundational pieces that lead to algebra.

19 There will be a discussion of those as well and an

20 elaboration. You can perhaps see the analogy to the

21 algebra piece here as well.

22 A third question, does the sequence of

23 mathematics topics at grade levels prior to algebra

24 affect algebra achievement? For this final question

25 we have a work in progress in this area. We're

26 intending to look at the following: programmatic

 65
1 research on recently developed curricula, benefits of

2 an integrated approach and the role of integrated

3 mathematics in this whole configuration of school

4 mathematics, particularly algebra at the secondary

5 level, and the research on the placement of algebra.

6 By that I mean the actual grade placement of algebra,

7 the percentages of eighth grade kids taking formal

8 algebra, or for that matter, lower than grade eight.

9 So that's an analysis that we've begun as well. And

10 that's where we are.

11 VICE CHAIRPERSON BENBOW: Are there any

12 questions? Well, hearing none, Skip, your coffee is

13 up here. All right, if I could now have a report from

14 the Learning Processes Task Group. Dave Geary is the

15 Chair of this task group and he's going to be

16 delivering the report.

17 DR. GEARY: Do I turn on the timer? No,

18 all right. This will be short anyway. Contributing

19 members to this group are myself, Dan Berch, Wade

20 Boykin, Susan Embretson, Valerie Reyna, Bob Siegler,

21 and Jennifer Graban is the Department of Education

22 staff member assisting us.

23 As you know, last time we presented a

24 detailed review of what we had done at that point

25 covering basic principles of learning in cognition,

26 mathematical knowledge children bring to school and

 66
1 math learning in whole number arithmetic. So I won't

2 bore you again with those details other than to remind

3 you that is what has been completed.

4 The other groups reviewed that work this

5 time and we found the comments to be very helpful and

6 suggestions for our revisions to be very helpful.

7 Between now and our next meeting in six weeks, we're

8 going to take these comments and suggestions into

9 consideration and revise these three sections

10 accordingly and hopefully bring it up to something

11 very close to a final draft. As part of those

12 revisions we will begin to extract out policy

13 recommendations more explicitly in there, as part of

14 the text and probably a separate summary section.

15 Between now and June we will also be

16 working on a drafted section of the social

17 motivational affective processes. We hope to have a

18 nearly complete section of that to be included with

19 the other three sections, and the revisions for your

20 review at that time. We hope to have those sections

21 completed after the June meeting, nearly finalized.

22 Between our June meeting and the meeting

23 in St. Louis in September we will complete the

24 sections on fractions, estimation, geometry and

25 algebra. The latter two areas may have less work than

26 the other areas, but nonetheless, we will review that

 67
1 and point out areas where there are substantial holes.

2 For the St. Louis meeting, as well, we

3 hope to review differences and similarities across

4 race, ethnicity, socio-economic status, and gender in

5 the key areas that are included in this report. We

6 will also have a section on recent work in the brain

7 sciences in math learning and mathematics cognition.

8 Of course, we will also take comments and

9 recommendations at that meeting and we will hope to

10 have all of those changes done by the October meeting,

11 to have our section of the report complete by then.

12 And of course, during all of these revisions we will

13 be working on integrating our aspects of the report

14 with the aspects of the other four sub-groups. That's

15 it.

16 VICE CHAIRPERSON BENBOW: Are there any

17 questions? Seeing none, hearing none, thank you. All

18 right, at this point in time we will move up with a

19 presentation from the Instructional Practices Task

20 Group. Russ Gersten chairs the Instructional

21 Practices Task Group, and Tom Loveless and Joan

22 Ferrini-Mundy will be joining him to present our work

23 so far.

24 DR. GERSTEN: Myself, Camilla Benbow, Doug

25 Clements, Bert Fristedt, Tom Loveless, Vern Williams,

26 Joan Ferrini-Mundy and Diane Jones are members of the

 68
1 group and Marian Banfield is our Department of

2 Education support person, team member.

3 Quickly, I just want to review again, we

4 shared this at the last meeting, but we have basically

5 firmed this up a little bit. The core of our report

6 on each of the six topics we've agreed to look at,

7 with the possibility of a seventh if time permits,

8 will be experimentally high quality, quasi-

9 experimental studies using criteria very similar to

10 the What Works Clearing House. I'm not going to go

11 into the technical details now. We've had an

12 excellent team from Abt Associates and have worked

13 collaboratively and productively with them.

14 Other studies that we will look at and use

15 to inform our interpretation of the findings, our

16 framing of the issues, and our thoughts about future

17 research include any other type of quantitative

18 studies, descriptive or correllational studies,

19 qualitative and K studies. We also have a group of

20 tier-four studies that are flawed experiments, studies

21 that have some level of serious problems with them.

22 We will only mention them with extreme caveats,

23 because these are the ones that the data is really not

24 interpretable due to the serious types of problems.

25 And again, the details of this are flushed out in our

26 preliminary writings.
 69
1 Tom, Joan and I are going to share just

2 where we are in the first three topics and we'll start

3 with Tom.

4 DR. LOVELESS: Thank you, Russell. As

5 Russell pointed out, Abt Associates performed a

6 meta-analysis for us. First they conducted a search

7 of the literature applying the criteria that Russell

8 described. The first topic that we wanted to look at

9 was the whole issue of student-centered learning

10 versus teacher-centered learning, considering that as

11 a continuum.

12 Within that literature the search produced

13 over 100 studies. I can't remember the exact number.

14 I think it was 126. And what we did was then apply

15 our criteria, which screened down the literature. Of

16 the remaining studies we then grouped them by their

17 common approach or intervention that was tested. The

18 one area that leapt out as having a sufficient number

19 of studies to really draw some conclusions about was

20 cooperative learning and peer assisted learning. And

21 those are the results I'd like to show you.

22 First of all, in cooperative learning one

23 technique that was studied was team-assisted

24 individualization. This is an intervention that

25 involves grouping students into groups of four or five

26 and then giving the students work on particular areas

 70
1 in which they have shown deficiencies. And then the

2 students work as a team for a period of time, as

3 opposed to say doing individual seatwork. And then

4 the students are tested, both pre and post tested.

5 In these particular studies, these are all

6 tier-one studies that we're looking at. The students

7 were randomly assigned to both treatment and control

8 groups. As you can see in terms of math concepts the

9 effect was trivial.

10 In math computation, however, there were

11 six studies that produced seven pooled effect sizes.

12 The pooled effect size is .340, which is statistically

13 significant. You can see the p-value, .002. So this

14 particular finding is actually the most robust finding

15 that we came up with.

16 I want to caution right up front that this

17 does not mean that simply putting students into groups

18 and then giving them math to do, necessarily produces

19 results. These are highly structured interventions.

20 They are not simply testing grouping, but they're

21 testing a particular form of grouping with a specified

22 award structure.

23 The second area in which we found

24 sufficient research to perform a meta-analysis was

25 student teams achievement division. This is another

26 Johns Hopkins invented intervention. And we found no

 71
1 significant effect.

2 In terms of peer assisted learning, again,

3 we found an effect on computation. This is one

4 particular study, just one study. It had classroom

5 level data, where classrooms were randomly assigned to

6 treatment and control. Lynn Fuchs was head of the

7 research team. We found a significant effect there in

8 this particular study of 0.441. Most researchers

9 would consider that to be a modest effect. And the

10 p-value of .021 shows that it is statistically

11 significant.

12 We called this next group of studies

13 “other cooperative learning strategies” because they

14 didn't fall under any particular definition of the

15 cooperative learning strategy, but they did test

16 cooperative learning. The Mevarech study, for

17 example, is out of Israel, and the effects size of

18 .230 is also statistically significant. In this

19 particular study the students were assigned in pairs

20 to a computer-assisted learning intervention.

21 So in one intervention, students worked

22 individually at the computer and received their math

23 instruction. In the experimental condition the

24 students worked in pairs at the computer and received

25 their instruction that way.

26 Finally, we call these mixed approached

 72
1 and interpret them with some caution because not only

2 was either peer assisted learning or cooperative

3 learning part of the intervention, but there were

4 other characteristics of the intervention. Other

5 things were modified. Curriculum was changed or

6 something else was going on as well as peer assisted

7 learning.

8 So we can't isolate cooperative learning

9 or peer assisted learning and say that was the thing

10 that produces this positive effect, but they should be

11 noted. Busato was a study out of the Netherlands.

12 And that's a large effect, the largest of the studies

13 that we looked at here, .634, and that is

14 statistically significant. This is another Fuchs

15 study of peer assisted learning and I talked about

16 that earlier.

17 DR. GERSTEN: I'm going to talk a little

18 about the work on formative assessment. We actually

19 found a set of high quality studies. The first

20 question is, does it help students? Is math

21 achievement raised if teachers weekly, every other

22 week, have some assessment of where kids are, what

23 they've learned or not learned and some valid

24 measure.

25 And the second one is for teachers to get

26 the raw data and to try to make sense of it and

 73
1 develop instructional plans. And the other things,

2 which we call enhancements, are giving specific tools

3 or strategies or procedures to teachers to help them

4 figure out how to use the data, what they might do

5 with it. So those are our two research questions.

6 We found ten high quality studies, which

7 is a lot for most topics in education. This would not

8 necessarily, in medicine or public health, be a lot,

9 but for education ten is a lot of this quality. All

10 are in the elementary grades. The measures are both

11 concepts measures and computation measures, very

12 similar to what Tom showed you.

13 The technical characteristics of the

14 measures seemed fine. But the content validity, we're

15 having three experts on our Panel review that in a

16 bit. And that is not completed yet.

17 This is the type of formative assessment

18 that was done in these particular studies. It isn't

19 the only way to do it, but it is the way it was done

20 in this set of ten studies. Basically a sample for

21 the year's state standards, the kinds of things kids

22 are supposed to know by the end of the year by May or

23 June, were used to generate items. And each of these

24 tests given usually every other week, typically on the

25 computer, kind of take random samples of the items.

26 So this is very, very different than the

 74
1 way formative assessment is done in most classrooms in

2 the U.S. or around the world. The idea is that this

3 way you can really track growth. They wind up just in

4 terms of psychometric and technical qualities to be

5 far superior to the typical weekly unit tests.

6 Because you also get at not just what the kid learned

7 during the week, but what they retained, and their

8 ability to use what they already know to figure out

9 stuff that might come out in the later half of the

10 year. So it winds up working better. There may be

11 other approaches, but we just don't have the level of

12 evidence on other approaches.

13 There is a consistent statistically

14 significant effect for teachers (using basically

15 random assignment, high quality designs). Use of

16 formative assessment does raise student achievement by

17 approximately a quarter of a standard deviation or ten

18 percentile points, which is not too bad on the fact

19 that it's repeated or replicated again and again.

20 The second thing in terms of these

21 enhancements is the effect more or less is doubled,

22 and I'll show you in a second what the enhancements

23 are. When you look at the whole set of them, what you

24 need to do to do statistical tests, you get a sense

25 that the effect is about double, so it gets closer to

26 18, 19 percentile points.

 75
1 The only thing is, these studies of

2 enhancement were almost all, with one exception, done

3 with special education students. So that is something

4 to keep in mind.

5 In one study after the performance data

6 was analyzed, these enhancements are basically the

7 computer-generated practice, which became the basis of

8 tutoring sessions. So kids were getting help on

9 material they needed help in.

10 In another study, the teachers didn't get

11 their hands on materials but had a sense for each

12 child and for the whole class. These are areas that

13 the kids need help in. So again, it was a way to guide

14 time for differentiated or individualized instruction.

15 In one case there was a bank of experts,

16 math coaches, math specialists who developed ideas

17 when kids are having trouble with place value and

18 hundredths and thousandths. Again, this is a way to

19 intensely work with a small group of kids.

20 And the last one was kids learn to monitor

21 their own progress. They themselves can see how

22 they're doing and figure out what are the areas they

23 need help in.

24 So that is where this stands. But it is

25 actually a pretty solid basis for making

26 recommendations in our view and we've got an input,

 76
1 which we will incorporate, from the other groups.

2 DR. FERINNI-MUNDY: The third category

3 where we've made some progress is in the area of “real

4 world problem solving.” And the reason that we've put

5 that in quotes -- we have found is that this notion of

6 "real world" problems is not an unfamiliar idea --

7 curriculum and it's been available --

8 VICE CHAIRPERSON BENBOW: Could you speak

9 into the mic --

10 DR. FERINNI-MUNDY: Sorry, sorry. Many

11 current policy documents call for the use of real

12 world problems in mathematics instruction and this is

13 reflected in some instructional materials as well.

14 Now the reason that "real world" is in

15 quotation marks is summarized here, and there is a

16 discussion of this in our draft material thus far

17 coming from the literature.

18 One of the issues with this topic is that

19 real world is an under-specified construct. We have a

20 variety of meanings that appear in the research, that

21 appear in the discussion by developers about what they

22 intend with this. And we've listed here just a few of

23 the areas that we're seeing come up in the

24 descriptions of what people mean by this general area.

25 So you see, for example, literature that

26 discusses real world problems as problems that would

 77
1 be meaningful, appealing and motivating for students

2 from contexts that they know, from imaginary

3 situations, from mathematics. Sometimes the discussion

4 focuses more on what are called authentic problems.

5 That would be similar to those in applications beyond

6 the school setting. Often there is description of

7 such problems as being complex with multiple steps and

8 involving integration of concepts. The idea of

9 open-ended problems, problems both with multiple

10 solutions and possibly multiple solution paths are

11 included sometimes in these descriptions.

12 We also are finding in the literature that

13 there are many arguments from a variety of places

14 based on beliefs, experience and research, both for

15 and against the various types of real world problem

16 emphases that you've seen in the previous slide.

17 This makes it complicated to review the

18 research, and at this moment we're looking at only 12

19 studies that Abt has located for us through their

20 searching. Three of these are quasi-experimental

21 studies that have examined the impact of what I would

22 call full-blown curricula that feature, in some sense,

23 a real world emphasis. And these studies all have

24 methodological issues, but they are providing us with

25 some insights and some ways of framing this discussion

26 that will be very helpful.

 78
1 There are nine other studies that we have

2 found that look at the impact of various types of

3 instruction using "real world problems" and/or

4 instructional strategies that are meant to help

5 students solve real world problems. And again, these

6 studies have methodological issues but they're raising

7 important conceptual issues for our discussion. So we

8 are in process with this but we wanted to let you know

9 where we are with it at this stage.

10 VICE CHAIRPERSON BENBOW: Do we have any

11 questions? Russ?

12 DR. WHITEHURST: More in the form of

13 suggestions/questions. Tom, as I looked at the

14 presentation there seemed to be occasions where you

15 would pool effect sizes across a group of studies and

16 other cases where you simply highlighted a positive

17 effect size for one study and left uncommented upon

18 smaller effect sizes for other studies. So, at some

19 point that needs to be rationalized.

20 DR. LOVELESS: If I could just respond?

21 DR. WHITEHURST: Sure.

22 DR. LOVELESS: We pooled when it was clear

23 the intervention was similar across the studies. In

24 the ones that we did not pool, we did not pool them

25 because the opposite was true. It looked as if there

26 were key parts of the interventions that just

 79
1 differentiated them.

2 VICE CHAIRPERSON BENBOW: I think we need

3 to stop. Can you hold those questions? We can pick

4 that up. I think we hear the bells. I say since we

5 stopped and we'll pick up this dialog and discussion

6 back and forth, but since we've stopped and we're so

7 close to 11:00 o'clock, we have one more minute, let's

8 just wait, we'll pick it up.

9 DR. LOVELESS: Can we leave now?

10 VICE CHAIRPERSON BENBOW: I suspect there

11 might be more questions coming, Tom.

12 DR. LOVELESS: I'm practicing the button

13 pushing.

14 (Whereupon, a short break was taken

15 for a message from the principal

16 regarding the Illinois day of mourning

17 for Virginia Tech.)

18 VICE CHAIRPERSON BENBOW: All right, Russ,

19 if you want to pick up where we left off?

20 DR. WHITEHURST: My other question or

21 point or suggestion is that as a Panel, I think we

22 need to be cautious or perhaps come to some

23 understanding, shared understandings as we're talking

24 about small, medium and large effects. There is

25 nothing out there that anchors those terms, an effect

26 that might be considered small, it could be large if

 80
1 it accumulated. Something that's an effect over a

2 two-year period would have a very different meaning

3 than an effect over a two-week period. Thanks.

4 DR. GERSTEN: That is one issue that we're

5 grappling with and we're going to be working with Mark

6 Lipsey on as he has some time for our group. And it's

7 an excellent point and one that, guidance from any

8 members of the Panel, Institute for Education

9 Sciences, et cetera, would be really appreciated.

10 VICE CHAIRPERSON BENBOW: Tom, did you

11 have a response that you wanted to make? I certainly

12 didn't catch it.

13 DR. LOVELESS: No, I agree totally.

14 VICE CHAIRPERSON BENBOW: Wade?

15 DR. BOYKIN: Yes, with regard to these 12

16 studies on this slide here, you recognize they all

17 have flaws methodologically, but are there any kinds

18 of tentative inferences you can draw from these

19 particular studies?

20 DR. FERINNI-MUNDY: Actually, we're still

21 really working on that. It's a little bit early. We

22 have to decide whether these flaws outweigh what we

23 actually might be able to say.

24 Part of the issue has to do with the

25 outcome measures, which vary greatly on these kinds of

26 studies. And some of them will feature only items

 81
1 that are aimed at testing students' ability to solve

2 “real world problems.” Others are more standardized

3 measures that include a range of outcomes. So I think

4 it's a mix of having our mathematician experts take a

5 look at these outcome measures so that we can say

6 something about what the results would mean. So we're

7 really mid process on that one.

8 VICE CHAIRPERSON BENBOW: Bob?

9 DR. SIEGLER: I'd like to ask Russell a

10 question about the formative assessment work that you

11 talked about.

12 If I understood it right, kids are not

13 only getting instruction, the teachers are getting

14 information, but also the computer program in some or

15 all of the studies is generating problems that are

16 designed to remedy the children's learning

17 difficulties. Was that a misunderstanding on my part?

18 DR. GERSTEN: Bob, that's only the case in

19 several of the enhancement studies. So when we look

20 at the whole set of ten there is a condition where the

21 teachers and sometimes the kids get the numbers, but

22 that's it. They get the feedback. The enhancement

23 studies, that smaller set with the special education

24 students, is where they get, in most cases some

25 additional, either information for instruction or

26 additional specific ideas for how to teach the kids.

 82
1 Is that clearer?

2 DR. SIEGLER: Yes. You might want to

3 consider the older literature on adaptive computer

4 assisted instruction as another way of thinking about

5 formative assessment, because here it isn't the

6 teacher that is getting the formative information but

7 rather the computer program is getting it for itself

8 and generating problems on the basis of that.

9 DR. GERSTEN: Those studies didn't come up

10 in the search. I think some ideas and leads on those,

11 I'm dimly familiar with them, but I think they would

12 be appreciated and we could look at those. We can

13 talk to Abt about expanding the search to look at

14 those.

15 VICE CHAIRPERSON BENBOW: Liping?

16 DR. MA: Yes, do you have any research

17 available about the relationship between real world

18 problem and regular world problem?

19 DR. FERINNI-MUNDY: We have research

20 studies in both areas that we're looking at, but I

21 don't recall that we have any that actually looked at

22 the relationship between the two. So if you know of

23 something or if others do, that would be helpful to

24 us.

25 DR. MA: Thank you.

26 VICE CHAIRPERSON BENBOW: Valerie?

 83
1 DR. REYNA: Thank you. I have a question

2 about tier-three evidence and just what you're

3 thinking was. I should say at the outset that that

4 level of evidence, qualitative research is certainly a

5 valid scientific method.

6 That having been said, the question for

7 your group in particular is really about efficacy, I

8 would think, instructional practices, that the

9 question ultimately is one of efficacy. So, what was

10 your thinking about inferences from samples to

11 populations or to questions of efficacy from tier-

12 three level research, as you characterize it?

13 DR. GERSTEN: That's something we've

14 discussed and thought a lot about. We do not

15 exhaustively review tier-three studies. But if there

16 is a study, and it's based on either the Panel's

17 judgment or the author's judgment, that helps us frame

18 an issue or interpret findings or interpret findings

19 that are erratic. So it's only used to aid but there

20 are no results emanating from those studies.

21 Definitely the ideas and concepts there can be used

22 for ideas for future research or to help us frame

23 current understandings of issues.

24 DR. REYNA: So you're saying you're using

25 them for theoretical purposes? And would there be any

26 sense of which evidence should necessarily bear on

 84
1 theory?

2 DR. GERSTEN: Evidence, well, so we're

3 using them if they help understand a phenomena or a

4 pattern or finding.

5 DR. REYNA: So you're saying that

6 qualitative research allows you to infer causal

7 mechanism?

8 DR. GERSTEN: If there are ideas in the

9 published literature that help us understand

10 phenomena, that’s helpful. So that's what they're

11 used as, as basically sources for ideas.

12 DR. REYNA: I won't continue the debate,

13 but what I'm saying is, therefore, this would be a

14 source of speculative opinion and it would be marked

15 as such?

16 DR. LOVELESS: Yes, it would be marked as

17 such and it would be used to generate future

18 hypotheses. For instance, in the cooperative learning

19 field we have this effect, this effect that's

20 statistically significant.

21 We may want to propose, and we haven't

22 gotten to this point, but we may want to propose

23 future hypotheses that could be tested as to what are

24 the mechanisms of this intervention that are

25 generating this positive effect. And the tier-three

26 studies could help us frame those hypotheses. It

 85
1 should be clearly labeled as not somehow causally

2 verified in the literature.

3 VICE CHAIRPERSON BENBOW: Do we have any

4 more questions? Well, seeing and hearing none, thank

5 you. We'll move on to the next group, teachers and

6 teacher development.

7 DR. LOEWENBERG BALL: Okay, I'm reporting

8 on behalf of the Teachers Task Group. The names of

9 the members of this group are on the slide and Ken

10 Thompson is the staffer with our group who's done a

11 great deal of work to help us.

12 So first, I just wanted to review for all

13 of you what the four questions are that the task group

14 is considering. We will only be reporting on question

15 one at this meeting.

16 The first question has to do with the

17 relationship between teachers' mathematical knowledge

18 and their students' achievement.

19 There are subsequent questions that we've

20 begun to work on and that you'll hear about at

21 upcoming meetings that include what is known about

22 programs that help to increase teachers' knowledge,

23 both pre-service and in-service. It also includes the

24 relationship of what teachers learn in those programs,

25 evidence about what they in fact learn, and the

26 relationship to, in particular, their students'

 86
1 achievement as a result of their opportunities to

2 learn.

3 One of the other areas has to do with

4 elementary math specialists. What we've been able to

5 determine so far is that we won't be uncovering

6 studies that link, as the question asks, the

7 effectiveness of math specialists programs or math

8 specialist staffing to student achievement. We will

9 go ahead and begin to explore what the range of models

10 is that exists out there, what the differences are

11 among them and what's known about what kinds of

12 qualifications are used to place people into such

13 roles. We'll also be looking internationally to

14 understand the ways in which a math specialist may be

15 employed in other countries.

16 And finally, we'll be looking at what's

17 known about strategies for recruiting and retaining

18 really highly qualified, skilled teachers in teaching

19 mathematics. Both of these last two areas will

20 probably turn out a little bit differently than our

21 first two. For example, in question four we'll have

22 to look at data and research beyond specifically

23 mathematics teaching, to understand what's known, in

24 particular, about the recruitment and retention of

25 teachers in general.

26 Question one is the one that looks at the

 87
1 relationship between teachers' mathematical knowledge

2 and their students' achievement. So our group thought

3 it would be useful just to reiterate for ourselves why

4 this is such as important question for the Panel. And

5 we saw three essential reasons.

6 There is substantial research and

7 anecdotal evidence that U.S. teachers’ levels of

8 mathematical knowledge are often too low for the work

9 they're being asked to do. That is, they don't know

10 math deeply or well enough.

11 There are many ways people describe this.

12 There is both robust research evidence on this and

13 plenty of anecdotes floating around. And our charge

14 was not to try to trace the documentation of that

15 weakness, but it is what compels this question.

16 We also wanted to note that there's an

17 increasing trend of increased requirements for

18 American students to take more mathematics. So for

19 example, in my own state, Michigan, where we've just

20 moved to a requirement whereby all students will take

21 four years of high school mathematics and that high

22 school mathematics is actually shaped at the state

23 level. You can see the increasing need to have

24 qualified teachers who can deliver that content to a

25 wider range of students than ever before.

26 And finally, we'll say a little bit more

 88
1 about this. There are some critical areas that we're

2 going to try to display still more than we have

3 already, in which there is a significant need for

4 qualified teachers to be teaching. And let me just

5 show you briefly what those are.

6 One is to look at the likelihood that a

7 minority student or a student living in poverty will

8 have a teacher who's either certified in mathematics

9 or has a major or minor in the field. Look at this

10 chart taken from the 2003 Condition of Education

11 Report. They are not as recent data as you might like,

12 but I think it helps to exemplify the problem. You

13 can see that minority students or students living in

14 poverty have roughly twice as high probability of

15 having a teacher who does not hold a major or minor in

16 the field or isn't certified in mathematics. You can

17 see that only science has a situation that's about

18 that dramatic.

19 Another way to think about it is to look

20 at the particular problem of high school and middle

21 school teaching. This graph shows percentages of

22 middle school and high school students who have

23 mathematics teachers who are qualified by either of

24 these criteria, hold a major or minor in the field or

25 are certified in mathematics.

26 You can see mathematics really sticking

 89
1 out up there, that dark red bar. This represents

2 middle schools students. So roughly one in four

3 middle school students is being taught by a teacher

4 right now who does not hold either of those ways of

5 being qualified to teach mathematics, and even one in

6 ten at the high school level. So these seem to us to

7 be critical reasons to highlight this in our report.

8 So what does that mean we might need to

9 know to inform policy better? One of the basic

10 questions that question was about is, how does

11 teachers' mathematical knowledge relate to students'

12 learning. But more than that to inform good policy we

13 would have to know how much mathematics do teachers

14 need to know to be effective and what mathematics do

15 they have to know, and in what ways. And you can see

16 why those subsequent two questions matter, because

17 simply knowing the teachers' knowledge in mathematics

18 relates to student achievement, which is something

19 everybody already believes anyway, doesn't provide all

20 that helpful guidance for cost effective and effective

21 interventions to improve and increase teachers'

22 mathematical knowledge.

23 If you think about this question about

24 what is the relationship is between teachers'

25 mathematical knowledge and their students'

26 achievement, there are two basic methodological

 90
1 issues.

2 One is how would you measure teachers'

3 mathematical knowledge and what would you mean by

4 students' mathematical achievements? I just want to

5 briefly say how these two things are treated so far.

6 So for measures of math teachers'

7 mathematical knowledge, in our review of the

8 literature we're looking at three different ways of

9 measuring teachers' knowledge.

10 One is teacher certification in

11 mathematics that is indirectly also the result of a

12 test. But it's separate in the studies from the

13 second type, which looks at teachers' educational

14 attainment in mathematics measured either by their

15 degree, a degree in mathematics or levels of course

16 taking.

17 And then we have what we are calling

18 currently more direct measures, that is measures of

19 teachers' mathematics of the curriculum they have to

20 teach or of the content of their level or beyond.

21 There's not as much research in this area, but we

22 consider these to be less indirect than the first two.

23 And then the question is, what about

24 students' mathematics achievement? How might

25 researchers examine this? The studies that we

26 selected and considered to be high enough quality were

 91
1 longitudinal data on students' performance using

2 pre-test controls. So that what we're looking at

3 essentially are either longitudinal growth models,

4 gained scores or some form of covariant adjustment

5 models. These are not cross sectional studies that

6 we're using.

7 Further, actually one would want to know

8 about what the outcomes are in gained scores and about

9 how teachers' mathematics impacts the instruction that

10 students receive and thereby the learning that they

11 are able to accomplish. Our studies don't have that

12 sort of measure and ideally we need more research that

13 traces this a bit more closely inside the so-called

14 black box of instruction so that we could make better

15 policy decisions.

16 Now I'm just going to report briefly what

17 we've learned in those three ways of measuring teacher

18 knowledge, teacher certification, course work and

19 direct measures. Looking at the effect of teacher

20 certification in mathematics on student achievement,

21 there are really three issues that we uncovered in the

22 studies that we examined.

23 First of all, it's worth noticing that

24 teacher certification is a pretty inexact measure of

25 what teachers actually know. There are some

26 substantial problems of selection bias in these

 92
1 studies. That is, it doesn't isolate the variables

2 very well. There are other things that might coincide

3 with certification that would make it difficult to say

4 that what you're measuring alone or testing alone is

5 certification.

6 And further, and we'll have to do a better

7 job than we've been able to do so far to sort out the

8 different things that actually are called

9 certification. There isn't some uniform single thing

10 that's being examined here.

11 But that said, in these studies the effect

12 of teacher certification remains somewhat ambiguous.

13 And maybe because of what I just said you can see why

14 that might be methodologically.

15 Of the other studies that met our quality

16 standards, four showed a positive effect of teacher

17 certification on students' learning and four others

18 showed no effect. Actually, it's not true that no

19 results are significant. Some of the results were

20 statistically significant.

21 There are some complications in these

22 studies partly because of the inexactness of this as a

23 measure of teachers' knowledge. But we consider this

24 to be, nonetheless, a very important policy question

25 since it is one of the ways that policymakers can

26 intervene to assure qualified teachers. So we're

 93
1 going to be surveying what the specific certification

2 requirements are, particularly at the middle school

3 level since that's, given what I said earlier, a

4 particular area we think we could say something about,

5 and try to look a bit more closely at studies that may

6 compare teachers with different kinds of

7 certification.

8 So now I'm going to move onto what we were

9 learning about teachers' mathematical study, which is

10 the second of the ways that math knowledge could be

11 measured.

12 Here we're looking at teachers' college

13 level mathematics study. One issue is that course

14 taking isn't a direct measure of what someone knows.

15 For example, there would be a lag effect of what they

16 learned a number of years ago and what they now know.

17 It is not necessarily going to tell you what they know

18 at the moment.

19 And furthermore, a different sort of

20 problem is that these courses, particularly

21 mathematics, may not correspond very closely to what

22 it is that teachers actually teach. If you look at

23 the content, for example, of the math major, many of

24 the courses in the math major don't align all that

25 closely with the content of the high school

26 curriculum. So there may be some issues of selection

 94
1 bias here.

2 But that said, there's slightly stronger

3 results here that are worth paying attention to. We

4 see more consistent findings here than we do in those

5 certification studies that I reported. Of the nine

6 studies that met our criteria for high quality

7 research, seven of those do show a positive impact of

8 teachers' course taking or level of attainment on

9 student achievement. One showed no impact and one

10 showed negative impact.

11 It’s worth pointing out here, given the

12 Panel's purview, that most of these focus on secondary

13 school students and we did not uncover evidence that

14 related teachers' course taking at the college level

15 positively affecting student achievement at the

16 elementary level. And further, again to say, we don't

17 know very much about what these courses are about. So

18 these are still somewhat distal from teachers' content

19 knowledge. But still, we're seeing that there's a

20 positive effect, which is a signal in the direction

21 that many people would believe anyway.

22 So the third area has to do with studies

23 in which there is more direct measure of teacher

24 knowledge using tests of some form. While there may

25 be closer estimates of what teachers actually know,

26 overcoming some of the problems I pointed to in the

 95
1 other two areas is still an issue.

2 Some of these measures haven't been

3 validated so it's not obvious what we should make of

4 measures invented by researchers to study teacher

5 knowledge. We also found extraordinarily few studies

6 of this kind.

7 Still, the numbers of studies that met our

8 criteria allowed us to say some things about what we

9 were learning. We had eight studies, five of which

10 met our standards. Two of those showed positive

11 effects that were significant. One showed positive

12 effects, although not statistically significant. And

13 two found more ambiguous results.

14 So we think that, generally we feel

15 supported in saying that hereto, there is support for

16 the notion that teachers' mathematical knowledge

17 having a positive impact on students' achievement.

18 So, if we were to make two tentative

19 claims at this point I think there are the two we feel

20 that we can say so far, that "knowing" mathematics is

21 likely a significant factor in teaching effectively.

22 Now, you might say, well, you didn't have to do all of

23 this to come up with that.

24 I'll say a little bit more about what we

25 think will be needed in order to be able to do

26 something policy-wise with that. And "knowing" is in

 96
1 quotes here precisely because what I've been saying

2 over and over is, none of these gets very close to the

3 notion of what exactly does somebody have to know and

4 in what way to teach well. We think that given the

5 scale problem of how many teachers we're talking about

6 it would be useful, from a policy perspective, to know

7 exactly where to target, how to increase and improve

8 teachers' knowledge.

9 It’s worth underscoring again that the

10 notion of college level study may predict

11 effectiveness for secondary school teaching, but we

12 did not define that for elementary. So given the

13 critical nature of the need to improve teachers'

14 content knowledge at the elementary and middle school

15 level, this gap in the research does suggest that we

16 have a need to probe more closely into the

17 mathematical knowledge needs for the K-8 level.

18 We don't know, as I've been saying, enough

19 about what teachers actually have to know. We don't

20 know quite enough about how teachers' knowledge

21 affects the quality of students' learning. That is,

22 how it interacts with instructional practices, for

23 example, or with teachers' knowledge of learning to

24 enable them to actually effectively address what

25 students produce in class or to design instruction.

26 We also don't know enough how much course

 97
1 work makes a difference at different levels of

2 schooling. If you think of the practical implications

3 either through assessments or course requirements, one

4 will need to know that simply saying, well, let's just

5 have everybody take a major, clearly isn't supported

6 by what we've been able to see so far.

7 So we wanted to end by saying what we

8 think might be needed to inform policy better. We

9 think there should be investments in better and more

10 reliable, more proximal measures of teachers' actual

11 mathematical knowledge. We need a better way to

12 understand how the teaching of mathematics demands

13 mathematical knowledge so that we can target the

14 research we're doing in a more focused way on the

15 actual mathematical demands of the work. We'd like to

16 see studies that had better designs that would permit

17 stronger causal inferences that, for example, overcame

18 some of the problems we uncovered. We need studies

19 that do a better job of isolating variables,

20 overcoming selection bias, and looking more closely at

21 the impacts on instruction.

22 So in terms of just telling you where

23 we're going next, we'll be trying to gather more

24 detail about certification requirements including not

25 only what's required to get certification, but what

26 the assessments, what the cut scores are and the

 98
1 nature of mathematics asked on some of those tests.

2 We're going to look more closely, as I said, at

3 teacher qualifications at the middle school level.

4 We’ll be trying to compare effects of different forms

5 of certification and we'll try to improve the way

6 we've consolidated our estimates of effects across the

7 studies that we've examined.

8 That's really the detail of our report on

9 question one. You'll be hearing more about questions

10 three and four at the June meeting and question two, I

11 think, by the following meeting. Question two, again,

12 is the programmatic intervention question.

13 VICE CHAIRPERSON BENBOW: Since I'm

14 controlling the microphone, let me go with the first

15 question. Could you clarify for me, when you say the

16 evidence regarding elementary math, is it that there

17 is no evidence assessing the importance of mathematics

18 for our elementary teachers' effectiveness? Or is it

19 that there is evidence that is showing no effect?

20 DR. LOEWENBERG BALL: No, no, I didn't

21 mean to say anything that broad. It's that the course

22 work studies don't show an effect of course work on

23 teachers. But in the third grouping of direct

24 measures, one of the high quality studies showed a

25 significant effect at the first grade and third grade

26 level of teacher's mathematical knowledge. If I said

 99
1 we don't know that it makes a difference at

2 elementary, I didn't mean to say that. I meant that

3 the course work studies don't show us that.

4 VICE CHAIRPERSON BENBOW: Thank you.

5 Doug?

6 DR. CLEMENTS: Your third question about

7 math specialists speaks to more instructional effects

8 rather than pure math knowledge. So I was wondering,

9 as a newcomer to the Panel, if you guys had

10 discussions similar to Shulman's seminal pedagogical

11 content knowledge versus content knowledge and whether

12 you were going to even try to look at the former?

13 DR. LOEWENBERG BALL: Well, we haven't yet

14 found studies that examine the effects of math

15 specialists on anything. So if we were to design

16 studies we'd presumably want to know how having a math

17 specialist effects instructional practice or student

18 learning. But we're not finding that.

19 DR. CLEMENTS: I'm sorry, all I meant is

20 that it looked like you were looking at instructional

21 kind of issues, but most of your presentation was

22 about mathematics content knowledge, right? Not about

23 pedagogical content knowledge, which I know you've

24 contributed to that literature. So I was interested

25 as to whether you thought that was just too difficult

26 a problem to also address or there just being no

 100
1 studies or was it just the decision, if not looking at

2 specific kinds of knowledge that are relevant

3 instructionally?

4 DR. LOEWENBERG BALL: Yes, I think maybe

5 what you're asking is, there are many kinds of

6 knowledge that potentially influence teachers'

7 effectiveness and this group appears to be focusing

8 primarily on content knowledge, and that's true. We

9 have been doing that. Although, as we look at the

10 literature we're open to looking rather broadly at

11 what is defined as mathematical knowledge. As it

12 happens, pretty much all the studies with maybe one or

13 two exceptions are looking at content knowledge

14 measured rather narrowly.

15 It will have to be one of the

16 recommendations of the Panel, however, to look more

17 broadly even with our intersection with the Learning

18 Processes group, which may help us to understand how

19 knowledge of students' learning of math might affect

20 teachers' effectiveness.

21 VICE CHAIRPERSON BENBOW: Vern and then

22 Wilfried?

23 DR. WILLIAMS: You may have looked at this

24 and possibly I missed it, but have you compared the

25 number of math courses and the types of math courses

26 taken by teachers in some of the high scoring

 101
1 countries, K-12, with teachers in our nation?

2 DR. LOEWENBERG BALL: Looking at, in other

3 words, requirements, and qualifications to teach at

4 different levels. We should do that. We haven't done

5 that yet. We've just begun now to look at

6 certification requirements. Initially we were looking

7 at studies that looked at effects. I think we should

8 broaden to look internationally. We've known we

9 should do that with math specialists and this would be

10 a close cousin, I think.

11 VICE CHAIRPERSON BENBOW: Wilfried?

12 DR. SCHMID: Two questions. First of all,

13 have you or will you look at, let's say, the effects

14 of professional development? Or will you do this only

15 indirectly through, let's say, assessing content

16 knowledge of teachers, which might or might not be

17 imparted by professional development?

18 DR. LOEWENBERG BALL: Our second question,

19 which we have not yet reported on will be examining

20 programs at the pre-service level and professional

21 development intended to increase teachers'

22 mathematical knowledge. And we'll be looking to see

23 whether and how they affect increases in teachers'

24 knowledge and their effectiveness. Is that what you

25 mean? Yes, we'll be looking at that directly. We

26 just haven't yet.

 102
1 DR. SCHMID: In the second question, the

2 question of retention of teachers, will you look at

3 the question of differential pay for mathematics

4 teachers?

5 DR. LOEWENBERG BALL: Yes, absolutely. So

6 teacher pay will be one of the aspects. We'll have a

7 grouping of different possible strategies for that and

8 teacher pay will be one of those.

9 DR. SCHMID: Well, I said, differential

10 pay meaning --

11 DR. LOEWENBERG BALL: A different pay for

12 math teachers.

13 DR. SCHMID: -- different incentives

14 specifically targeted to mathematics teachers.

15 DR. LOEWENBERG BALL: Yes, you said it

16 right and I said it wrong. That's what we would be

17 looking for.

18 VICE CHAIRPERSON BENBOW: Tom?

19 DR. LOVELESS: Have you done anything with

20 these studies to pool their effects, applied any

21 analytic techniques so that we could get an idea of

22 the size of the effects overall and whether or not

23 they're statistically significant?

24 DR. LOEWENBERG BALL: This has been one of

25 the challenges that we're still engaged in trying to

26 find a way to do that. So, I saw the Abt staff

 103
1 shaking their heads back there, not to say no, but

2 reminding us of this challenge. We have been looking

3 for a way to do that, that we think makes sense.

4 DR. LOVELESS: And then secondly, on the

5 literature addressing the question about college level

6 courses, do those studies differentiate between

7 courses taken in math education as opposed to

8 mathematics departments. It was mentioned earlier

9 about international patterns on this and the United

10 States really is an outlier in terms of our eighth

11 grade algebra teachers. At least TIMMS show most of

12 our algebra teachers in eighth grade received their

13 math education in a school of education. And around

14 the world, most eighth grade algebra teachers received

15 their education in math departments. So I'm wondering

16 if the studies allow you to take a look at where those

17 math courses were housed?

18 DR. LOEWENBERG BALL: I think there's

19 several questions embedded in yours. One is where the

20 math courses are housed. Another one, which you may

21 not have meant to ask, but needs to get asked, is that

22 some of these studies also look at the effects of

23 course taking and math methods. And interestingly,

24 those sometimes show stronger effects on student

25 achievement than the pure math content. So we haven't

26 broken that out well.

 104
1 And third, we are going to look very

2 closely at middle school requirements. So it's sort

3 of circling around your question. These studies don't

4 necessarily tell us where they're taking them. But

5 your question is pointing to something that we'll try

6 to get into in several different ways.

7 VICE CHAIRPERSON BENBOW: Sandra?

8 DR. STOTSKY: I know that you haven't

9 reached the second question yet. Do you have any

10 sense now whether any of these studies will be looking

11 at the pre-service programs, student teacher issues,

12 placement issues for student teaching, the evaluations

13 that are done as part of student teaching and what

14 those look at in relation to mathematics knowledge as

15 opposed to mathematics teaching? This is before a

16 prospective teacher exits --

17 DR. LOEWENBERG BALL: So part of what I

18 think your question might point at is how deeply we're

19 going to go into learning about what programs actually

20 are, as opposed to simply looking at programs and

21 whether they have affects. And I think that isn't a

22 question the group has had a chance to discuss yet,

23 but I think it would be helpful, in the same way that

24 looking at certification requirements for some of

25 these details helps with question one. Because I

26 suspect, as you do, that there will be clues there

 105
1 about the preparation that would be useful to uncover.

2 We'll have to see. I think we'll just have to see

3 what we can do with that.

4 VICE CHAIRPERSON BENBOW: Are there any

5 more questions? If not, thank you. Oh, sorry.

6 DR. WU: What I believe part of the next

7 steps for question number one is to pin down the

8 nature of the knowledge teachers need to teach. I

9 think that's one of the main issues.

10 DR. LOEWENBERG BALL: Yes, so Wu's talking

11 about something we began to discuss here, is how far

12 our group may go into actually making some hypotheses

13 based on our judgment and what we've read about the

14 correct answer to the what question. Since the

15 studies don't uncover all that much about the what,

16 how far might we go at least framing what we think to

17 be reasonable hypotheses about that. And you're

18 right, that's one of our issues. That's why you get

19 to ask a question.

20 VICE CHAIRMAN BENBOW: Any more questions?

21 Okay, thank you. Now we'll have a report from the

22 Assessment Task Group, and giving that report will be

23 Susan Embretson.

24 DR. EMBRETSON: Well, it was mentioned

25 earlier that this was the first meeting of the

26 Assessment Task Group. This is a slide showing the

 106
1 members who were present. Actually, we have another

2 member who was present today and will probably meet

3 with us, Douglas Clements.

4 Our main charge, since this was our first

5 meeting, is to determine what kind of research

6 question we want to look at, or research questions,

7 plural. And we determined that really most important

8 was a single question with many different aspects.

9 And that question is, to determine the correspondence

10 of National Assessment of Educational Progress (NAEP)

11 fourth and eighth grade tests to selected state

12 accountability tests for validity in assessing

13 mathematics proficiency.

14 Now we limit the comparison to fourth and

15 eighth grades because that is where NAEP is available,

16 and NAEP, of course, is regarded as the national test.

17 Now when we look at validity, four aspects

18 are particularly relevant for comparing NAEP to the

19 state accountability tests. These four aspects are

20 content validity, substantive validity, consequential

21 validity and generalizability.

22 Let me say a little bit about what those

23 kinds of validities are for those of you who are not

24 familiar with that. Content validity is probably what

25 you're most familiar with in educational tests. It's

26 representing the content of mathematics. Tests are

 107
1 constructed with blueprints, which outline topic areas

2 and their relative representation. That goes toward

3 the content validity of the test.

4 Now, the content validity of the test

5 needs to be made clear so people can compare it to

6 say, some idea they have about what should be in that

7 content.

8 The substantive aspect has to do with the

9 underlying processes and theory about what is going

10 into solving the test items. This kind of area might

11 make contact more with our Learning Processes

12 sub-panel or also helps define the nature of what is

13 tested by particular items. Items can be formed on

14 the same content topic in many different ways and,

15 hence, can involve different processes required by the

16 students.

17 Consequential validity is the impact of

18 the test on defined groups of people such as gender,

19 racial ethnicity, English as a second language, or

20 disabilities.

21 Finally, generalizability looks at the

22 impact of some features of testing that may impact

23 score levels. Such as whether or not the test was

24 presented by computer or paper and pencil, whether or

25 not the questions are given in multiple-choice format,

26 a constructive format, so on like that.

 108
1 So these are the aspects that we think are

2 relevant to look at when comparing NAEP to the state

3 accountability tests.

4 Now what are the possible differences

5 between NAEP and the state tests? Well, first the

6 content may be weighted very differently between and

7 within strands. And we don't know at this point

8 without looking at the blueprints for NAEP and for the

9 state tests just how the different content is

10 represented. Secondly, the cognitive complexity of

11 items may vary between NAEP and state tests. The same

12 content can be tested by items that vary substantially

13 in complexity. If you put in an extra sub-goal or

14 extra vocabulary, for instance, it becomes a more

15 complex thing. So there are many things that go into

16 cognitive complexity. And I think this is one of the

17 reasons why items, which measure the same content,

18 differ in difficulty.

19 Now three, we want to look at the

20 empirical difficulty of items that measure the same

21 content. The distribution of the difficulties, again,

22 may vary substantially between NAEP and the state

23 accountability tests.

24 A fourth thing to look at is tool

25 inclusion, calculators in particular, but also perhaps

26 manipulative materials. These, again, may have impact

 109
1 on assessed proficiency levels and affect test

2 validity, certainly generalizability, but also the

3 substantive aspect of validity. When you think about

4 calculator use, it may change the processes people are

5 employing to solve the problem.

6 A fifth thing to look at is test delivery

7 mode. We particularly want to look at differences

8 between computer based versus paper and pencil tests,

9 keeping in mind that computer-based tests are going to

10 increase in popularity as time goes on.

11 A sixth thing to look at is the

12 representation of items on NAEP versus the state

13 tests. And of course, we have item formats ranging

14 from true/false, multiple choice, constructed

15 response, word problems and so forth. Now what we

16 want to look at, of course, is how that impacts

17 proficiency as well.

18 So, our actual comparison variables are

19 pretty much what I just outlined. We're going to look

20 at proportional representations of content from test

21 blueprints, and cognitive complexity and conceptual

22 skill level of actual items. This is probably going

23 to be the most difficult because we have to have

24 access to items to look at items and further, we have

25 to have someone to do the looking. So this is going

26 to be a most substantial effort. We certainly are not

 110
1 going to do this with every state. We are going to do

2 a selected subset of states to look at this issue.

3 We also have not discussed what will go

4 into the measures of cognitive complexity in the Panel

5 and hopefully we'll discuss some of that today.

6 You would think the empirical item

7 difficulty is easy to get, but it isn't. You're

8 probably going to have to go on site and link

9 particular items to difficulty. So that also needs to

10 be done.

11 With regards to tool inclusion, there are

12 some tests that allow tools and some that do not. So

13 we want to look at that and try to understand what the

14 impact is.

15 We will look at test delivery mode, which

16 I've mentioned before, and item format, particularly

17 as crossed with content. It could be that in some

18 tests certain content is only measured by a particular

19 format, and we want to understand better what impact

20 that would have.

21 So, this gives you kind of an overall view

22 of what we're looking at and how that's related to

23 validity. The areas are proportional representation,

24 complexity, and skill level, item difficulty, tool

25 inclusion, test delivery mode, and item formats. Here

26 are the areas that they are relevant to in validity.

 111
1 I put down complexity and skill level

2 under content because state blueprints do include

3 cognitive complexity as part of their stratification.

4 As far as I know there's not a great deal of

5 satisfaction in many locations with this variable, but

6 we want to look at that and see how that is determined

7 and what their categories are.

8 That is also relevant to the substantive

9 aspect of what is really behind the responses of

10 students to items as well. I have also included item

11 difficulty in a variety of locations; in particular,

12 the generalizability is what comes most to mind. I

13 think it also might be related to the substantive

14 aspect of what's being measured.

15 Tool inclusion, calculators, as I

16 mentioned, might change the nature of the problem

17 solving process for the student and so we put that

18 under generalizability and consequential validity.

19 And finally, item formats can also make a

20 difference of the validity of a measurement. We

21 should note consequential validity is everywhere. We

22 would like to see the impact of varying

23 representations of content, varying levels of skill,

24 varying levels of item difficulty. I think this is

25 most easily available from NAEP, but I don't know how

26 easily available it is at the state level.

 112
1 So this describes the groups that we're

2 going to be looking at under the consequential

3 validity aspect. So that's the conclusion. That's as

4 far as we have gotten so far. Now we're going to

5 determine how we're going to get this data.

6 VICE CHAIRPERSON BENBOW: Thank you. As

7 you can see, we already have lots of questions.

8 Wilfried?

9 DR. SCHMID: With the questions you have

10 outlined, what kind of policy recommendations do you

11 think you would potentially be able to make, depending

12 on what you find?

13 DR. EMBRETSON: Well, that's hard to say,

14 because we don't know what the status quo is on this

15 relationship. I mean it could be that we'll look at

16 the states and we'll think that the content there is

17 better representative of some of the concerns of the

18 other task groups in this Panel. Or we might not. We

19 might like NAEP's content. I don't know what we're

20 going to find. I'm also, for the first time, a member

21 of the Learning Processes group and I've had some

22 ideas of content that I want to look at, in

23 particular, to see if it's represented on the

24 different tests.

25 DR. SCHMID: The Department of Education

26 has commissioned the study of the validity of NAEP

 113
1 specifically. I have been involved with that effort.

2 It is clear to me that to study NAEP alone and to see

3 how valid it is in the sense of phrasing questions

4 correctly mathematically, having an alignment between

5 the test questions and the NAEP framework, and having

6 confidence that the methodology of the test questions

7 is okay. These are very difficult questions to answer

8 about NAEP alone. I am rather surprised if you

9 propose to answer questions of this sort not just

10 about NAEP, but also about several state tests.

11 DR. EMBRETSON: No, I don't think we're

12 doing the same thing that you are. We're really

13 looking more at the bigger picture especially in terms

14 of content validity area and proportional

15 representation of items. We're not going to check the

16 reliability of categorization of items like you are in

17 the various areas, nor to check their validity from

18 the perspective of mathematical principles. So, no,

19 we're going to do a more global analysis of this.

20 DR. SCHMID: But then again --

21 VICE CHAIRPERSON BENBOW: Last question,

22 then I have to give some others a chance. Then we can

23 come back to you.

24 DR. SCHMID: Well, I mean, it seems to me

25 that from a point of view of actually coming up with

26 policy recommendations we would need to know whether,

 114
1 first of all, NAEP is a reliable measure of whatever

2 questions you ask about it. And then secondly, how

3 well the state tests track that measure.

4 DR. EMBRETSON: Well, I agree with that.

5 We can look at other features of NAEP as available in

6 the literature. The standard areas around those scale

7 scores and so on. We've already requested some data

8 on that by level. However, you might mean reliability

9 in a difference sense. I'm not sure if you have in

10 mind the reliability of the categorizations of items

11 but fulfilling the framework. That's a different sort

12 or question.

13 VICE CHAIRPERSON BENBOW: Russ?

14 DR. WHITEHURST: Wilfried asked several

15 good questions and raised several of the issues I

16 wanted to raise. I still don't understand the purpose

17 of the exercise you're engaged in. I don't understand

18 how the analysis you intend to conduct will inform

19 matters before the Panel. I'm not sure why the

20 relationship between state NAEP, between state tests

21 and NAEP is an issue. I'm not saying that it couldn't

22 be an issue, but I don't think you've articulated what

23 the issue is that you're addressing. At least I don't

24 understand it.

25 To take a point that Wilfried has made and

26 expand it a little, there are a number of studies

 115
1 going on in my office and in the department with

2 respect to NAEP, to state NAEP correspondence. So it

3 would be good, if you're not aware of them, to become

4 familiar with them so that you don't try to do work

5 that's duplicative.

6 It strikes me that another area perhaps

7 that would be as interesting or more interesting would

8 be the relationship between what's being assessed in

9 this country, whether it's NAEP or state tests, which

10 are most frequently based on the NAEP framework,

11 versus international assessments. What do we know

12 about what skill set it takes to be proficient on

13 NAEP, how that would correspond to levels of

14 proficiency that might occur in high-performing

15 countries. And in fact, there are data that relate to

16 NAEP standards to international standards that might

17 be well worth exploring.

18 But to come back to the principle

19 questions, it's just not really clear to me why you're

20 doing what you're doing. I think that's probably a

21 matter of explanation.

22 DR. EMBRETSON: Okay. I'm going to speak

23 for myself. This panel has only met for three hours.

24 So I certainly can't speak for that many people. But

25 I believe that the state tests are closer to teaching

26 than is NAEP because there are more consequences of

 116
1 the state tests.

2 And so by understanding, in fact, how

3 proficiency is regarded at the state level, which is

4 closer to teaching, I think we'll be in a better

5 position to consider the factors that might be changed

6 or recommended to be changed there and also on NAEP.

7 NAEP is included in the mix because that's the

8 national test.

9 VICE CHAIRPERSON BENBOW: Anybody from the

10 committee like to add anything? I figured you would,

11 Tom.

12 DR. LOVELESS: And this gets to Russ'

13 question, the feeling of the task group was that when

14 anyone hears the word assessment, certainly they think

15 of NAEP. It's known as the national report card. If

16 this National Math Panel issues a report that does not

17 discuss NAEP it is in essence overlooking the most

18 important test that the nation feels that it has that

19 represents the United States' performance by students.

20 So, it seems illogical for us to say anything about

21 assessment without saying something about NAEP. Now

22 we hope that the studies that are ongoing will provide

23 some illumination of what these issues are.

24 The second reason, the second point I'd

25 like to make is that we included state tests because

26 those are the tests today in all 50 states, students

 117
1 take them in grades 3-8 as required by No Child Left

2 Behind. Those, the results of those tests have

3 consequences for schools. Schools are being held

4 accountable for the results of those tests. So again,

5 to issue a report from the National Math Panel without

6 saying something about those assessments also seemed

7 to us illogical. So that's the justification for NAEP

8 and the state tests.

9 VICE CHAIRPERSON BENBOW: Okay, Russ, yes?

10 DR. WHITEHURST: Since Tom was looking at

11 me during most of those comments. Let me just try to

12 be clear, I think it is quite important for the

13 National Math Panel to address what is being assessed

14 with NAEP and what is being assessed at the state

15 level. I think that's a critical issue.

16 What I was attempting to say is I didn't

17 understand your framing of that problem and how it was

18 going to generate answers that are relevant to policy

19 concerns. To be specific, it would be of terrific

20 interest, and I think you'll see an answer from my

21 office before the Panel's final report, to find out

22 what is the relative difficulty of state tests versus

23 NAEP tests in defining proficiency. Or just have

24 states set the bar higher or lower than is set on

25 NAEP? And I just didn't see that kind of issue, which

26 is kind of a natural policy issue emerging from the

 118
1 framework that had been provided.

2 DR. EMBRETSON: We understand that there's

3 some data available on that as well that we can get

4 from the Department of Education.

5 VICE CHAIRPERSON BENBOW: Let's move onto

6 a couple more topics, Wu is next, Wade and then Bob.

7 DR. WU: Okay, I think what I want to ask

8 really is in some sense colored by what Russ asked

9 earlier. It seems to me that one of the main reasons

10 for the founding of this Panel was that the student

11 achievement in mathematics is behind international

12 standards, levels. And so it seems to me that it

13 would be good to look, we have to look at NAEP, yes,

14 but the, one of the key questions is, if not the key

15 question, is whether NAEP is measuring the right

16 thing. Whether it's up to level.

17 And so it seems like, Russ says that

18 you're doing the international comparison. And is our

19 Panel going to duplicate or is it going to be a

20 cooperative effort?

21 DR. LOVELESS: The department has

22 conducted one study comparing NAEP and TIMMS, TIMMS of

23 course being the international test. But I think

24 Russ' point was a little different if I understood it

25 right. They have national assessments and it would be

26 comparing NAEP to those national assessments of other

 119
1 countries. And I'm unaware of any study that's done

2 that.

3 DR. WU: I thought that's what Russ was

4 saying, that you are doing that?

5 DR. LOVELESS: Russ was suggesting that we

6 do that. We haven't discussed that yet. Again, we've

7 only met three hours, so we're not really ready to

8 issue a report yet.

9 DR. WU: Well, I don't ask for report. I

10 ask for declaration of intention. But Russ says, can

11 you confront him, I mean, maybe --

12 DR. LOVELESS: No, Russ' suggestion is

13 very good and we're meeting this afternoon actually

14 and --

15 VICE CHAIRPERSON BENBOW: You'll take it

16 under advisement.

17 DR. WU: No, no, no, the issue is I want

18 Russ to clarify. I thought you said your office was

19 conducting an international comparison of NAEP with

20 other countries' assessment. Is that right?

21 DR. WHITEHURST: What I said was that

22 there are studies that, for example, draw conclusions

23 about how many students in Singapore would be judged

24 to be proficient on NAEP by cross walking results on

25 international assessments. And I think those findings

26 are pertinent and relevant to what the panel is

 120
1 considering.

2 VICE CHAIRPERSON BENBOW: All right, Wade?

3 DR. BOYKIN: I know it's early in the game

4 for you all on your sub-panel, but in the executive

5 order it explicitly says that this particular

6 sub-panel should address the role of assessment in

7 promoting math proficiency. I didn't quite hear that

8 issue being addressed in your comments. I know it's

9 early. Is that something that's going to be addressed

10 directly?

11 DR. EMBRETSON: Well, you know, tests by

12 their nature are involved in math proficiency and its

13 promotion. We have the tests to gauge where the

14 students are at and, you know, what we want to do to

15 move them to another location.

16 But our concern is merely with validity in

17 terms of what is being assessed and I think that's

18 like a first question. What kind of proficiency will

19 these tests promote exactly? That's the question of

20 validity. So, it's being examined first and then

21 given that we don't have a lot of time to put together

22 a lot of research, we might not get as far as you want

23 in that direction.

24 VICE CHAIRPERSON BENBOW: Bob?

25 DR. SIEGLER: Yes, I'd like to ask you to

26 look up the road a little bit with regard to what you

 121
1 expect this state versus state and state versus NAEP

2 set of comparisons to yield, because for sure there's

3 going to be some overlap and they're going to be some

4 differences. Neither the state tests nor NAEP are

5 immune from criticisms. Many of them are the same

6 criticisms, a few are unique to particular states or

7 to NAEP. But there's no gold standard there. It

8 isn't like anyone is just delighted with these tests,

9 as far as I know.

10 So what you'll have, it seems to me, is

11 this kind of set of descriptive results where it will

12 be extremely difficult to make recommendations

13 regardless of the particulars of how it comes out.

14 Whereas, the international comparisons strike me as a

15 more potentially promising way to go as well as

16 relations to TIMMS, PISA and other international

17 tests. I'm trying to think of what a good outcome of

18 this set of comparisons would be.

19 DR. EMBRETSON: Well, I think a good

20 outcome is to see how it interfaces with the other

21 concerns from the other task groups. We can't do that

22 until we know what the current situation is.

23 I mean, as far as the state level tests,

24 we could have tests that are perhaps formally certain

25 content, but they're tested at a very low level and

26 they're tested with item formats that are not well

 122
1 regarded. We could look at that and we could make

2 some conclusion about those kinds of items being not

3 as represented in the definition of proficiency.

4 VICE CHAIRPERSON BENBOW: Valerie?

5 DR. REYNA: Thank you, Susan. And by the

6 way, welcome to the Panel. I think you should take

7 all this as a sign of tremendous enthusiasm for the

8 topic that you're chairing.

9 I want to mention another theme that I

10 think is beginning to emerge as a cross-cutting theme

11 across the different subgroups, and that is this

12 notion of computation skill versus conceptual

13 understanding. We've seen that in the Learning

14 Processes group and we saw that in that very

15 informative summary that Deborah just gave in which

16 she used those two subtopics to assess the effects of

17 teacher training.

18 But I don't know whether that would fit

19 under content, under cognitive complexity, and whether

20 you can break that out. And what I'm talking about

21 is, to what degree do tests assess computational

22 skills, conceptual understanding, or both?

23 DR. EMBRETSON: Yes, that is intended

24 under the complexity level factor for sure. Whether

25 or not that can be reliably assessed looking at the

26 items, that we do not know yet. So I would say that's

 123
1 unknown. I think we need to start with NAEP, which

2 would be a smaller set and we need to look at to the

3 extent to which we can classify that.

4 VICE CHAIRPERSON BENBOW: Sandra?

5 DR. STOTSKY: I'd like to just pick up on

6 the point that was made that we have only had a few

7 hours together and we're still in the process of

8 clarifying some of the questions.

9 But one of the questions that we will try

10 to make a little clearer will be, from at least my

11 perspective, will be with regards to the state

12 assessments. We’d like to look at how state

13 assessments drive instruction, how they change

14 instruction, if there's any research or literature

15 available. Also we could look at how they drive

16 teacher training and how they drive professional

17 development, because they play key roles in all of

18 these areas. So there is a lot more that can be put

19 into this study beyond what its relationship to NAEP

20 is. That is what we haven't had a chance to really

21 think about i.e., how to develop more of these other

22 areas that will be part of this one overall rubric.

23 VICE CHAIRPERSON BENBOW: Deborah and then

24 Wilfried. Oh, Diane too. Wilfried, you're going to

25 have to be last but you'll get your turn.

26 DR. LOEWENBERG BALL: Valerie's question

 124
1 actually reminded me of something that I should have

2 asked you a long time ago, which is, is this group

3 planning to take up anything about teacher assessment?

4 DR. EMBRETSON: We had discussed that. It

5 was on the list and at the moment it's not on the

6 list. It probably is still somewhat open, I'd have to

7 say.

8 DR. LOEWENBERG BALL: It might be good for

9 at least the two groups to talk because obviously one

10 thing that we're dealing with in our group is the

11 psychometric quality of the measures being used to

12 assess teachers' knowledge.

13 DR. EMBRETSON: Yes, well, it's a good

14 idea, but at the moment we feel overwhelmed just

15 looking at the student level. So, I don't know.

16 VICE CHAIRPERSON BENBOW: Diane?

17 DR. LOVELESS: Just a piggy-back, we

18 actually, we were going to leave that to your group.

19 VICE CHAIRPERSON BENBOW: Diane?

20 DR. JONES: There's a lot of discussion

21 among many of the groups about what is algebra, for

22 example, what constitutes algebra. And I'm wondering

23 if it would be a worthwhile experience for your group

24 to maybe compare perhaps the eighth grade data with

25 some of the much older tests. Maybe you could look at

26 the tests that those of us in the post-Sputnik

 125
1 generation took, Iowa, Stanford, those are just the

2 ones that I can think of. I wonder if it would be

3 worthwhile to sort of compare what constituted

4 competency then versus now to satisfy some of the

5 questions about changes in our perception about what

6 constitutes algebra.

7 I ask that because policymakers are who

8 they are and many of them were engineers or people who

9 came through school in the post-Sputnik generation and

10 really, that's how we should be teaching algebra. It

11 might answer some questions to sort of do a

12 non-judgmental comparison about, you know, what were

13 the standards then versus what are the standards now,

14 and how might those standards differ in terms of

15 computational ability and conceptual understanding.

16 DR. EMBRETSON: That is a good question in

17 and of itself, I think. If we look at long term NAEP

18 I'm not sure how well that represents it, but it does

19 go back to 1978 at least. But you probably want to go

20 back further.

21 VICE CHAIRPERSON BENBOW: Skip would like

22 to address that and Skip is a member of the assessment

23 group.

24 DR. FENNELL: Diane, only in the sense

25 that the Conceptual Knowledge and Skills group is

26 addressing what you said to an extent as we review

 126
1 curriculum frameworks and as we review textbooks and

2 the like, but not so much in assessments, at least in

3 this country.

4 So I guess my point is I don't know that

5 we want to overload this group, but I think the point

6 is well taken. We may have it covered at least to

7 some extent in task group one.

8 DR. EMBRETSON: Well, if you have

9 suggestions, the committee is certainly going to be

10 open to hearing them. Some things that I have seen

11 from other committees is the topic of patterns as

12 algebra and that's something that we could look at.

13 We also could look at estimation and its

14 representation on tests. That is not a direct one

15 right now. If you have some suggestions then we

16 certainly want to hear them because I think that's the

17 hardest area. If you're going to change the content

18 you have to have more things being looked at in the

19 items.

20 VICE CHAIRPERSON BENBOW: Joan?

21 DR. FERRINI-MUNDY: This is just quick and

22 please don't interpret it as making more work for the

23 assessment group.

24 Maybe it's tied to Val's question, but in

25 looking at computational skill versus conceptual

26 performance I'm wondering if something about real

 127
1 world problems might be included there as a piece of

2 the complexity level discussion. Maybe our groups

3 could connect on what we've seen and some definitions.

4 DR. EMBRETSON: Yes, I was just going to

5 say, send us a definition.

6 VICE CHAIRPERSON BENBOW: All right,

7 Wilfried has been very patient.

8 DR. SCHMID: Maybe I'm being presumptuous

9 here, but it seems to me that the agenda that you

10 outlined, which would include looking at various tests

11 themselves and not just the outcomes, is just much

12 more than you have time for.

13 I would suggest that you don't even

14 attempt to look at test items or complexity, not

15 because it's not important or interesting, simply

16 because you do not have the time.

17 What I would suggest is that you take a

18 very hard look at data that exists or are being

19 generated about NAEP, various other tests and

20 comparisons of NAEP to TIMMS and to other

21 international tests. The question that Sandy raised,

22 what is the effect of state tests practically

23 speaking? Is it good? Is it questionable? I think

24 that looking at the individual test items, test

25 construction is just going to prevent you from doing

26 something that you can't do in the time you have.

 128
1 DR. EMBRETSON: Well, you may be right

2 about that, because this committee is just starting

3 now. Maybe we should have started earlier.

4 DR. LOVELESS: And if any committee

5 members want to volunteer to serve on the Assessment

6 Task Group we certainly would welcome you.

7 DR. SCHMID: I'll talk to you.

8 VICE CHAIRPERSON BENBOW: I'm on the

9 committee and as we've all been working on committees

10 we realize that we begin with big agendas. We're going

11 to look at what's possible in the time frame that we

12 have and we make compromises that sometimes hurt. We

13 end up not being able to address issues that we would

14 like to because we are time limited. Obviously, this

15 is a very important question. We want to make sure we

16 hit the right one and that we do prioritize our

17 questions. Give us a little bit of a break. This

18 could take a couple years to do well.

19 Well, at this point I'd like to bring this

20 session to a close. I would like to thank the public

21 and everyone for coming and attending and listening to

22 us today. I would also like to announce that the next

23 National Math Panel meeting will be hosted by Miami-

24 Dade College in Miami, Florida on June 6th. So if you

25 want to continue to hear the rest of the story, join

26 us in Miami.
 129
1 Thank you.

2 (Whereupon the meeting concluded at 12:11

3 p.m.)