Jennifer Bell-Ellwanger’s analysis does not refute the data that I offered in my article in

The New York Times. The editor at the Times required documentation for every single
fact in the article, and I supplied it.

Her biggest error is that she uses the wrong baseline data in presenting state test scores.
She chooses 2002 as her starting point, which is wrong. The baseline begins when
Chancellor Klein’s reforms begin, and not sooner. The Legislature approved mayoral
control legislation in June 2002, and Chancellor Klein assumed office in August 2002. He
spent the next six months conferring with consultants. He and the mayor announced their
plans and their pedagogical choices in January 2003. At that very time, students in the
city and state were taking the state tests. The chancellor had done nothing in the schools
prior to January 2003 to raise student achievement. His reforms were introduced into the
schools in September 2003. Thus, it is inappropriate for Dr. Bell-Ellwanger to take credit
for any gains registered on the state tests that were administered in January 2003.

The reason that Dr. Bell-Ellwanger wants to claim credit for that particular year is that
there were huge gains across the city in both reading and mathematics. But at the time,
Chancellor Klein did not claim credit for the gains of that year. It was widely understood
that his program would not start until September 2003.

David Herszenhorn wrote in the Times on May 21, 2003, “New York City public school
students posted sharp gains on the state’s standardized reading and writing test this year,
with striking double-digit jumps in some of the city’s poorest and historically lowest-
performing school districts…In both the city and the state, black and Hispanic fourth
graders significantly narrowed the gap with white and Asian students…City officials,
who might otherwise have been jubilant about yesterday’s results, offered a muted
reaction, saying that the gains were not broad enough and that the school system as a
whole was failing at least half the city’s children.”

A few months later, Elissa Gootman wrote in the Times on October 22, 2003:

Fourth graders across the state made stunning gains in their math scores last
spring, with even sharper increases in New York City…In the city, news of the
gains, which were particularly pronounced in the Bronx and in some of the
poorest-performing districts, elicited cheers among teachers and principals. But
not everyone greeted the news so enthusiastically:

The suggestion that city schools were on the upswing put Chancellor Joel I. Klein,
who is overhauling them, in a tricky position. While the chancellor’s critics
pounced upon the higher scores as evidence that the school system did not need
such an overhaul, some of his allies acknowledged that he would now be under
even more pressure to show gains next spring.

Mr. Klein’s reaction to the good news was muted, as it was to news of higher
reading scores last spring.
If Chancellor Klein thought he had produced those sharp gains, why was his reaction
“muted”? Obviously he did not take credit for the scores because the scores were
announced before his reforms had been introduced. Why does he take credit for them
now? Quite simply, the gains in that one year were so large that they became irresistible.
Fourth grade reading scores from 2002-2003 were up by 6 points, a larger gain than in
any subsequent year. Fourth grade math scores jumped by an astonishing 14.7 points, a
larger gain than in any subsequent year under Chancellor Klein.

In the interests of honesty, Ms. Bell-Ellwanger must deduct the scores of 2002-2003 from
her claims for this administration and revise the city’s gains on state tests accordingly.

Yes, I agree that the massive investment in test prep, interim assessments, and testing, as
well as bonuses for principals and some teachers tied solely to test scores has certainly
produced increases in state test scores. No question about it. But it is worth remembering
that the city’s education budget has grown from $12.5 billion to $21 billion, and one
would certainly expect that expenditures of that size should buy some improvements.

I don’t agree with Ms. Bell-Ellwanger that the state test scores are somehow more valid
than the federal NAEP scores. Quite to the contrary. The NAEP test is a far superior test
to the state tests; Congress has invested tens of millions of dollars in making NAEP the
best testing program in the nation. As for the state tests, they are not reliable because the
Regents and the State Education Department have repeatedly changed the testing pool by
manipulating which groups of students are tested and which are not. Just this past
September, the Regents decided that students who are LEP and who have tested
proficient may be excluded from state testing for up to two years after they passed the
English proficiency test. So, once again, the testing pool has changed, and some
potentially low-scoring students may be excluded. I feel that it is safe to predict another
big increase in state scores that will soon be released because of the removal of these
students from the testing pool.

The claim that the state tests are more valid because students practice for them is silly. I
recommend that Dr. Bell-Ellwanger read Harvard Professor Dan Koretz’s recent book
Measuring Up, in which he demonstrates that when students practice repeatedly for a test,
the test becomes less valid. I suspect she already knows this.

If students do well on the state test, but not on other tests because they didn’t practice for
those specific tests, then they really haven’t mastered the skills. If all they learn in a NYC
public school is to take state tests, then their skills will not be transferable to reading in
college or in the workplace. I have never heard anyone claim — aside from the NYC
Department of Education — that NAEP is less valid because students don’t practice for
it. The point of NAEP is that it is an audit test. Students are not supposed to practice for
it. As an audit test, it is far more consistent and meaningful than the state tests.

Dr. Bell-Ellwanger gets the NAEP results wrong. As I said in the article, NYC showed
significant progress in fourth-grade math, but those gains were suspect because of the
extraordinary number of students who were given accommodations (extra time, extra
help). I served on the National Assessment Governing Board for seven years, and I know
that a sharp change in the accommodations rate raises questions. In fourth grade math,
25% of NYC students got accommodations in 2007; this compares to only 12% who
received them in 2003. Why the sharp increase? NYC’s accommodations rate far
exceeded that of any other city. Los Angeles, with many more limited-English-proficient
students gave accommodations in this subject and grade to only 8%.

On the NAEP fourth-grade reading test, she again errs by giving the Klein regime credit
for the big gains of 2002-03, before he had introduced Balanced Literacy as the standard,
mandated reading curriculum in the elementary schools. The correct baseline year is not
2002 but 2003. New York City fourth-graders made no significant gains in reading from
2003 to 2007. I have often wondered why the Chancellor did not replace Balanced
Literacy after he saw these unimpressive results. NAEP showed no significant gains in
fourth grade reading for black students, white students, Hispanic students, Asian students
or lower-income students. NAEP found no narrowing of the gap between the city and
state from 2003 to 2007.

On the NAEP eighth-grade reading test, NAEP showed no significant gains for any racial
or ethnic group from 2003 to 2007. Fifty percent of black students and 49 percent of
Hispanic students in eighth grade were “below basic,” which is the lowest
classification (only 20 percent of whites and 21 percent of Asians scored so low on the
reading test). And these were students who had spent four years in the Children First
reforms.

There were some shifts in the racial gaps. But once again, Dr. Bell-Ellwanger mistakenly
uses 2002 data for reading, which do not belong to the Klein regime. In addition, her data
for the fourth-grade math scores are just plain wrong. I have the NAEP report in front of
me (it is also online and anyone can check: Google NAEP TUDA 2007 mathematics, p.
62).

From 2003 to 2007, these were the changes in the gaps, as reported by NAEP:

• On fourth grade reading, the black-white gap narrowed by four points, from 30
points to 26 points.
• On fourth grade reading, the Hispanic-white gap increased by two points, from 26
points to 28 points.
• On eighth grade reading, the black-white gap increased by five points, from 25
points to 30 points.
• On eighth grade reading, the Hispanic-white gap increased by six points, from 23
points to 29 points.
• On fourth grade math, the black-white gap decreased by 3 points, from 25 to 22
points.
• On fourth grade math, the Hispanic-white gap decreased by 6 points, from 24 to
18 points.
• On eighth grade math, the black-white gap decreased by six points, form 36 to 30
points.
• On eighth grade math, the Hispanic-white gap decreased by 3 points, from 29 to
26 points.

So, yes, there were some small improvements up and down, but not the large gains which
she erroneously claims. And NAEP does not say that any of these are statistically
significant changes.
As for the graduation rate, I do believe that it has been increased by dubious means,
including “credit recovery” and discharges. Elissa Gootman wrote an article in the Times
on April 11, 2008 (“Lacking Credits, Some Students Learn a Shortcut”) about students
who failed a course or never attended a single class, and who used “credit recovery” to
get enough credits to graduate. This is social promotion.

Regarding the discharge rate, a report was released just today by the Public Advocate's
office showing that it has increased significantly during the past few years, and that more
students are now being pushed out. This is another way of pumping up the graduation
rate. If social promotion had been ended, surely we would not see three-quarters of the
New York City high school graduates who enter CUNY community colleges requiring
remediation in basic skills. The rate of remediation may have been higher in the past, but
at least no one then claimed to have ended social promotion.

Also, though I did not mention it in my article, the city’s SAT scores were down this past
year for the first time since 2003. The city’s SAT reading average is 438, a full 50 points
behind the state average of 488. This is appalling when one considers that these are our
college-bound, if not college-ready, students.

To the question of independent verification of New York City’s improvement: Dr. Bell-
Ellwanger says that New York City won the Broad foundation’s annual prize in 2007; this
is true, but it was based on state scores alone. The NAEP scores did not appear until
November, several weeks later. I was not impressed that the city won a prize based on
state scores, since the state scores are as inflated as the city’s. Bear in mind that the state
has been reporting dramatic improvements in test scores for the past several years, yet the
state scores on NAEP were as flat from 2003-2007 as the city’s.

The Brookings report (on which I was an advisor) does not provide independent
verification of the city’s claims because it covered the span from 2000 to 2007. That
means that three of the seven years were unaffected by mayoral control, and one of those
years was 2002-2003, the year of sharp gains preceding the Klein reforms.

The report “Closing the Graduation Gap” by Christopher B. Swanson for Editorial
Projects in Education shows that New York City had a graduation rate of 50.5 percent in
2005. This placed us at #33 of 50 cities, behind the District of Columbia (57.5 percent)
and even Chicago (51 percent). The report showed that NYC had gained by 12.8 percent
from 1995 to 2005, well behind the much-maligned city of Philadelphia, which posted a
gain of 23.2 percent during the same period.

Every expert on graduation rates acknowledges that the problem of tallying the rate is a
huge mess because different districts use different methodology. Any report at the
national level must rely on those at the local level to give them the data, and the data
must be consistent from city to city. I am hopeful that Secretary Duncan will follow
through and require all states and cities to produce consistent data, using the same metrics
from district to district.

But so long as the city tolerates the misuse of credit recovery (social promotion), and so
long as it continues to discharge students in the ninth and tenth grades who have not
moved out of the district and are legally entitled to stay in school, then the graduation rate
figures are meaningless.

I know this is a vain hope, but I wish that once in a while the Department of Education
would admit error or at least show some humility. It might make them human.

Diane Ravitch