Too little and much too late, the Times finally reports on the state test score scandal

In yesterday’s front page story, entitled "On NY School Tests, Warning Signs Ignored," the NY Times' account of the state test score scandal left its own deficient reporting conveniently off the hook.

Anyone who was paying attention knew at least as far back as 2007 that there was rampant test score inflation, primarily through articles by Erin Einhorn and other reporters at the Daily News. These articles, which themselves relied on analyses from testing experts like Fred Smith, revealed that the test score inflation started as early as 2002, with questions and scoring on the state exams becoming easier over time.

See this 2007 article on our blog by Steve Koss, relating the ingenious experiment done by Einhorn in which she gave the 2002 and 2005 math tests to the same bunch of children, with the results showing that the 2005 exam was much simpler, a fact also reflected in the changing "P" values of the questions. Or this follow-up Einhorn article, where leading testing experts called for an independent audit, which of course did not occur until three years later.

Where was the NY Times amidst all these revelations? Absolutely nowhere. Even now, the Times article omits any mention of the Daily News’ earlier exposes – which brought attention to this issue to the wider public – and instead recounts as somehow meaningful that a few individuals who supposedly had doubts about the apparent rise in test scores, like Pedro Noguera and Kathleen Cashin, didn’t directly mention them to Klein– as though he might otherwise not have noticed the evidence that was splashed all over the Daily News!

The article also offers a rather irrelevant story, relating how Joel Klein earnestly tried to convince the state to change its scoring system to use a value-added method instead, as though that would have addressed any of the problems regarding the score inflation. In fact, that might have made things worse, as indicated by the way in which the inflation led to 84% of all NYC elementary and middle schools receiving "A" last year, based primarily on the value-added method, falling precipitously to only 25% this year, when the state decided to reset the cut scores.

The article also gives Regent Merryl Tisch a pass, letting her have the last word, saying “We came in here saying we have to stop lying to our kids,” without mentioning that throughout the test score inflation period, she was Deputy Chancellor of the Regents, and yet reliably supported Bloomberg and Klein's claims of great improvement.

The Times itself had plenty of reason to know about concerns about the state test score inflation throughout this period but not only failed to report on it, but generally toed the company line.

On August 4, 2009, at the very moment when Bloomberg was pressing for the extension of mayoral control of the schools, and two years following the Daily news exposes, the Times published a credulous story that recounted the steep increase in state test scores and the apparent narrowing of the achievement gap, including this quote from Joel Klein:

“Mr. Klein, for his part, said he was confident that rising scores reflected real improvements. “No matter how you look at them,” he said, “the picture is one that shows that the city is making dramatic progress.”

In the article , Klein put more emphasis on the apparent rise in proficiency levels rather than scores, “saying that the pass-rate was the more critical measure because it indicated proficiency, an important gateway to success…Our job is to get all kids to basic proficiency and then continue to move them forward, and I think we can do that.”

Now, of course, Klein emphasizes the higher scale scores on the state tests, since the reported proficiency levels have dropped so dramatically.

To the degree that the Aug. 2009 Times article expressed any doubts about NYC's dramatic rise in test scores, it was only the possibility that the drive towards high-stakes accountability had led to excessive test prep-- not any of the overwhelming evidence that the tests were simply and the scoring more lenient.

Indeed, yesterday's article focused on the same set of concerns -- excessive test prep, the public release of prior exams, and the limited number of questions on the exams -- rather than any of the more damning findings, showing how the state had rigged the results with easier questions as well as lower cut scores, both of which would directly implicate the state in the fraudulent outcome.

As I wrote in August 2009 to the Times editor, Ian Trontz:

“… there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. Not only are huge jumps in the scores occurring in nearly all districts and in all grades– the middle grade increases last year were especially unprecedented -- but as has been widely reported in the Daily News and elsewhere, the cut scores have been lowered each year. To leave this out of your story seems negligent at best, especially given the room and the time your reporter had to expound on this issue.”

I also pointed out that the article had wrongly attributed gains to Bloomberg/Klein on the NAEPs by giving them credit for the increase in test scores since 2002, even though their reforms had not begun until 2003. Here is how Trontz responded to this point:

"We do not, however, think that your email points out any inaccuracies. It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect."

I concluded at the time:

“Given the evident bias of this article, it appears clear that the Times has been captured by the Bloomberg/Klein PR machine, and can no longer be trusted to provide objective analysis of their education record."

Shortly afterward, Wayne Barrett wrote about the controversy in the Village Voice,

"The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true."

Two days after the NY Times article ran, on August 6, the NY State Senate voted to renew mayoral control without any checks and balances, essentially allowing Bloomberg to retain his stranglehold over our schools. One of the few Senators who voted against the renewal, Sen. Carl Kruger, argued that the achievement gains claimed by Bloomberg and Klein would soon be found to be fraudulent.

In response, the Daily News editorial page, as Bloomberg-sycophantic as all the other NYC major dailies, argued vociferously against Kruger's claims. In the piece, the News editors referenced the recent NY Times article (since they could not cite the far more explosive reporting of its own staff ):

“Good luck with that. He'll sooner find Sasquatch under Chancellor Joel Klein's desk. The New York Times double-checked test results and concluded this week that they showed "a steady march upward."

Indeed, Sasquatch was hiding in plain sight under Klein’s desk all along.

Just a few days after the Senate vote, on August 11, 2009, Meredith Kolodner of the Daily News pointed out that the cut scores had been set so low on the 2009 exams that a sixth grade student could pass the ELA just by randomly guessing, while 7^th graders had to get just one extra question right to pass. At Gotham Schools, Diana Senechal tried taking the exams herself and confirmed that in the 5^th grade ELA, 6^th grade math, and 7^th grade ELA exams, a student could indeed achieve a level 2 through random guessing .

What’s most scary is that according to the latest Times article, the state is now apparently going to keep the questions on the exams secret forever, supposedly to ensure that they can keep their difficulty level stable -- to guard against excessive test prep using old tests.

So in future, it will be impossible for another reporter like the intrepid Einhorn to test the same bunch of kids with exams from two separate years, to prove how much easier they have become. Or for any parent to examine how flawed the questions may be. (Remember Brownie the Cow, the absurd questions on the 2006 4^th grade ELA exam?) As though keeping the questions secret is the only method that can be used to keep standardized exams stable over time!

Meanwhile, companies like the College Board manage to release the SATs each year, and still are able to equate them, by holding back the ungraded questions (what the Times article calls “field test questions”). Why that could not occur in the case of the NY State exams is never explained by the article.

Sorry, NY Times, this article is too little and too late. Years before, when Bloomberg was pushing to retain unilateral control over our schools based upon these inflated test scores, the paper of record owed it to its readers to inform them of these issues, and yet utterly failed.

(See my critique at the time of their August 2009 article, NY Times falls in line with the Bloomberg PR spin control; and the response from Times editor, Ian Trontz: The NY Times response, and my reply.)

The NY Times response, and my reply

See the response from the NY Times editor below to my critique of their test score article yesterday – and my reply. Read from the bottom up. See also this front page story from the Times from November 16, 2007 – in which, in opposition to the paper’s current stance, it was clearly stated that there were no significant gains in any area but fourth grade math since the Bloomberg/Klein reforms took hold:

“The national scores also showed little narrowing of the achievement gap between white students and their black and Hispanic counterparts. The results for New York and 10 other large urban districts on the federal tests, the National Assessment of Educational Progress, paint a generally stagnant picture for the city, although there are gains in fourth-grade math.”

Why have they changed their story now? If you’d like to chime in, please do! You can ask for a correction at nytnews@nytimes.com; write a letter to the editor at letters@nytimes.com, or email the public editor at public@nytimes.com

Amazingly, the same testing consultant who was quoted in yesterday's NY Times as saying the state tests are “about as good as we can build them” is quoted in today's Gotham Schools that "“It’s hard to trust the data right now."

____ ____ _____ _____

Dear Ian:

Thanks for responding. The broad evidence that the state test scores are inflated is clearly relevant to the Times reporting here and was totally glossed over.

Not one independent expert was quoted as to this issue – only those who have a strong interest in claiming credit for improvements, including the head of SED’s test advisory committee, the head of the Regents and the Chancellor himself. This is not balanced reporting. In fact, the headline that refers to “critics” is rather strange, when not a single legitimate critic is cited.

As I’m sure Elissa is well aware, there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. Not only are huge jumps in the scores occurring in nearly all districts and in all grades– the middle grade increases last year were especially unprecedented -- but as has been widely reported in the Daily News and elsewhere, the cut scores have been lowered each year. To leave this out of your story seems negligent at best, especially given the room and the time your reporter had to expound on this issue.

Moreover, the lack of progress on the NAEPs in NYC provides a natural audit of the reported rise in state test scores.

As to whether or not the 2003 increases could be attributed to the leadership of Bloomberg and/or Klein, you are surely right that “It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect.”

But perhaps you might have mentioned the latter statement in your story!

Or, if you are implying instead that the increases in 2003 could be legitimately due to the Bloomberg/Klein leadership, as the administration is desperate to claim, the NY Times seems to have flipped positions. As was widely recognized at the time, by your own reporting staff, the 2003 gains were quite uncomfortable for Klein et. al. when they first occurred.

Let’s take a little trip back in time.As Diane Ravitch has recounted:

“The Legislature approved mayoral control legislation in June 2002, and Chancellor Klein assumed office in August 2002. He spent the next six months conferring with consultants. He and the mayor announced their plans and their pedagogical choices in January 2003. At that very time, students in the city and state were taking the state tests. The chancellor had done nothing in the schools prior to January 2003 to raise student achievement. His reforms were introduced into the schools in September 2003. Thus, it is inappropriate for Dr. Bell-Ellwanger [or any DOE official, as they have recently claimed] to take credit for any gains registered on the state tests that were administered in January 2003.”

This reality was well recognized at the time, by both the Klein administration and the NY Times reporting staff.

As David Herszenhorn wrote in the Times when the state ELA scores were first reported on May 21, 2003:

“The city's positive results come at a time when Mayor Michael R. Bloomberg and his schools chancellor, Joel I. Klein, are trying to overhaul the public school system and impose a uniform reading and math curriculum at all but the highest performing schools.

City officials, who might otherwise have been jubilant about yesterday's results, offered a muted reaction, saying that the gains were not broad enough and that the school system as a whole was still failing at least half the city's children.;;

''Although we are pointed in the right direction, there is still a lot of work that needs to get done,'' said a deputy schools chancellor, Diana Lam. ''Even with these gains, we have one out of two students that are not fluent readers at the fourth-grade level.''

Experts said the sharp increase in test scores could prove problematic for Mr. Bloomberg and Mr. Klein, since it is too early for them to take credit and sets a benchmark for next year that may be hard to match….

The higher scores, particularly at schools that have showed sustained increases since 1999, also gave new ammunition to critics of Mr. Bloomberg's changes, who said that they might do more harm than good by disrupting existing programs….

And he quoted some of these dissenters, which at the time included Norm Fruchter and Randi Weingarten, as saying that the scores instead resulted from the reforms introduced by the previous Chancellor, Rudy Crew.

Of course, Klein and company are now quite eager to take credit for these 2003 gains – which as regards the 4^th grade ELA NAEPs, show the only significant improvements since Bloomberg took office. But this revisionist history is hard to square with your paper’s own past reporting.

Let's see how Elissa Gootman herself reported the state math gains when they were released on October 22, 2003:

“Fourth graders across the state made stunning gains in their math scores last spring, with even sharper increases in New York City…In the city, news of the gains, which were particularly pronounced in the Bronx and in some of the poorest-performing districts, elicited cheers among teachers and principals. But not everyone greeted the news so enthusiastically:

The suggestion that city schools were on the upswing put Chancellor Joel I. Klein, who is overhauling them, in a tricky position. While the chancellor’s critics pounced upon the higher scores as evidence that the school system did not need such an overhaul, some of his allies acknowledged that he would now be under even more pressure to show gains next spring.

''While I am gratified by the test results released today for fourth and eighth graders in New York City, I must emphasize that it is hard to tell the true significance of any one set of results in isolation,'' the chancellor said in a statement. ''We must always look at results in comparison over a number of years. Only through comparison can we truly measure the progress we're making.''

That’s true of course. And there has been no significant improvement in the NAEPs in the years since – except in 4^th grade math [in which there was a 25% exclusion rate].

Since the NAEPs are widely seen as the gold standard “audit” test, admitting that you didn’t see the need to report in any depth on NYC’s lack of progress on the NAEPs is like saying you can report a story on the DOE’s finances without seeing the need to mention any of the numerous audits from the State or City Comptroller’s office that reveal widespread waste on the part of DOE, including misappropriation of state class size funds, huge growth in private contracts with 20% going over budget, and repeated awarding of no-bid contracts, without registering or documenting their rationale.

But that’s another story…..perhaps you might look into the no-bid contracts for Agudath of Israel, with the amounts awarded the organization by DOE dwarfing what the Times’ political reporters wrote about today in relation to the Mayor’s use of his discretionary funds.

Thanks, Leonie Haimson

---

From: Ian Trontz [mailto:trontz@nytimes.com]
Sent: Monday, August 03, 2009 6:20 PM
To: leonie@worldnet.att.net
Subject: test score article

Ms. Haimson:

We appreciate the attention you've given to this story, as much as we've appreciated your being a source for us over a number of years.

We do not, however, think that your email points out any inaccuracies. It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect.

Indeed, there are many interesting and important ways to look at the NAEP results, but the NAEP comparison was one small part of this report; we could not devote unlimited space to it given the parameters of this article, which is about the city’s performance on state tests. We therefore included a very broad, basic summary of the NAEP results.

We also, unfortunately, do not have the most recent NAEP results, but you can be assured that we'll report them in full when we get them.

As for whether the tests themselves are deficient, I think the article discusses that, perhaps not with the depth you are seeking, but as with the NAEP, we cannot go down every road.

Ian Trontz, Assistant Metro Editor/Education