The NY Times issues a correction, too little and too late.

An article in the NY Times magazine two weeks ago written by Jonathan Mahler on April 10 , about Middle School 223 in the Bronx, contained the following passage describing student achievement gains under Joel Klein:

“Since 2006, the city's elementary and middle schools have seen a 22-point increase in the percentage of students at or above grade level in math (to 54 percent) and a 6-point increase in English (to 42 percent).”

Upon reading this statement, I immediately knew it to be untrue. There has been little or no improvement in student achievement in NYC since 2006 – or even since 2003, when the Klein first implemented his policies, according to the most reliable national assessments called the NAEPs. 

In fact, after the NY State Education Department recalibrated the exams, increasing the scale scores needed for proficiency this summer in response to overwhelming evidence that the state tests and their scoring had gotten much easier over this period, the percent of NYC students at or above grade level actually dropped precipitously compared to 2006. 

See these charts, for example, from the DOE’s own website, showing the sharp fall in the percent of students at proficiency in English Language Arts, as calculated by the state, following the apparent (but highly unreliable) increase that had occurred between 2007 and 2010:

And below in math:

I wrote to the reporter, Jonathan Mahler, to ask where he got his data.  Here is his reply:  “The DOE was my source, and the fact-checker went back over the numbers with them.”

That didn’t exactly satisfy me, so I sent in a request for a correction via email to the editors, reproducing the charts above, as well as their source.  

A few days, I received a phone message from Aaron Retica, head of the research department at the NYT magazine, saying “we aren’t wrong” but that he understood why I might have been “confused.” 

 I followed up with phone call, and Retica explained that they were doing an “apples to apples comparison” by taking the DOE's re-estimate as a more reliable comparison, assuming the scale scores needed for proficiency had not changed. In other words, they bought the DOE spin, hook line and sinker, as portrayed on this slide, from the same power point deck:

This is a slide that Shael Suransky presented throughout the city last fall, including during my debate with him at NY Law school, in an attempt to convince his audience that NYC achievement levels had actually increased over the last four years.   

Though this slide may have confused many, it convinced very few; that is, aside from (apparently) NY Times editors.  In small words at the bottom, the slide says, ”Starting in 2010, NYSED changed the scale scores required to meet each of the proficiency levels, increasing the number of questions students needed to answer correctly to meet proficiency.”  

The slide, in claiming that the proficiency of NYC students in ELA in 2010 had risen to 68.2% rather than 42.4%; and in math; to 83.3% rather than 54%, disputes the judgment of the State, and ignores the fact that the number of questions needed to achieve proficiency was increased, not out of some arbitrary motive to punish NYC or devalue the accomplishments of its students, but because the state -- as well as all independent experts -- realized that the questions had themselves gotten easier over time. 

Indeed, this recalculation reeks of an attempt to rig the results, to back up improbable claims of great gains when the NAEPs have shown negligible increases. (Of course the meaningless nature of this exercise should be obvious in that there are two very different versions of achievement gains displayed in the above slide, depending on whether 2009 or 2010 cut scores are used.)

Like the DOE, the NYT is hugely averse to correcting errors, even of the most egregious kind –as in a Steve Brill article published in the magazine section last year on Harlem Success Academy, some of whose errors I catalogued here. 

The larger problem is that for many years, the NY Times ignored and/or omitted any reporting of NY’s test score inflation, delaying until after the bubble had burst, and long after the scandal had been covered extensively by the Daily News,  Gotham Schools, and even the NY Post .

Instead, the Times editors seem chronically disposed to give credit to Bloomberg and Klein where none was due; as anyone who was paying serious attention to this issue, and to the relatively flat results of NYC on the national assessments called the NAEPs, would be aware.

See for example, this 2007 post on our blog by Steve Koss, relating an ingenious experiment carried out by Erin Einhorn of the Daily News, when she gave the 2002 and 2005 math tests to the same bunch of children, revealing how they got better results on the 2005 exam. (For some reason the original Einhorn article is no longer available online.) Or this follow-up News article, in which leading test experts called for an independent audit of the state exams, which did not occur until three years later.

Where was the NY Times amidst all these revelations? Absolutely nowhere.   

To the contrary, on August 4, 2009, at the very moment when Bloomberg was pressing for the extension of mayoral control of the schools, and two years following extensive and continuing coverage of test score inflation in the News and elsewhere, the Times published a credulous account that recounted the steep increase in state test scores and the apparent narrowing of the achievement gap, featuring this quote from Joel Klein:

“Mr. Klein, for his part, said he was confident that rising scores reflected real improvements. “No matter how you look at them,” he said, “the picture is one that shows that the city is making dramatic progress.”

As I wrote in August 2009 to the Times education editor, Ian Trontz:

“… there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. … as has been widely reported in the Daily News and elsewhere … To leave this out of your story seems negligent at best…

(See my critique at the time of their August 2009 article, NY Times falls in line with the Bloomberg PR spin control; and the response from Trontz: The NY Times response, and my reply.)

Shortly afterward, Wayne Barrett wrote about the controversy in the Village Voice,

"The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true."

Two days after the Times article ran, the NY State Senate voted to renew mayoral control witout any checks and balances, essentially allowing Bloomberg to retain his stranglehold over the schools.  The "paper of record" could not have done a better job at burying the story that DOE's gains were illusory than if they had actually tried. 

After State Ed officially burst the test score bubble last summer, the Times finally covered the issue in a front page story in October, entitled "On NY School Tests, Warning Signs Ignored."  But here, too, the paper left its own deficient reporting off the hook, and refrained from mentioning any of the abundant exposes that had appeared over the last three years, not only in the Daily News, but in GothamSchools, the NY Post, and our blog; any of which should have alerted interested observers to the reality.   (Here is my account of the failure of the NYT to cover this story in their revisionist history.)

So what happened with this example of the NYT editors credulously swallowing the latest DOE’s test score spin? 

The editors did run a correction, but phrased in a very confusing manner.  Unless a reader received their information elsewhere, it would be very hard for them to interpret its meaning.  Here is the correction they printed today, with the specific passage in bold:

An article on April 10 about Middle School 223 in the Bronx misstated the reasons the school does not qualify for some state financing earmarked for poorly performing schools. It does not meet some of New York State’s criteria for failing schools and it is relatively successful on state tests and other measures. It is not because of the school’s report card from New York City. The article also omitted an attribution for the increases in percentages of students at or above grade levels in math and English from 2006 to 2010. Those figures came from the New York City Department of Education, which did its own analysis of state testing data using 2010 proficiency levels for 2006 test scores. (Without that adjustment, the percentage of proficient learners in both math and English actually dropped from 2006 to 2010.) (emphasis added)

Even worse, here is how they amended the offending passage in the article itself:

Since 2006, according to an analysis of state testing data by the city's Department of Education (which used 2010's recalibrated proficiency levels to compare 2006's testing data to 2010's), the city’s elementary and middle schools have seen a 22-point increase in the percentage of students at or above grade level in math (to 54 percent) and a 6-point increase in English (to 42 percent).

So in the article and the correction, they failed to explain the larger context and why the state had recalibrated the proficiency levels – to correct for the fact that the exams had gotten easier over time. 

Instead, their explanation of this matter could lead an unsuspecting reader to believe that the DOE’s “adjustment” was entirely appropriate, and provided reasonable evidence of improvement – while omitting any mention of more reliable evidence from the NAEPs of largely stagnant results.

On the same day his article was published in the magazine section, Jonathan Mahler wrote an article in the Week in the Review section of the Times, entitled “Deadlocked Debate over Education Reform.”  In it he wrote:

“The data can appear as divided as the rhetoric. New York City’s Department of Education will provide you with irrefutable statistics that school reform is working; opponents of reform will provide you with equally irrefutable statistics that it’s not. It can seem equally impossible to disentangle the overlapping factors…”

In the future, I would hope that the editors of the Times might try a little harder to look at the data with a more discerning eye, and not swallow the distortions of the Department of Education. 

Perhaps they might even refer to independent experts who could dissect the data if they find it too difficult to interpret it themselves.  They owe it to their readers.  There is little point in trying to cover the NYC public schools if they continue to be incapable of weighing the evidence objectively and presenting the facts with a more practiced eye, rather than simply regurgitating what is handed them by the spinmeisters at Tweed. 

After all, if the Daily News and the NY Post can do it, why not the Times?

Another Reality Check on NYC's State Exam Scores

The privately-backed Fund for Public Schools has once again entered the public relations arena on behalf of Mayor Bloomberg and Chancellor Klein with television ads touting the “remarkable” gains made by NYC public school students in this year’s NYS Math and ELA exams. In so doing, they have joined every imaginable constituency from the UFT to the State Education Department to the editorial boards of major NYC newspapers in extracting as much credit as is humanly possible. After all, nobody’s personal or political agenda is harmed by rising test scores, so why not bathe in their afterglow, real or otherwise?

A few reports, such as Elizabeth Green’s story in the New York Sun or Jennifer Medina's in the New York Times have cast clouds of doubt over this year’s “too good to be true” results. It’s interesting, however, to step back for a moment, take a longer look at the data, and insert a little historical perspective (credit here goes in part to Sol Stern’s City Journal story, “Grading Mayoral Control” and Diane Ravitch's earlier posting in this blog from 2007).

To begin with, the DOE’s formal presentation of this year’s results makes the following self-congratulatory assertion:

Since 2002, the percentage of students meeting or exceeding State standards is up 27.7 points in 4th grade math and up 29.8 points in 8th grade math. In ELA, the percentage is up 14.8 points in 4th grade and 13.5 points in 8th grade.

Governor Pataki signed the legislations authorizing mayoral control of the NYC public schools on June 12, 2002. Joel Klein was appointed Chancellor in August, 2002 and spent his first year learning the system, creating task forces, and the like. As Sol Stern wrote, “Klein knew he couldn’t convincingly claim credit for the 2003 test scores, and he didn’t even hold a press conference to celebrate them.” Clearly, the earliest reasonable base year the Mayor and Chancellor can use for measuring the impact of their efforts would be 2003. Even starting with 2003 as the pre-Mayoral control base year assumes that Mr. Klein moved so quickly and forcefully in one school year as to achieve results in 2004 that could be attributed to the changes he had effectuated, but let’s be charitable and concede that possibility. So what do we see in looking at this year’s Summary Report numbers since 2003?

In 4th Grade Math, the five-year percentage point gain of students at Levels 3+4 shrinks from 27.7 points to 13.0, or by more than half. Furthermore, an inexplicable jump of 9.3 points in 2005 has never been duplicated; NYC 4th graders’ proficiency has only increased 2.3 percentage points in the last three years despite teacher bonuses, principal incentives, accountability threats, cell phone minutes and cash for students, and endless test preparation. In 4th grade English, we see the identical story. The DOE’s claimed 14.8 percentage point increase shrinks to just 8.9 points, and another inexplicable 9.9 percentage point increase in 2005 leaves today’s 4th graders only 1.8 points better off in terms of proficiency that they were three years ago.

In 8th Grade Math, the story is a bit different, but suspiciously so. The DOE’s claimed increase of 29.8 points in percent of students at Levels 3+4 shrinks slightly, to 25.2 points. However, after an 8.0-point jump to 42.4% in 2004, those scores had declined in 2005 and again in 2006 before reaching 45.6% in 2007, a three-year gain of 3.2 percentage points. After essentially flat-lining over four years (2004-2007), this year’s scores rose by an astonishing 14.0 percentage points, a logic-defying increase that certainly calls for further examination. In 8th Grade ELA, the increase since 2003 has been 10.5 percentage points (compared to the DOE’s claim of 13.5 points), and a respectable (and believable) 7.4 points since 2004.

To the extent that some of the test result increases are undoubtedly “real” (despite stories about cheating and suspicions that the tests are being "dumbed down"), consider the following viewer comment posted anonymously on NY1’s The Call blog (thank goodness there are still a few honest souls out there):

I am a fourth grade teacher in the Bronx, and my class is comprised entirely of Second Language Learners. Starting in November, we had to basically drop everything and teach to the test. I had to sit through meetings where my administrators would talk about "strategies" to help them succeed, like looking for key words. During this time, I was never teaching for learning, I was teaching to just scrape by on this test. We are told to stop teaching Science and Social Studies so we can do test prep. The students go until 5pm to the "Test Prep Academy" and in the two weeks before the test, we have to do ALL DAY TEST PREP. While the Language Arts test is challenging for second language learners, the math test is so far below what fourth graders should know it's just embarrassing! There is no long division, no double digit multiplication, no real problem solving that requires actual critical math application skills. I don't get to teach content, and I will leave for the suburbs as soon as I finish my Master's Degree.

Is this the price we have agreed and accepted to pay in our children’s education in order for Messrs. (and Ms.) Bloomberg, Klein, Weingarten, and others to wallow in their accolades? A devil’s bargain if ever there was one. SHAME ON US. ALL OF US.

ELA and Math Test Scores -- Experts Question Increases

State test scores for 2008 can be found here (Math) and here (ELA)

An expert who serves as the State's top technical adviser on testing is concerned about grade inflation and has called for an independent study of this year's unusual and unprecedented rise in test scores across the state. Writing in the NY Sun, Elizabeth Green has the story here.

Elizabeth also has a disturbing story about cheating on the state tests here. An excerpt:

A sixth-grader at M.S. 201. said that a teacher once looked over his shoulder and said, "Ooh, is that right? Is that the right answer?" encouraging him to erase and try again.

Meanwhile, 11 of 12 P.S. 48 graduates interviewed last week said they were coached during the state tests.

They said that teachers would look over their shoulders and instruct them to try again and again until they got answers right.

"They'd be like, 'Is that the right answer?' — until they make sure it's right," a sixth-grader said.

"When I was at 48, I never went to class, and I still passed the test," a seventh-grader said. "If you go to graduation, you pass."

Higher test scores could pay off for M.S. 201's teachers this year. The school is one of about 200 participating in a trial project to give teachers bonuses if their students perform well on state tests.

The bonuses average $3,000 a teacher.

Under the Bloomberg administration, test results have been woven into a complex system of carrots and sticks where principal bonuses, teacher merit pay, school ratings, school budget bonuses, principal dismissals and school closings all hinge on test scores. It is not surprising that pressure to score high has lead to a culture of test prep, grade inflation and cheating.

Update: see the NY Sun for a properly skeptical oped about the sharp rise in NY State test scores.