Too little and much too late, the Times finally reports on the state test score scandal

In yesterday’s front page story, entitled "On NY School Tests, Warning Signs Ignored," the NY Times' account of the state test score scandal left its own deficient reporting conveniently off the hook.

Anyone who was paying attention knew at least as far back as 2007 that there was rampant test score inflation, primarily through articles by Erin Einhorn and other reporters at the Daily News. These articles, which themselves relied on analyses from testing experts like Fred Smith, revealed that the test score inflation started as early as 2002, with questions and scoring on the state exams becoming easier over time.

See this 2007 article on our blog by Steve Koss, relating the ingenious experiment done by Einhorn in which she gave the 2002 and 2005 math tests to the same bunch of children, with the results showing that the 2005 exam was much simpler, a fact also reflected in the changing "P" values of the questions. Or this follow-up Einhorn article, where leading testing experts called for an independent audit, which of course did not occur until three years later.

Where was the NY Times amidst all these revelations? Absolutely nowhere. Even now, the Times article omits any mention of the Daily News’ earlier exposes – which brought attention to this issue to the wider public – and instead recounts as somehow meaningful that a few individuals who supposedly had doubts about the apparent rise in test scores, like Pedro Noguera and Kathleen Cashin, didn’t directly mention them to Klein– as though he might otherwise not have noticed the evidence that was splashed all over the Daily News!

The article also offers a rather irrelevant story, relating how Joel Klein earnestly tried to convince the state to change its scoring system to use a value-added method instead, as though that would have addressed any of the problems regarding the score inflation. In fact, that might have made things worse, as indicated by the way in which the inflation led to 84% of all NYC elementary and middle schools receiving "A" last year, based primarily on the value-added method, falling precipitously to only 25% this year, when the state decided to reset the cut scores.

The article also gives Regent Merryl Tisch a pass, letting her have the last word, saying “We came in here saying we have to stop lying to our kids,” without mentioning that throughout the test score inflation period, she was Deputy Chancellor of the Regents, and yet reliably supported Bloomberg and Klein's claims of great improvement.

The Times itself had plenty of reason to know about concerns about the state test score inflation throughout this period but not only failed to report on it, but generally toed the company line.

On August 4, 2009, at the very moment when Bloomberg was pressing for the extension of mayoral control of the schools, and two years following the Daily news exposes, the Times published a credulous story that recounted the steep increase in state test scores and the apparent narrowing of the achievement gap, including this quote from Joel Klein:

“Mr. Klein, for his part, said he was confident that rising scores reflected real improvements. “No matter how you look at them,” he said, “the picture is one that shows that the city is making dramatic progress.”

In the article , Klein put more emphasis on the apparent rise in proficiency levels rather than scores, “saying that the pass-rate was the more critical measure because it indicated proficiency, an important gateway to success…Our job is to get all kids to basic proficiency and then continue to move them forward, and I think we can do that.”

Now, of course, Klein emphasizes the higher scale scores on the state tests, since the reported proficiency levels have dropped so dramatically.

To the degree that the Aug. 2009 Times article expressed any doubts about NYC's dramatic rise in test scores, it was only the possibility that the drive towards high-stakes accountability had led to excessive test prep-- not any of the overwhelming evidence that the tests were simply and the scoring more lenient.

Indeed, yesterday's article focused on the same set of concerns -- excessive test prep, the public release of prior exams, and the limited number of questions on the exams -- rather than any of the more damning findings, showing how the state had rigged the results with easier questions as well as lower cut scores, both of which would directly implicate the state in the fraudulent outcome.

As I wrote in August 2009 to the Times editor, Ian Trontz:

“… there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. Not only are huge jumps in the scores occurring in nearly all districts and in all grades– the middle grade increases last year were especially unprecedented -- but as has been widely reported in the Daily News and elsewhere, the cut scores have been lowered each year. To leave this out of your story seems negligent at best, especially given the room and the time your reporter had to expound on this issue.”

I also pointed out that the article had wrongly attributed gains to Bloomberg/Klein on the NAEPs by giving them credit for the increase in test scores since 2002, even though their reforms had not begun until 2003. Here is how Trontz responded to this point:

"We do not, however, think that your email points out any inaccuracies. It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect."

I concluded at the time:

“Given the evident bias of this article, it appears clear that the Times has been captured by the Bloomberg/Klein PR machine, and can no longer be trusted to provide objective analysis of their education record."

Shortly afterward, Wayne Barrett wrote about the controversy in the Village Voice,

"The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true."

Two days after the NY Times article ran, on August 6, the NY State Senate voted to renew mayoral control without any checks and balances, essentially allowing Bloomberg to retain his stranglehold over our schools. One of the few Senators who voted against the renewal, Sen. Carl Kruger, argued that the achievement gains claimed by Bloomberg and Klein would soon be found to be fraudulent.

In response, the Daily News editorial page, as Bloomberg-sycophantic as all the other NYC major dailies, argued vociferously against Kruger's claims. In the piece, the News editors referenced the recent NY Times article (since they could not cite the far more explosive reporting of its own staff ):

“Good luck with that. He'll sooner find Sasquatch under Chancellor Joel Klein's desk. The New York Times double-checked test results and concluded this week that they showed "a steady march upward."

Indeed, Sasquatch was hiding in plain sight under Klein’s desk all along.

Just a few days after the Senate vote, on August 11, 2009, Meredith Kolodner of the Daily News pointed out that the cut scores had been set so low on the 2009 exams that a sixth grade student could pass the ELA just by randomly guessing, while 7^th graders had to get just one extra question right to pass. At Gotham Schools, Diana Senechal tried taking the exams herself and confirmed that in the 5^th grade ELA, 6^th grade math, and 7^th grade ELA exams, a student could indeed achieve a level 2 through random guessing .

What’s most scary is that according to the latest Times article, the state is now apparently going to keep the questions on the exams secret forever, supposedly to ensure that they can keep their difficulty level stable -- to guard against excessive test prep using old tests.

So in future, it will be impossible for another reporter like the intrepid Einhorn to test the same bunch of kids with exams from two separate years, to prove how much easier they have become. Or for any parent to examine how flawed the questions may be. (Remember Brownie the Cow, the absurd questions on the 2006 4^th grade ELA exam?) As though keeping the questions secret is the only method that can be used to keep standardized exams stable over time!

Meanwhile, companies like the College Board manage to release the SATs each year, and still are able to equate them, by holding back the ungraded questions (what the Times article calls “field test questions”). Why that could not occur in the case of the NY State exams is never explained by the article.

Sorry, NY Times, this article is too little and too late. Years before, when Bloomberg was pushing to retain unilateral control over our schools based upon these inflated test scores, the paper of record owed it to its readers to inform them of these issues, and yet utterly failed.

(See my critique at the time of their August 2009 article, NY Times falls in line with the Bloomberg PR spin control; and the response from Times editor, Ian Trontz: The NY Times response, and my reply.)

Who is the real dictator?

Earlier this week, Erin Einhorn, the fearless City Hall reporter for the Daily News, tried to ask Mayor Bloomberg about whether he should have allowed the public to vote on extending term limits, as did Hugo Chavez, the president of Venezuela. As you can see, the Mayor brushed her aside:

Q: Mayor, it’s hard to compare New York City to Venezuela but as you know, Hugo Chavez did his second effort - this time sucessful - to extend term limits. You chose to go through City Council. Do you have any second thoughts about this? Do you wish you should have had a chance to take to the...

A: I don’t understand your question. What on Earth do we have to do with Hugo Chavez?

Q: Well, like you, he wanted to extend his term.

A: if you wanted to ask Hugo Chavez, call him up! Maybe he’ll take your call. My suspicion is he doesn’t have press conferences and let people ask questions or if they ask questions, he probably throws them, I don’t know he does with them...Who knows? (Laughs). I still fail to see a connection.

While saying that Erin should win an award for her question, the Economist magazine also pointed out that Chavez holds regularly televised press conferences in which ordinary citizens are allowed to query him; one can hardly imagine our petulant Mayor allowing that.

For more on the contrast between Chavez or Bloomberg, and who acted as though he believed in democracy more, see Clyde Haberman's column in today's Times:

The people voted a while back to impose term limits on their political leadership. That was a nuisance for the man in charge. He was in his second term and prohibited by law from running again. But he sorely wanted to try for a third term.

Here’s what he did. He went back to the people to ask if they’d had a change of heart.

Obviously, he was not from New York.

Seven hundred teachers in the rubber room

See the latest expose from Erin Einhorn of the Daily News – the number of teachers in the “rubber room” have doubled in the past four years to 700 – and are now costing the system at least $65 million a year. (photo credit: Daily News.)

Read the profiles of three teachers in this position, and see for yourself whether they should have been consigned to this limbo.

Many spend years without being charged with any specific misconduct. Indeed, the “average accused educator waits four months as investigators interview witnesses and decide whether to bring formal charges, then nine months for a hearing and six more for a decision.”

And: “Only 20 arbitrators have been jointly approved by the city and union to hear these cases, and they work just five days a month during the school year and two during summer vacation. “

Between the 700 teachers sitting idly in the rubber room, and the 800 teachers in ATR, the Joel Klein and his cadre of management geniuses are spending at least $150 million to keep 1500 full-time educators idle. What a waste of money and manpower – particularly when so many schools continue to have class sizes of 30 or more.

Gifted and Talented Admissions -- Confusion and Delay?

UPDATE (April 9th): As reported, the PEP will hold an emergency vote on April 10th to lower the cutoff to 90% for district G&T programs. Notification letters to be mailed "sometime this month".

In today's Daily News, Erin Einhorn reports DOE is rethinking where to set the cutoff for G&T admissions. The policy to set the bar at 95% cutoff on the two standardized tests , the OLSAT and BSRA was set in November.

At the time, many parents tried to convince the DOE that they were making a mistake. The District 3 CEC passed a resolution in opposition to the revised policy. In our letter to Manhattan parents explaining our vote against the G&T policy when presented to the Panel for Educational Policy, Manhattan Borough President Stringer and I raised the concern: "this higher cutoff score for admissions raises the very real possibility that a number of current programs will not be filled to capacity and will therefore be closed, leaving parents with fewer programming options".

Let's hope the DOE refrains from shuttering successful programs. Meanwhile, notifications have not been sent out, putting the whole process behind schedule, which was already pushed too far out into the spring.

Parents, No Seat at This Table

Hard on the heels of Erin Einhorn’s stories this week in the Daily News questioning whether NYC public school students’ progress from 2002 to 2007 in the NY State math exams for 4th Graders is a result of easier test questions comes an announcement today from Chancellor Klein’s office. A new organization called the Research Partnership for New York City Schools has been formed with a charter to review data provided by the DOE and verify operational and student performance results through independent analysis. The new group will be privately funded by ten foundations and has pledged to maintain independence in its analysis and reporting.

Not surprisingly, the Research Partnership’s oversight board consists almost exclusively of people with close ties to the DOE and vested interests in the research and analysis outcomes. Chancellor Klein’s and UFT President Randi Weingarten’s presence on the board alone creates conflicts of interest, and these are furthered by the presence of Robert Hughes (New Visions for Public Schools) and Kathryn Wylde (Partnership for NYC, and already a Board Member of the DOE’s private sector-funded Leadership Academy and the conservative think tank The Manhattan Institute).

The first and most basic rule of auditing is arms length independence from the organization being audited. Where is the independence here? Even more disconcerting, where are the real independent voices? Where are Betsy Gotbaum, Diane Ravitch, Alfred Posamentier, Sol Stern, or a host of others from the NYU, Columbia, or Bank Street Schools of Education? And where are the parents representatives: CEC Presidents such as Tim Johnson or David Bloomfield, parent advocates like Leonie Haimson, or just interested and capable parent representatives? Reflecting the Chancellor’s continued policy of ignoring and dismissing NYC public school parents, he could not even bring himself to include Martine Guerrier, the DOE’s so-called new “Chief Family Engagement Officer.”

As Sol Stern points out in today’s Daily News article, “The whole thing is a fraud…it will have the illusion of objectivity.” Likely he’s correct. It seems the head fox has just recruited another set of foxes to verify independently their positive progress in the hen house. And once again, the message to parents is clear: “You have no seat at this table.”

“Here are the numbers. Everything is improving. Trust us.”

“Tell me the result you want, and I’ll find a way to make the numbers show it."

Anyone with a good feel for numbers and a background in math or statistics will recognize the ironic truth underlying this statement about mathematical deception. Nowhere in recent years has this approach seemed more evident than in the NYC Department of Education, most recently in the NY Daily News story about the decreasing level of difficulty in the NY State 4^th Grade math exams.

As Erin Einhorn reported in yesterday's Daily News , NYC pass rates on the 4^th Grade exams in math since 2002 have tracked very closely to the degree of difficulty of the exam itself. As the tests have gotten easier, the NYC (and NYS) pass rates have increased almost in perfect lockstep. This determination was made by obtaining information from the State Education Department, but the Daily News supplemented it with their own small-scale lab experiment with 34 youngsters who were participating in a Brooklyn College summer program. The kids took the 2002 and 2005 NYS math exams and, for the most part, concurred that the 2005 test was easier. In fact, 24 of the 34 young students did better on the 2005 exam.

Even for those well-versed in mathematical statistics, psychometrics (the science of testing and intelligence measurement) is an arcane subject. Suffice to say, all standardized tests are created from large banks of potential questions, each of which is field tested to determine its degree of difficulty, called a p-value. A question with a p-value of 0.65 means that 65% of students tested will likely get that question right, based on the field test results. A test can therefore be “rated” with a p-value according to the values of the questions chosen to make up that test.

Since 2002, the p-value of the state 4^th Grade math exam has risen from 0.61 to 0.74 in 2007. Remarkably, the number of questions needed to get a passing score has actually decreased at the same time, from 40 in 2002 to 39 in 2007. Not surprisingly, the NYC pass rate has increased from 52% to 74.1%, with the largest year-to-year increase occurring in 2003 when the p-value had its biggest increase (from 0.61 to 0.68). Defenders of the tests argue that the pass rate for an easier test can be adjusted by requiring more correct answers to pass, but how many of us (adult or child) would prefer an easy test requiring 40 correct answers to a more representative test requiring 39 or 38 correct answers? I certainly would.

In The Truth about Testing, author W. James Popham points out that questions with p-values between 0.4 and 0.6 are best suited to creating the degree of score spread necessary for a standardized test questions to be reliable differentiators of student performance (higher score spreads are interpreted as indicators of higher test reliability in measuring whatever it’s supposed to measure). It seems odd that the State can be creating tests with a p-value of 0.74, suggesting that 74% of students should be able to pass (and, in fact, exactly 74.1% did just that last year in NYC). Yet the Mayor and Chancellor have been solely crediting their own new policies for increases that would be expected just by the increasingly easier nature of the exam itself.

Mayor Bloomberg and Chancellor Klein have aggressively adopted the business model mantra that whatever isn’t measured can’t be managed (and incentivized) to the world of public school education. This policy has resulted in more and more standardized testing of students and increased collection and manipulation of school-level data on every aspect of school operations. Success or failure of Mayoral control, we are led to believe by the Chancellor, is entirely contingent on these measures: test scores, attendance levels, graduation rates, school report cards, and the like.

Bad enough that this is the factory model par excellence applied to education – you can’t measure a student’s found passion for theater or chorus, a discovered pleasure in reading, the classroom excitement over a new insight into the process of history, the joy of seeing one’s poem or short story in print, or the deep self-satisfaction that comes from further, teacher-led independent exploration of a science topic. Worse, it is solely the DOE that collects, cleans, massages, selects, and controls this information behind closed computer room and office doors. Since no external oversight, audit agency, independent review or verification exists, the message is clear: “Here are the numbers. Everything is improving. Trust us.”

Disturbingly, it’s the fox who is not only running the hen house but also reporting the excellence of the results to the farmer.

-- Steve Koss, PTA President of Manhattan Center for Science and Math

Update: see today's Daily News, where leading experts call for an independent audit agency as exists in other states, to verify the validity of these exams.

Speaking truth to power; Diane Ravitch and Erin Einhorn

As an antidote to all the PR feel-good nonsense put out by DOE, listen to the wise words of Diane Ravitch on WNYC radio today.

Diane said our kids are being turned into “test-taking puppets….At the end of the day, you really have to ask is that what education is all about?”

And: “We’re not getting honest answers from Albany…we really need an independent audit agency to look at these test scores.”

Lots more great points from Diane -- about how the public's voice has been completely shut out of this administration.

Erin spoke about her blockbuster scoop in today's Daily News, showing that the dramatic rise in the 4th grade math scores in 2005 -- just in time for the Mayoral elections -- was a direct result of a much easier test.