Why the school progress reports and NYC education reporters deserve a big fat “F”

Gary Rubinstein, a math teacher at Stuyvesant, totally outclasses the numerous education reporters in this city in his analysis of the recent school grades.  

He shows via this graph to the right and discusses in his blog how there is little or no correlation in rank order for schools between last year or this year's progress reports. 

(Incidentally, in a subsequent posting, Rubinstein also points out that NYC charters are twice as likely to get "Fs" in progress than non-charters.)

Unfortunately the mainstream media continue to repeat without dispute Suransky’s claim that the progress reports were much more “stable” this year, even though 60% of schools changed grades.    

Not one reporter, to my knowledge anyway, has bothered to point out how experts have shown that 32-80% of the annual gains or losses in scores at the school level are essentially random – and yet 60% of the school grade is based upon these annual gains or losses. 

See the Daily News oped I wrote in 2007, in which I offer even more criticisms of the progress reports, including their inherent instability, “Why parents and teachers should reject the new grades”.  

 Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

Yet here is one typical headline from last week: School report cards stabilize after years of unpredictability. Here is the NY Times account:

“We have a really high level of stability this year, which is a good thing,” said Shael Polakow-Suransky, chief academic officer for the city’s Department of Education…. There is movement and that’s good because we are measuring one year of data and we expect schools will go up and down, but we don’t want to see movement caused by something that’s external to the kids,” Mr. Polakow-Suransky said, referring to changes in the state exams that caused incredible increases and then a drop-off in schools’ grades.

Of course, if one year’s movement up and down is primarily random, that by definition is “external.” 

Nor have any education reporters bothered to report that Jim Liebman, who designed the system, testified to the City Council when the grades were introduced that the DOE would improve the reliability of the system to incorporate three years worth of test score data instead– which both he and Suransky have refused to do.

Indeed, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read,  Liebman is quoted as responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results this way:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

And yet no one, including Fertig, has mentioned this discrepancy and DOE’s lack of rationale for intentionally allowing an essentially single unreliable grade to determine a school’s future – with 10% of schools now guaranteed to be given a failing grade and thus liable to being closed, based largely on chance. 

Read my earlier 2010 column, Why the school grading system, and Joel Klein, still deserve a big "F".

The absurd, silly, ridiculous school grades

See articles on this year’s elementary and middle school grades, or “progress reports”: Times, NY Post, Daily News, NY1, and this piece in Gotham schools (the best).

They are replete with examples of the absurd results, with schools such as PS 82 in the Bronx jumping from an “A” to an “F” and back this year to an “A”. PS 8 in Brooklyn which got an “F” last year and this year got an “A”, with the principal saying they did nothing different. The three schools that Klein tried to close last year to put charter schools in their place-- P.S. 194, 241 and 150 – all got “As”.

In fact, 84% of the schools got an “A” this year, and only two schools out of more than one thousand got “F”s. This is grade inflation that would put any human being other than Joel Klein to hide his head in shame.

And yet, according to the NY Times, “he clearly took pride in the results. “If you’re asking whether I would rather see less A’s,” he said, “the answer is no.”

Meanwhile, 87% of principals said in a recent survey that their schools were unable to provide a quality education because of excessive class sizes.

The absurdity of the grades this year derive from two profound flaws: First, 85% of the grade is based on one year’s gains or losses in test scores, which experts have found to be statistically unreliable and extremely erratic.

And two, the state tests have become so much easier and their scoring so lax that students can pass them without reading the questions – as long as they manage to fill in a few bubbles along the way. (For more on this scandal, well-reported everywhere except the Times, see the Daily News here and here, Gotham Schools and this NY Post column by Diane Ravitch.)

I wrote an oped for the Daily news about the new grading system when it was first announced in the fall of 2007: “Why parents & teachers should reject new school grades.” It starts out this way: “The new school grading system unveiled this week by Chancellor Joel Klein is a fiasco.”

But what do you expect when you had two guys in charge, Liebman and Klein, who are ignorant as to statistics, the unreliability of test scores, and even the larger goals of education?

Perhaps the best result is where we have now arrived: hopefully everyone realizes that the emperor has no clothes and they should ignore these silly grades, as they should have in the first place.

A Failing Grade for Mr. Liebman

Several articles appeared today about James Liebman's resignation after serving three years as head of the Tweed's Office of Accountability -- finally returning to Columbia University law school full time: Chief Accountability Officer for City Schools Resigns (NY Times); and New accountability chief says he’ll carry on Liebman’s legacy (Gotham Schools).

Let us remember that this man had no qualifications for the job, and proved this repeatedly over the years. In fact the only person who probably knew less about education and how to nurture conditions for learning was the man who hired him: Chancellor Klein. Columbia University finally woke up to the fact that he had been double-dipping: while holding the office of Chief Accountability Officer at Tweed, he was also supposedly on the full-time law faculty for the last year.

The progress reports he designed were widely derided as unreliable and statistically untenable; the quality reviews were an expensive waste of time and paperwork, and ignored when DOE was deciding which schools to close and which schools to commend; the $80 million supercomputer called ARIS was a super-expensive super-mugging by IBM, according to techies who found it laughable how much DOE was taken for a ride.

The surveys were badly designed, and counted for only a small percentage of school grades. Yet because principals were terrified of bad results, parents were pressured into giving favorable reviews for fear their schools would otherwise be punished. And the top priority of parents on these surveys — class size reduction — was ignored; worse, it was repeatedly derided by Liebman et. al. as a goal not worthy to pursue.

Under his leadership or lack thereof, the Accountability office continued to mushroom with more and more high priced educrats, "Knowledge Managers" and the like, few of whom, like him, had any experience or qualifications for the job, no less an understanding of statistics or the limitations of data.

One would think that a man who had focused professionally on the large error rate in capital punishment cases would have a little humility in terms of recognizing the fallibility of human judgment -- but no such luck. When confronted with the question of why schools should be given single grades, rather than a more nuanced system that might recognize their variety of attributes, he opined that a single grade, from A to F was useful "to concentrate the mind."

The ostensible point of the test score data from the periodic assessments and standardized tests, collected and spewed out by ARIS, to be analyzed by each school's "data inquiry teams” and "Senior Achievement Facilitators" was supposedly to encourage “differentiated instruction” to occur , although this goal was severely hampered by the fact that under Klein's leadership or lack thereof, overcrowding and excessive class sizes have continued.

No matter how much data is available — even assuming it is statistically reliable— the best way to allow differentiated instruction to occur is to lower class size.

And let us not forget Liebman’s cowardly run out the back door of City Hall in order to escape parents and hundreds of petitions collected by Time out from Testing — even though City Council Education Chair Robert Jackson had specifically requested that he leave through the front door of the chambers after he testified so that he could receive the petitions with the respect that they deserved. A perfect emblem of his three years at DOE.

ARIS: a view from the field

About ARIS, the $80 million supercomputer that Jim Liebman, the DOE accountability czar, insists is a valuable tool to improve instruction, here is a report from a teacher who attended a recent training sponsored by the Accountability office:

“ The whole afternoon was devoted to the "connect" features. These include a profile page, blogging, group formation, discussion boards, wiki documents and file uploading that come with everyone's ARIS account. The package is less than meets the eye. All of the features I examined were inferior to similar tools that are readily available online.

A couple examples. If you write a blog or set up a discussion board there's (apparently) no way to get email notification if someone responds. You'll have to keep logging into ARIS and checking for activity. That's will be a big drawback for those teachers who are willing to participate. Many of the participants at the workshop said basically, the Big Brother factor would preclude them from sharing anything meaningful. There was a lot of disbelief this was seriously being offered. Another very weak feature is the wiki documents. No tool bar. I've never seen anything so bare bones.

Before the training started the instructor asked everyone to share their feelings about ARIS in one word. The most common word was "skeptical." She said she'd do that again at the end to see if workshop had changed anyone's mind but we never got to do that. A lot of questions came up. There seems to be an attempt to present ARIS as both data and "community." Well there's going to be data ..."

See this post in Gotham Schools about how New Visions sent out an email pointing out problems with ARIS, including widespread errors in the data, which was later withdrawn -- undoubtedly after pressure from DOE.

Tweed still making unilateral -- and illegal -- decisions about closing schools

The DOE announced three schools that would be closed or phased out starting next year: MS 44 in Manhattan’s District 3, PS 225 in D27 in Queens, and PS 90 in District 9 in the Bronx.

MS 44 was the site of a big press event more than a year ago, when the Mayor, the Chancellor, Speaker Quinn, and Randi Weingarten of the UFT announced a new middle school initiative, supposed to help schools like this one succeed.

They said that a list of low-performing middle schools would receive $5 million in additional funds and professional development services free of charge (imagine that!), and that there would be a new position created for Superintendent of middle schools. See the press release here. I wonder what happened to those middle schools, and how many of them have actually improved.

Jennifer Freeman writes on the InsideSchools blog that the District 3 Community Education Council was not consulted before the DOE decided to close MS 44. This issue is also explored in the Gotham Schools blog here: UWS parents gear up for renewed diversity fight over school closure.

According to the state law that created Community Education Councils, these bodies are supposed to be consulted before any decision is made to close a school in their district:

"The chancellor shall consult with the affected community district education council before: (a) substantially expanding or reducing such an existing school or program within a community district.”

See this story from NY1 last year – which cites the law and adds this comment: "The CECs, as in the past, were not consulted before the announcement. They're being consulted now,” said James Liebman of the DOE.

If the CECs are still not being consulted about school closures, this is a violation of state law and they should contact their state legislators and consider taking legal action.

Klein Stiffs Parents of Children Receiving Special Ed Services to Defend Accountability

At Monday's Panel for Educational Policy meeting, Chancellor Joel Klein made a last minute agenda change to make room for Chief Accountability Officer Jim Liebman's rambling and defensive account of how much money his office spends. To make time for Liebman, Klein postponed an update on special education, drawing howls of protest from the many parents of children receiving special education services who had come specifically to hear the briefing. Citywide Council on Special Education member Patricia Connelly waited patiently for the public comment period, then gave an impassioned speech on how Klein's indifference was emblematic of his attitude toward the 180 thousand public school children with IEPs.

The impetus for the Accountability briefing was a report by the Independent Budget Office demonstrating how DOE will spend $129.6 million this year and $105 million next year on its Accountability initiative. The report, which drew wide press coverage, must have struck a nerve because Liebman came loaded for bear with a thirty-page Powerpoint, three live testimonials from DOE administrators and a live demonstration of the much-delayed ARIS system.

You can find the Powerpoint here. From start to finish it is a preposterous document. Some highlights:

• There are the usual statistics on DOE "success" which carefully omit any mention of Federal NAEP tests showing a lack of progress.
• An absurd statement from Columbia's Jonah Rockoff insisting school progress reports released only a couple of months before children sat for state tests were responsible for improving their scores.
• Not one but two pages of endorsements from those great sages of pedagogy, the editors of the Daily News.
• A blatantly false assertion that the $80 million spent on ARIS had to be spent on technology as part of the city's capital budget (actually, new schools, gyms or science labs would have been fine -- exactly the things Bloomberg tells us we can no longer afford)
  
• An extensive assault on the IBO analysis, the only point of which appears to be that, yes DOE is spending the hundreds of millions IBO said they are but technically speaking, it's not really on "accountability".

The reality is that Liebman is spending even more than even what IBO enumerated; they agreed to exclude the massive Interim Assessments expense which will be $26 million this year alone and another $8+ million in Senior Achievement Facilitators went uncounted.

At the end of his presentation, I asked Liebman how he could justify the six positions posted on the DOE web site with titles like "Knowledge Manager" and "Achievement Facilitator" when we are likely to see layoffs of art & music teachers and cuts in custodians. His reply that he was not necessarily really hiring people drew chuckles from the audience. The net of all this is that the Bloomberg administration is set on protecting this standardized testing juggernaut even if it means passing on the cuts on to our classrooms.

The PEP meeting ended with a long barrage of public comment. The parents who had come for the special education briefing were bitter, ATR and rubber room teachers were out in force and several parents spoke on the ever-present overcrowding crisis. See EdNotes for another account of meeting.

NYC standardized tests -- formative or not?

See Test Industry Split Over 'Formative' Assessment, in Education Week, about how standardized testing companies are selling their products as “formative” assessments, even though, according to most experts, formative assessments are diagnostic tools that are supposed to be classroom-based and designed by teachers themselves.

The Princeton Review described their much-derided NYC interim assessments as “formative” --the tests that DOE paid millions for and later rejected. See this Princeton Review presentation:

“Interim Assessment with Instructional Impact: How to use the formative, low-stakes testing system to support teaching and learning in New York City. …. We have a 20+ year history of using formative testing to support teaching and learning.

The subsequently renamed “periodic assessments” that the Accountability office under James Liebman contracted out to McGraw-Hill at $22 million annually, also known as “Acuity,” are commonly characterized as “formative” by Liebman et. al. See this recent pdf document from the DOE Accountability office:

Periodic Assessments support schools by providing …. formative, instructionally valuable feedback to support differentiation of instruction, determination of professional development needs, and selection of instructional resources.”

See the long list of periodic assessments now required in all NYC public schools in the chart above.

The EdWeek article quotes James Popham, professor emeritus at UCLA and former President of the American Educational Research Association: “When test companies sell things and call them formative...these vendors are being disingenuous—we used to call it lying.”

Unfortunately, they appear to be lying to a very eager clientele at the DOE.

More statistical malpractice from Tweed: Joel Klein and his claims do not measure up

Check out Elizabeth Green’s article today in the NY Sun, in which she asked three academics to scrutinize the validity of the administration’s claim of narrowing the achievement gap between ethnic and racial groups.

See, for example, Bloomberg’s recent testimony before Congress, in which he said that “over the past six years, we’ve done everything possible to narrow the achievement gap – and we have. In some cases, we’ve reduced it by half.”

Yet the evidence for this is weak to non-existent. On the national tests called the NAEPs, there has been no narrowing of the achievement gap in any area since the Bloomberg/Klein reforms were instituted:

An analysis by the National Center for Education Statistics, the research arm of the federal Education Department, concludes that no achievement gaps have narrowed at all in New York City between 2003 and 2007. The only gap that moved in any significant direction is the one between poor students and the rest of the population, which widened slightly, that analysis said. The National Center for Education Statistics also concludes that upward trends in the reading scores of black and Hispanic fourth-graders lauded by Mr. Klein are not statistically significant.

In the article, Joel Klein reveals his statistical illiteracy:

“Those are just confidence levels. Nobody is saying this is a science," Mr. Klein said. He added: "If three points is flat, and four points is statistically significant, then what you're doing is, you're playing something of a game."

Chief press officer David Cantor called the memo from NCES "a politicized gloss.”

Instead, it is the DOE who insists on playing games – and politicizing the issue, by continuing to slander experts as somehow biased when they provide objective evidence that the non-stop PR spin issuing from Tweed has no basis in reality.

This is hardly the first time the DOE has revealed such statistical malpractice. Jim Liebman, law professor and head of the DOE accountability office, is a repeat offender. I recall one episode in particular when Liebman, testifying before the City Council, insisted that he wasn't basing school grades primarily on the results of a few tests, since each test was really "multiple assessments" given out over "multiple days," resulting in "multiple measures" of proficiency.

Another instance of this was the presentation of Jennifer Bell-Elwanger, head of testing for DOE, who was giving a power point presentation to the Panel for Educational Policy in late November, following the release of the NAEP results. She continually pointed out gains that, according to the NCES, were not statistically significant. Patrick Sullivan, Manhattan rep to the PEP and fellow blogger here, who seems to understand data better than anyone currently employed by the DOE, questioned her closely, saying, "But by definition these are insignificant gains, no?" Which she, of course, fervently denied.

Indeed, the memo from the NCES was prepared in response to a highly misleading email that Klein sent to nearly every NYC resident after NAEP results were first reported. In his email, he falsely claimed that the results showed “good progress that is consistent with the overall picture” and said that the NAEP showed a narrowing of the gap in nearly all areas, when the data itself revealed quite the opposite.

According to NYC’s results on the state exams, the situation is more complicated. The achievement gap is narrowing in some areas when one looks at “proficiency” levels, that is whether a student is at a level 1, 2, 3, etc., but not in terms of the actual scale scores.

Some testing experts consider proficiency levels less meaningful than scale scores, as they can be arbitrary, subjective and easy to manipulate. Daniel Koretz, a professor at Harvard and a national expert on testing, has just published the must-read book of the summer, Measuring Up: What Educational Testing Really Tells Us. Here is what Koretz has to say about proficiency levels:

….the percents deemed proficient [on state tests] are largely unrelated to states’ actual levels of student achievement…are also often inconsistent across grades or among subjects in a grade....[This system] obscures a great deal of information [because] of the coarseness of the resulting scale.

Even more importantly, why should we trust the state scores at all, when we know that the gains that they purport to show are not reflected in more trustworthy measures like the NAEPs?

The best thing about the Koretz’ book is his lucid explanation of why “test score inflation” inevitably occurs when you attach high-stakes to exams, and how this undermines the integrity and validity of the results; this has increasingly been the case throughout the nation as a result of NCLB, but even more here in NYC, as a result of the increasingly high-stakes policies of the Bloomberg/Klein administration.

Steve Koss has written about this eloquently on our blog, in relation to Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

I have tried to explain this phenomenon to many elected officials, staff, and reporters over the years, apparently with little success. I certainly don’t know a single NYC media outlet that has ever mentioned it, though Campbell’s Law was cited in some recent letters to the NY Times in response to the administration’s experiment to pay students for high scores.

I recall a lengthy discussion of this issue several years back with NY Times reporter David Herszenhorn while he was still on the education beat, to explain my opposition to the Mayor’s newly-proposed 3rd grade retention policy. One of the reasons I so vociferously opposed this policy, and still do, was not just that it was unfair to the student to base such a life-altering decision on the basis of one single, fallible test score, with such large margins of error; and not just that retention has been shown to have a racially-disparate impact and hurt rather than help most low-performing students.

My opposition was also due to the fact that the more significant consequences are attached to any test, the less its results can be trusted as a reliable gauge of real learning.

Since then, of course, the administration has piled on more and more high-stakes consequences -- for students, teachers, and schools – by adding fifth and seventh grade retention, awarding principals, teachers and students monetary rewards for high scores, and threatening to close down schools if scores don’t improve fast enough. The scores themselves have been rendered entirely meaningless as a result, as excessive test prep, teaching to the test, cheating, and other strategies to “game” the system has totally overtaken our schools.

Here is what Koretz has to say about this:

"One might expect that with the huge increase in the amount of testing in recent years, we would know more…Ironically, the reverse is true. While we have far more data now than we did twenty of thirty years ago, we have fewer sources of data that we can trust. The reason is simple: the increasing in testing has been accompanied by a dramatic upsurge in the consequences attached to scores. This is turn has created incentives to take shortcuts --- various forms of inappropriate test preparation, including outright cheating – that can substantially inflate test scores, rending trends seriously misleading or even meaningless.”

To the administration, all this appears acceptable, because as long as test scores go up, this justifies their policies; in truth, they don’t seem to care if the increases are meaningful or not.

Their laissez-faire attitude is revealed by the total lack of interest evinced in following-up on even well-documented cases of cheating. (See for example this story in the NY Sun, which though it says the DOE is “investigating” this will likely lead nowhere, as such stories have in the past.)

This “anything goes” attitude is also reflected in Klein’s remarks in a recent interview in the NY Post:

Q: What about complaints about the report-card grades for schools?

A: The report cards were probably one of the noisy periods. But . . . I can't tell you how many principals said to me, 'You know, chancellor, I didn't get the right grade but I promise you I won't get the same one next year,' so I think that had a big impact.

Here, Klein implicitly acknowledged that even while principals do not accept the fairness of these grades, based primarily on one-year gains in scores, he is content as long as they guarantee that they will get these scores to rise in the future, by any means possible.

The mystery continues: who is funding the Klein/Sharpton operation?

See the latest email from David Cantor below – DOE’s press secretary-- on the fact that the new Klein/Sharpton operation is being funded "anonymously" -- though apparently not by Bloomberg himself. In our earlier posting, we reported his assertion that the Gates/Broad foundations were not implicated either.

One would think that given the kind of public campaign that these men say they are embarking upon, including staging "events at both political conventions” and attempting to influence the position of the next President, they should be obligated to reveal their source of financing.

David also questions my description of the DOE press office as large and well-funded.

Here is an excel file from last October with the names, salaries and positions of thirteen people employed in the DOE Communications office, as well as many additional PR staffers, including seven in the “Strategic Response Unit”, those tasked with responding to outraged emails from parents, whose canned non-responses usually result in outraging them even more.

According to those in the know, the DOE press office is at least four times the size of any previous administration and much larger than the press operation of any other city agency. It is more than twice as big as the PR department of the US Department of Education.

Not that its members don't earn their salaries, working overtime to cover the blunders and misstatements of their superiors.

Speaking of which, the file also contains the salaries of the top administrators at Tweed as of September 2007 – with Jim Liebman, former Columbia law professor, tied for third at $196,575.00. Liebman was appointed as head of the so-called Accountability office, despite the fact that he had no experience in the fields of public education or testing and proceeded to prove this.

See this newsclip from Channel 2 back in December; with Klein trying to justify the fact that 18 officials at Tweed were paid more than $190,000 a year, saying that they could earn three times their salaries in the private sector. I responded that if so, they should go back there, because they had abundantly proven that they don't know anything about public education.

As to the ongoing mystery about who is funding the Klein/Sharpton operation, I followed up by asking David who is paying his salary when he acts as the chief spokesperson for this effort. Is he getting paid extra by this "anonymous" donor -- or are his additional duties being covered by his regular salary, i.e. through taxpayer money? In the midst of budget cuts to schools, one would think this was rather hard to justify.

Secondly, is he thinking of writing an expose a la Scott McLellan about his adventures in the land of Tweed when Klein's term in office is over? I myself would pay a pretty penny for such a book, and I bet others would as well.

I was forced to turn down David’s offer to come to Tweed to fix his copying machine; as I don't have any particular expertise in that area (not that ever stopped the Chancellor in his hiring decisions.)

Perhaps by cutting down on the high salaries of some of Tweed’s top educrats -- or eliminating one or two positions in the burgeoning Accountability office, they might be able to afford to pay a repairman. Or they could save on Liebman’s salary, as he is supposed to return to his professorship at Columbia in July.

As soon as I receive a response from David, I will let you know.

---

From: david cantor [mailto:cantorrac@gmail.com]
Sent: Friday, June 13, 2008 11:20 PM
To: Leonie Haimson
Subject: Re: question for David Cantor: who is funding this project?

Leonie: The project is being funded anonymously. No public money will be spent. The mayor is not funding the project.

Re comments on your blog: If Class Size Matters ever wants to hold a press conference in Washington, the National Press Club room we used (Zenger Room) is available for $500. Also, I invite you to come over to the press office when next you're at Tweed and check out our "huge" communications "juggernaut" at work. I think you'll be surprised. If you're any good at fixing a copy machine we may put you to work.

David Cantor
Press Secretary
NYC Dept of Ed