Why the school progress reports and NYC education reporters deserve a big fat “F”

Gary Rubinstein, a math teacher at Stuyvesant, totally outclasses the numerous education reporters in this city in his analysis of the recent school grades.  

He shows via this graph to the right and discusses in his blog how there is little or no correlation in rank order for schools between last year or this year's progress reports. 

(Incidentally, in a subsequent posting, Rubinstein also points out that NYC charters are twice as likely to get "Fs" in progress than non-charters.)

Unfortunately the mainstream media continue to repeat without dispute Suransky’s claim that the progress reports were much more “stable” this year, even though 60% of schools changed grades.    

Not one reporter, to my knowledge anyway, has bothered to point out how experts have shown that 32-80% of the annual gains or losses in scores at the school level are essentially random – and yet 60% of the school grade is based upon these annual gains or losses. 

See the Daily News oped I wrote in 2007, in which I offer even more criticisms of the progress reports, including their inherent instability, “Why parents and teachers should reject the new grades”.  

 Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

Yet here is one typical headline from last week: School report cards stabilize after years of unpredictability. Here is the NY Times account:

“We have a really high level of stability this year, which is a good thing,” said Shael Polakow-Suransky, chief academic officer for the city’s Department of Education…. There is movement and that’s good because we are measuring one year of data and we expect schools will go up and down, but we don’t want to see movement caused by something that’s external to the kids,” Mr. Polakow-Suransky said, referring to changes in the state exams that caused incredible increases and then a drop-off in schools’ grades.

Of course, if one year’s movement up and down is primarily random, that by definition is “external.” 

Nor have any education reporters bothered to report that Jim Liebman, who designed the system, testified to the City Council when the grades were introduced that the DOE would improve the reliability of the system to incorporate three years worth of test score data instead– which both he and Suransky have refused to do.

Indeed, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read,  Liebman is quoted as responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results this way:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

And yet no one, including Fertig, has mentioned this discrepancy and DOE’s lack of rationale for intentionally allowing an essentially single unreliable grade to determine a school’s future – with 10% of schools now guaranteed to be given a failing grade and thus liable to being closed, based largely on chance. 

Read my earlier 2010 column, Why the school grading system, and Joel Klein, still deserve a big "F".