Why the school progress reports and NYC education reporters deserve a big fat “F”

Gary Rubinstein, a math teacher at Stuyvesant, totally outclasses the numerous education reporters in this city in his analysis of the recent school grades.  

He shows via this graph to the right and discusses in his blog how there is little or no correlation in rank order for schools between last year or this year's progress reports. 

(Incidentally, in a subsequent posting, Rubinstein also points out that NYC charters are twice as likely to get "Fs" in progress than non-charters.)

Unfortunately the mainstream media continue to repeat without dispute Suransky’s claim that the progress reports were much more “stable” this year, even though 60% of schools changed grades.    

Not one reporter, to my knowledge anyway, has bothered to point out how experts have shown that 32-80% of the annual gains or losses in scores at the school level are essentially random – and yet 60% of the school grade is based upon these annual gains or losses. 

See the Daily News oped I wrote in 2007, in which I offer even more criticisms of the progress reports, including their inherent instability, “Why parents and teachers should reject the new grades”.  

 Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

Yet here is one typical headline from last week: School report cards stabilize after years of unpredictability. Here is the NY Times account:

“We have a really high level of stability this year, which is a good thing,” said Shael Polakow-Suransky, chief academic officer for the city’s Department of Education…. There is movement and that’s good because we are measuring one year of data and we expect schools will go up and down, but we don’t want to see movement caused by something that’s external to the kids,” Mr. Polakow-Suransky said, referring to changes in the state exams that caused incredible increases and then a drop-off in schools’ grades.

Of course, if one year’s movement up and down is primarily random, that by definition is “external.” 

Nor have any education reporters bothered to report that Jim Liebman, who designed the system, testified to the City Council when the grades were introduced that the DOE would improve the reliability of the system to incorporate three years worth of test score data instead– which both he and Suransky have refused to do.

Indeed, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read,  Liebman is quoted as responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results this way:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

And yet no one, including Fertig, has mentioned this discrepancy and DOE’s lack of rationale for intentionally allowing an essentially single unreliable grade to determine a school’s future – with 10% of schools now guaranteed to be given a failing grade and thus liable to being closed, based largely on chance. 

Read my earlier 2010 column, Why the school grading system, and Joel Klein, still deserve a big "F".

A court decision on the teacher data reports that will hurt our kids

It is unfortunate that the day after a court decision held that NY teachers should be evaluated by use of multiple assessments, with student scores on state standardized tests only one minor factor, today, the appellate court said that the DOE could release the teacher data reports to the public, based only on these same test scores. 

Most testing experts agree that these reports are highly unreliable and reductionist, and they will unfairly tarnish the reputation of many excellent teachers:

1.     The state tests were never designed for such a purpose – and are technically unable to make year to year judgments on “progress” or value added. 

2.    Many studies have shown the extreme volatility of these measures, and how the results differ even from one sort of test to another.  See Juan Gonzalez’s column on how DOE consultants themselves believe these reports are highly unreliable; here are links to the original documents revealing this, obtained through a  FOIL.

3.    As John Ewing, former executive director of the American Mathematical Society, recently concluded, ”if we drive away the best teachers by using a flawed process, are we really putting our students first?  Mike Winerip reported on a top-notch NYC teacher who was denied  tenure in just this manner.

If NYC goes ahead and releases this data it would likely be the first school district in the country to do so willingly and enthusiastically; when the LA Times generated its own value-added data for Los Angeles teachers, the paper was widely criticized.  Chris Cerf, former deputy Chancellor and now acting State Superintendent of NJ schools, was originally in charge of creating the teacher data reports; he promised that they would never be used for teacher evaluations and that the DOE would fight against any effort to disclose them publicly. In a 2008 letter to Randi Weingarten, Cerf wrote: "It is the DOE's firm position and expectation that Teacher data reports will not and should not be disclosed or shared outside the school community."

Chancellor Walcott should think twice before releasing this data, if he cares about real accountability, the morale of teachers,  and the potential damage to our kids.

Here are some of the recent studies from experts on the unreliability of this evaluation method:

Sean P. Corcoran, Can Teachers be Evaluated by Their Students’ Test Scores? Should they Be? The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice. As  the author concluded from his analysis, “The promise that value-added systems can provide a precise, meaningful, and comprehensive picture is much overblown… .Teachers, policy-makers and school leaders should not be seduced by the elegant simplicity of value-added measures. Given their limitations, policy-makers should consider whether their minimal benefits outweigh their cost.”  

Economic Policy Institute, “Problems with the Use of Student Test Scoresto Evaluate Teachers.” 2010.

National Research Council, Henry Braun, Naomi Chudowsky, and Judith Koenig, eds., GettingValue Out of Value-Added: Report of a Workshop, 2010: “Value- added methods involve complex statistical models applied to test data of varying quality. Accordingly, there are many technical challenges to ascertaining the degree to which the output of these models provides the desired estimates.”  

John Ewing, former executive director of the American Mathematical Society, current president of Math for America;  MathematicalIntimidation: Driven by the Data; “Why must we use value-added even with its imperfections? Aside from making the unsupported claim (in the very last sentence) that “it predicts more about what students will learn…than any other source of information”, the only apparent reason for its superiority is that value-added is based on data. Here is mathematical intimidation in its purest form—in this case, in the hands of economists, sociologists, and education policy experts…And if we drive away the best teachers by using a flawed process, are we really putting our students first?"

Sean P. Corcoran, Jennifer L. Jennings, Andrew A. Beveridge, Teacher effectiveness on high- and low-stakes tests; April 10, 2011. " To summarize, were teachers to be rewarded for their classroom's performance on the state test or alternatively, sanctioned for low performance many of these teachers would have demonstrated quite different results on a low-stakes test of the same subject.  Importantly, these differences need not be due to real differences in long-run skill acquisition…

That is, teachers deemed top performers on the high-stakes test are quite frequently average or even low performers on the low-stakes test. Only in a minority of cases are teachers consistently high or low performers across all metrics… Our results… highlight the need for additional research on the impact that high-stakes accountability has on the validity of inferences about teacher quality. "