Breaking: the Lederman decision and Gallup poll: the beginning of the end of high-stakes testing?

Today, the court decision in the Sheri Lederman case was issued.  Judge Roger McDonough of the NY State Supreme Court concluded that rating teachers via their students' growth scores on the state exams is "arbitrary and capricious."  He cited a wealth of evidence from affidavits of academic experts such as Linda Darling-Hammond, Sean Corcoran, Aaron Pallas, Carol Burris, Audrey Amrein-Beardsley, Jesse Rothstein  and others, showing that the system of evaluating teachers by means of test scores is unreliable, invalid, unfair and makes no sense.  The full court decision is below.

Here is the message from her attorney (and husband) Bruce Lederman:

"I am very pleased to attach a 13 page decision by J
udge Roger McDonough which concludes that Sheri has “met her high burden and established that Petitioner’s growth score and rating for the school year 2013-2014 are arbitrary and capricious.” The Court declined to make an overall ruling on the rating system in general because of new regulations in effect. However, decision makes (at page 11) important observations that VAM is biased against teachers at both ends of the spectrum, disproportionate effects of small class size, wholly unexplained swings in growths scores, strict use of curve.

The decision should qualify as persuasive authority for other teachers challenging growth scores throughout the County. Court carefully recites all our expert affidavits, and discusses at some length affidavits from Professors Darling-Hammond, Pallas, Amrein-Beardsley, Sean Corcoran and Jesse Rothstein as well as Drs. Burris and Lindell . It is clear that the evidence all of these amazing experts presented was a key factor in winning this case since the Judge repeatedly said both in Court and in the decision that we have a “high burden” to meet in this case. The Court wrote that the court “does not lightly enter into a critical analysis of this matter … [and] is constrained on this record, to conclude that petitioner has met her high burden” ...To my knowledge, this is the first time a judge has set aside an individual teacher’s VAM rating based upon a presentation like we made.

THANKS to all who helped in this endeavor."

At the same time, a national poll was released by the Gallup organization showing how most parents, teachers, students and administrators do not believe state exams are useful:

 Most teachers find their quality of the state exams are only "fair" or "poor":

And most families, whatever their income level, do not believe that these exams improve learning:

Let's hope that together these poll results, along with the Lederman decision, sound the death knell for the obsession with high-stakes testing that has overtaken our schools.

NY Judge finds rating teachers through growth (test) scores "arbitrary and capricious" by Leonie Haimson

A court decision on the teacher data reports that will hurt our kids

It is unfortunate that the day after a court decision held that NY teachers should be evaluated by use of multiple assessments, with student scores on state standardized tests only one minor factor, today, the appellate court said that the DOE could release the teacher data reports to the public, based only on these same test scores. 

Most testing experts agree that these reports are highly unreliable and reductionist, and they will unfairly tarnish the reputation of many excellent teachers:

1.     The state tests were never designed for such a purpose – and are technically unable to make year to year judgments on “progress” or value added. 

2.    Many studies have shown the extreme volatility of these measures, and how the results differ even from one sort of test to another.  See Juan Gonzalez’s column on how DOE consultants themselves believe these reports are highly unreliable; here are links to the original documents revealing this, obtained through a  FOIL.

3.    As John Ewing, former executive director of the American Mathematical Society, recently concluded, ”if we drive away the best teachers by using a flawed process, are we really putting our students first?  Mike Winerip reported on a top-notch NYC teacher who was denied  tenure in just this manner.

If NYC goes ahead and releases this data it would likely be the first school district in the country to do so willingly and enthusiastically; when the LA Times generated its own value-added data for Los Angeles teachers, the paper was widely criticized.  Chris Cerf, former deputy Chancellor and now acting State Superintendent of NJ schools, was originally in charge of creating the teacher data reports; he promised that they would never be used for teacher evaluations and that the DOE would fight against any effort to disclose them publicly. In a 2008 letter to Randi Weingarten, Cerf wrote: "It is the DOE's firm position and expectation that Teacher data reports will not and should not be disclosed or shared outside the school community."

Chancellor Walcott should think twice before releasing this data, if he cares about real accountability, the morale of teachers,  and the potential damage to our kids.

Here are some of the recent studies from experts on the unreliability of this evaluation method:

Sean P. Corcoran, Can Teachers be Evaluated by Their Students’ Test Scores? Should they Be? The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice. As  the author concluded from his analysis, “The promise that value-added systems can provide a precise, meaningful, and comprehensive picture is much overblown… .Teachers, policy-makers and school leaders should not be seduced by the elegant simplicity of value-added measures. Given their limitations, policy-makers should consider whether their minimal benefits outweigh their cost.”  

Economic Policy Institute, “Problems with the Use of Student Test Scoresto Evaluate Teachers.” 2010.

National Research Council, Henry Braun, Naomi Chudowsky, and Judith Koenig, eds., GettingValue Out of Value-Added: Report of a Workshop, 2010: “Value- added methods involve complex statistical models applied to test data of varying quality. Accordingly, there are many technical challenges to ascertaining the degree to which the output of these models provides the desired estimates.”  

John Ewing, former executive director of the American Mathematical Society, current president of Math for America;  MathematicalIntimidation: Driven by the Data; “Why must we use value-added even with its imperfections? Aside from making the unsupported claim (in the very last sentence) that “it predicts more about what students will learn…than any other source of information”, the only apparent reason for its superiority is that value-added is based on data. Here is mathematical intimidation in its purest form—in this case, in the hands of economists, sociologists, and education policy experts…And if we drive away the best teachers by using a flawed process, are we really putting our students first?"

Sean P. Corcoran, Jennifer L. Jennings, Andrew A. Beveridge, Teacher effectiveness on high- and low-stakes tests; April 10, 2011. " To summarize, were teachers to be rewarded for their classroom's performance on the state test or alternatively, sanctioned for low performance many of these teachers would have demonstrated quite different results on a low-stakes test of the same subject.  Importantly, these differences need not be due to real differences in long-run skill acquisition…

That is, teachers deemed top performers on the high-stakes test are quite frequently average or even low performers on the low-stakes test. Only in a minority of cases are teachers consistently high or low performers across all metrics… Our results… highlight the need for additional research on the impact that high-stakes accountability has on the validity of inferences about teacher quality. "

Update on FOILed backup for DOE claims as regards teacher data reports

Yesterday in court, the United Federation of Teachers argued that the DOE should not release the teacher data reports to the public, despite FOIL requests from media outlets, because the value-added methodology on which they are based are statistically unreliable, among other reasons, a point also made by many researchers, including Sean Corcoran of NYU. (See articles about the court case in today's Gotham Schools, NY1, Daily News, NY Times, and Post)

In February 2009, almost two years ago, I submitted a FOIL request to DOE for a number of items related to these reports, including the supposed "panel of technical experts" who had approved the DOE's methodology, according to the statement in the 2008 document, “Teacher Data Initiative: Support for Schools; Frequently Asked Questions":

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

When the FOIL was partially responded to fifteen months later, DOE admitted that this expert panel had not actually approved its methodology, and sent me a report in which the panel expressed grave doubts about its reliability.

Juan Gonzalez of the Daily News wrote about this here; I wrote about it and provided back up documentation here.

In its 2008 FAQ, DOE had also claimed that there was a research study that confirmed their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

In February 2009, I also asked for a copy this "research study." Coincidentally,I just received yet another email from DOE today, informing me that this study is still not complete, more than two years after the above claim was made, and nearly two years since I filed my original FOIL. (see letter above).

The Leadership Academy: the real deal?

Last week the Aspiring Principals Program of the New York City Leadership Academy, made headlines. On August 24, a NYU press release announced:

“Public elementary and middle schools in New York City led by ‘Aspiring Principals Program’-trained principals have achieved comparable or higher rates of student improvement than schools led by other new principals ... These results were obtained even though APP-trained principals were more likely to be placed in chronically low-performing schools.”

The New York Times chimed in: “Graduates of a program designed to inculcate school principals with unconventional thinking have gone on to help drive up English test scores even though the graduates were often placed at schools with histories of academic failure.” The article went on to explain that the APP graduates helped increase English Language Arts scores at elementary and middle schools “at a faster pace than new principals with more traditional résumés”; while in math the APP principals made progress, but “at a pace no better than their peers.”

The report

What did this report actually say? Written by Sean P. Corcoran, Amy Ellen Schwartz, and Meryle Weinstein of NYU’s Institute for Education and Social Policy, it compared the performance of schools under the leadership of graduates from the Aspiring Principals Program (APP) with that of schools under other new principals.

Both groups had to have been placed as new principals and to have remained in their positions for three years. Of the 147 graduates in the 2004 and 2005 APP cohorts, 88 (60 percent) met the inclusion criteria. 371 non-APP principals met the criteria; of these, 334 were in schools with comparable grade configurations. So there were 88 APP principals and 334 comparison principals in the study.

The schools in the two categories were significantly different. Compared to other new principals, APP principals tended to be placed in lower-performing schools and schools trending downward in ELA and math. There were also demographic and geographic differences.

The study used two types of comparison: (a) a straightforward comparison of average achievement in both types of schools and (b) a regression analysis (controlling for various school and student characteristics). It was the regression analysis that suggested an APP edge in ELA (but not for math) for elementary and middle schools.

The findings

The study found that test scores at schools in both groups improved over the period of the study in terms of test scores– but not as much as schools in the rest of the city. More specifically, the regression analysis indicated that the ELA standardized scores of APP elementary and middle schools were relatively stable, compared to schools headed by new principals who were not APP graduates. In math, APP elementary and middle schools fared slightly worse than comparison schools in relation to the city, but the differences were not statistically significant.

At the high school level (not mentioned in the NYU press release or NYT article), the differences between APP and comparison schools were “minor and inconclusive.”

Unanswered questions

There are many questions that the study did not address. Only 88 out of 147 graduates in the 2004-2005 and 2005-2006 cohorts met the inclusion criteria. More than 18 percent of APP graduates were never placed as principals at all. The rest stayed in their positions for fewer than three years.

Is this a high or low number? The authors wrote that they did not have comparative mobility information for the non-APP principals, but they presumably could have reported the average attrition rate for New York City principals overall.

Also, the study only analyzed test score data – which alone are insufficient to fully evaluate a school’s performance. Wasn’t there other data that could have been examined? What about the parent and teacher surveys at APP-headed schools compared to schools run by other new principals?

Though the study compared the size of the schools for both cohorts (APP graduates on average headed smaller schools) they did not compare class sizes – or other school-level conditions that could have contributed to the relative performance of both groups.

Most intriguing is the finding that the relative test scores at both sets of schools continued to decline compared to the rising achievement of schools citywide, but schools headed by APP principals declined less –at least in terms of their ELA results:

“... relative student test performance falls modestly in the years following the installation of a new principal, in both APP and comparison schools…..we find a statistically significant negative relationship between new principals and achievement in both mathematics and ELA.”

It was only after doing a regression analysis, by controlling for various factors (including student background), that they found that the relative performance of APP schools was relatively stable while the comparison schools continued to decline. See this graph:

Thus, the reigning philosophy of the Klein administration – that new leaders properly trained in the methods propounded by the Leadership Academy will spark significant improvements in low-performing schools does not seem to hold true. Instead, these appointees may stabilize what otherwise would be expected to be continued decline resulting from a new principal.

As a way to deal with ongoing attrition and fill positions in elementary and middle schools, the Leadership Academy might be said to be “promising”. But as a way to “turn around” schools it does not seem to be promising at all.

Unmentioned in any of the news articles was the fact that the research organization Mathematica had originally been commissioned by DOE to do an in-depth, multi-year study of the Leadership Academy. Yet after several years of analysis, this study was cancelled by DOE, just months before the results were supposed to be released. What Mathematica might have been discovered about the program and its graduates will probably never be known.

For another close look at this study, see Aaron Pallas’ critique at Gotham Schools.