Breaking: the Lederman decision and Gallup poll: the beginning of the end of high-stakes testing?

Today, the court decision in the Sheri Lederman case was issued.  Judge Roger McDonough of the NY State Supreme Court concluded that rating teachers via their students' growth scores on the state exams is "arbitrary and capricious."  He cited a wealth of evidence from affidavits of academic experts such as Linda Darling-Hammond, Sean Corcoran, Aaron Pallas, Carol Burris, Audrey Amrein-Beardsley, Jesse Rothstein  and others, showing that the system of evaluating teachers by means of test scores is unreliable, invalid, unfair and makes no sense.  The full court decision is below.

Here is the message from her attorney (and husband) Bruce Lederman:

"I am very pleased to attach a 13 page decision by J
udge Roger McDonough which concludes that Sheri has “met her high burden and established that Petitioner’s growth score and rating for the school year 2013-2014 are arbitrary and capricious.” The Court declined to make an overall ruling on the rating system in general because of new regulations in effect. However, decision makes (at page 11) important observations that VAM is biased against teachers at both ends of the spectrum, disproportionate effects of small class size, wholly unexplained swings in growths scores, strict use of curve.

The decision should qualify as persuasive authority for other teachers challenging growth scores throughout the County. Court carefully recites all our expert affidavits, and discusses at some length affidavits from Professors Darling-Hammond, Pallas, Amrein-Beardsley, Sean Corcoran and Jesse Rothstein as well as Drs. Burris and Lindell . It is clear that the evidence all of these amazing experts presented was a key factor in winning this case since the Judge repeatedly said both in Court and in the decision that we have a “high burden” to meet in this case. The Court wrote that the court “does not lightly enter into a critical analysis of this matter … [and] is constrained on this record, to conclude that petitioner has met her high burden” ...To my knowledge, this is the first time a judge has set aside an individual teacher’s VAM rating based upon a presentation like we made.

THANKS to all who helped in this endeavor."

At the same time, a national poll was released by the Gallup organization showing how most parents, teachers, students and administrators do not believe state exams are useful:

 Most teachers find their quality of the state exams are only "fair" or "poor":

And most families, whatever their income level, do not believe that these exams improve learning:

Let's hope that together these poll results, along with the Lederman decision, sound the death knell for the obsession with high-stakes testing that has overtaken our schools.

NY Judge finds rating teachers through growth (test) scores "arbitrary and capricious" by Leonie Haimson

Who to believe among "experts" on teacher evaluation at Albany Summit on teacher evaluation?

Today, from Albany NYSED is livestreaming what they call a "Learning Summit on Annual Professional Performance Review (APPR), the teacher and principal evaluation system.   The ostensible purpose of the meeting is to get input from experts, educators and parents about how to go about crafting their new teacher evaluation system that they are supposed to come up with by June 30, but that is severely restricted by the damaging rubric imposed by the Governor.

On the agenda from 1-2 PM, is a panel of  "National experts in the field on education, economics and psychometrics."  The invitees include:

• Thomas Kane, an economist from Harvard University who strongly supports test-based teacher evaluation and led the Gates Foundation’s Measures of Effective Teaching study;
• Catherine Brown, VP of the Center for American Progress, which has published papers endorsing the use of value-added and has received more than $5 million from the Gates Foundation for its education work.  Brown is also married to Robert Gordon, formerly of NYC DOE, OMB and the US Dept of Education, who pushed test-based teacher evaluation in NYC and throughout the country.
• Sandi Jacobs, a vice president at the National Council on Teacher Quality which also strongly supports test-based teacher evaluation and has gotten more than $12 million from the Gates Foundation;
• Leslie Guggenheim of TNTP, an advocacy organization whose 2009 paper “The Widget Effect” promoted test-based teacher evaluation and has gotten more than $33 million from the Gates Foundation.

On the other side with a more skeptical view include academics who are not on the Gates payroll: Aaron Pallas of Teachers College, Jesse Rothstein of UC Berkeley, and  Stephen Caldas of Manhattanville College.

So here we have three representatives from inside-the-Beltway advocacy groups that collectively received more than $50 million to make the case for test-based teacher evaluation and one professor who led the $45 million MET project for Gates, vs three independent academic scholars.

Also  speaking at 4 PM is a parent panel selected by the NYS PTA, including a representative from NY State Allies for Public Education, a coalition of more than 50 parent and advocacy groups statewide (full disclosure: including Class Size Matters.)  NYSAPE has helped lead the anti-testing movement that garnered at least 200,000 students opting out this spring.  Also on that panel, strangely enough, is Matt Barnum, the policy director of Educators for Excellence, which has received  $4 million from the Gates Foundation.

Who to believe?  You be the judge.

Concerns with the MDRC study on small schools released today

MDRC released a  study today, which the NY Times writes “appeared to validate the Bloomberg administration’s decade-long push to create small schools to replace larger, failing high schools.” The report mentions the current controversy over the massive number of school closings, here in NYC and across the country, and thus there may be a political element in the timing of its release:

MDRC’s findings about SSCs are relevant to current federal policy on high school reform, particularly the U. S. Department of Education’s School Improvement Grants (SIGs) for failing schools. Reforms funded by SIGs include school transformation, school restart, school closing, and school turnaround. SSCs straddle several of these categories since they are typically replacements for schools that have closed and they operate as regular public schools.

This is the second MDRC study to conclude that students who attended the new small schools had significantly improved outcomes.  The first MDRC study, released in 2010, looked at students who entered these schools in 2005; this one adds students who entered in 2006 to that group.

I have a lot of reservations about using this study or the previous study to justify the small schools initiative and especially to justify the current massive round of school closings.  I am no expert in statistics, but my concerns revolve around these issues: 

1-      Though the study points out that the small schools were supposedly “unscreened” and evaluates their results by comparing the outcomes of students who applied to the school through lotteries,  compared to those who lost the lottery, it  ignores that the students who attended these small schools were far less high-needs on average, as evidence by their lower rates of English language learners and special needs, as shown by this Annenberg study by Pallas and Jennings.  In fact, they were allowed to openly exclude special needs students during the first two years. Thus even with a “lottery” for admissions, there are substantial peer effects for students who are grouped with higher-achieving students which this study does not mention.  (This is also a problem with many of the charter school studies, like this one, which tend to ignore peer effects.)

2-      The results in terms of higher graduation rates and college readiness (based on Regents scores and credit accumulation) ignore how in NYC, teachers and principals are able manipulate these in ways that do not reflect real learning (especially as teachers grade the Regents of students in their own schools, and in these schools, their own students!)  It has also been alleged that the small schools pioneered the now widespread and largely discredited practice of “credit recovery.” With newer teachers at many of the small schools, who did not have a memory or tradition as to earlier practices, it may have been easier to pressure these teachers into employing such methods.

3-      The study ignores that the small schools on average were allowed to have smaller classes and were far less overcrowded than the large high schools, which legitimately could have led to better results.  The class size at the small schools during these years were from 13 to 20 students per class, according to the PSA first year report, compared to 30 or more at the larger schools.  If the higher needs students in the larger schools had been provided with smaller classes, very likely their chance of success would have been improved substantially as well.

4-      Yet the study doesn’t examine how the opening of the small schools had a negative impact on the system as a whole, by flooding nearly large high schools with the most disadvantaged and academically challenged students, leading to even more overcrowding, larger class sizes, and damaging their opportunity to learn, as reported by many observers and confirmed by the New School’s report, The New Marketplace.

5-      The MDRC study deals with only a subsection of the small schools that were oversubscribed and required a lottery for admissions, so like the charter school studies which use a similar methodology, their success rate may not be representative of small schools overall.

6-      It study ignores the reality that these so-called random “lotteries” may be far from actually random. The MDRC study compares the baseline characteristics of students who “won” the lotteries to those who lost, in a  Supplemental Table 1, which purports to show that both groups were “virtually identical”; but the comparison does not include students who required collaborative team-teaching or self-contained classes; does not include  prior attendance rates, a key factor that principals often examing when selecting students; and does not differentiate between free and reduced price lunch students.  The table also does not include data on previous rates of suspension.  According to an earlier evaluation done by Policy Studies Associates. the ninth graders who entered the small schools had far better attendance records (91% compared to 81%), and were less likely to have been suspended as 8^th graders compared to students at the schools they replaced.

Even in the categories the MRDC study does compare, there are greater numbers of higher achieving students, though the differences are not statistically significant, according to the model used.   Most strangely, the table compares the baseline data for four cohorts – entering ninth-graders from the 2004-2005 to 2007-2008 years; while the report compares outcomes for only the first two of these cohorts, students who entered these schools from 2005-6 and 2006-7. This is despite the fact that that the need level of students increased significantly after that point – and likely, the challenges faced by these schools as well.  See the above chart, for example, from the Annenberg study by Jennings and Pallas.

I have no idea why the MRDC study lumped together all these cohorts to examine their baseline characteristics, even as they only compared the outcomes for the first two, but it may considerably bias their conclusions.  Indeed, more than half of the middle and high schools being closed by the DOE this year for poor results are small schools that were founded after 2003.

7- There are several ways in which, especially in the early years of the small schools initiative, principals were able to manipulate the admissions process to get the students who were more likely to succeed, even though by definition these schools were supposedly “unscreened” and used “random” lotteries for admissions.  See this excerpt from a published study by Jennifer Jennings, who embedded herself with three small schools between March 2004 and September 2005: 

My observations revealed that many schools used applications, mandatory information sessions, and much stronger language to deter unwanted applicants. For example, 12 unscreened schools shared a similar application requiring that students provide the most recent report card and two letters of recommendation, one from an eighth-grade teacher and one from a guidance counselor, assistant principal, or principal. The application also asked for the student’s test scores, retention history, and involvement in advanced courses during the eighth grade. Finally, the application included additional questions requiring a narrative response….

The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. ….

The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. This data file included each student’s English-language-learner and special education classification, reading and math test scores, absences, grades, address, and junior high school. Schools were told to identify students who made an ‘‘informed choice’’ by assigning them a 1, while students who did not make an informed choice but the school was willing to accept were assigned a 2. If the school did not fill all of its seats with students making an informed choice, additional seats would be filled by students in the second category.  The Department of Education prohibits unscreened schools from using student performance data to select students. Nonetheless, both Marlena and Anna [pseudonyms for two principals of small schools] learned through their relationships with other principals that such regulations were loosely enforced….

 In addition to the English language learners and full-time special education students whom new schools had a waiver to eliminate, Renaissance [pseudonym for one of these small schools] eliminated part-time special education students and chose only those with 90 percent or higher attendance. Excel eliminated full- and part-time special education students and chose students with attendance rates of 93 percent or higher. 

There are many more revealing insights in the Jennings study about how the small schools were able to deflect over-the-counter students and counsel out low-performers to achieve better results, despite the fact that they claimed to be non-selective.  (Update: I have removed the link to the paper at the author's request; the abstract is here.)

All of these concerns  should provide one with reservations about the validity of the MDRC study and especially its apparent endorsement of the mayor's policies.

video: Leonie Haimson, Aaron Pallas and Danielle Lee on mayoral control

On Nov. 17, I appeared on the BronxNet TV show Perspectives, which asked the question, "Are We Experiencing an Education Crisis?"  The show was hosted by Darren Jaime, and the other panelists included  Aaron Pallas of Columbia University; Sophia James, consultant, and Dr. Danielle Lee, head of the Harlem Education Activities Fund (HEAF).

The entire show is well worth watching.  Here is an excerpt focused on who is ultimately responsible for the failure of NYC public schools:

Why the school grading system, and Joel Klein, still deserve a big "F"

Amidst all the hype and furor of the release of today’s NYC school "progress reports", everyone should remember how the grades are not to be trusted. By their inherent design, the grades are statistically invalid, and the DOE must be fully aware of this fact. Why?

See this Daily News oped I wrote in 2007, in which all the criticisms still hold true, “Why parents and teachers should reject the new grades”.

In part, this is because 85% of each school’s grade depends on one year’s test scores alone – which according to experts, is highly unreliable. Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

(source: Thomas Kane, Douglas O. Staiger, “The Promise and Pitfalls of Using Imprecise School Accountability Measures, The Journal of Economic Perspectives, Autumn, 2002.)

Now Jim Liebman admitted this fact, that one year’s test score data was inherently unreliable, in testimony to the City Council, and to numerous parent groups, including to CEC D2, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read.” In responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results, he admitted:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

Nevertheless, the DOE and Liebman have refused to comply with this promise, which reveals a basic intellectual dishonesty. This is what Suransky emailed me about the issue, a couple of weeks ago, when I asked him about it before our NY Law school “debate.”

“We use one year of data because it is critical to focus schools’ attention on making progress with their students every year. While we have made gains as a system over the last 9 years, we still have a long way to reach our goal of ensuring that all students who come out of a New York City school are prepared for post-secondary opportunities. Measuring multiple years’ results on the Progress Report could allow some schools to “ride the coattails” of prior years’ success or unduly punish schools that rebound quickly from a difficult year.”

Of course, this is nonsense. No educators would “coast” on a prior year’s “success”, but they would be far more confident in a system that didn’t give them an inherently inaccurate rating.

Given the fact that that school grades bounce up and down each year, most teachers, administrators and even parents have long figured out how they should be discounted, and justifiably believe that any administration that would punish or reward a school based on such invalid measures is not to be trusted.

That DOE has changed the school grading formula in other ways every year for the last three years also doesn’t give one any confidence….though they refuse to change the most fundamental flaw. Yet another major problem is while the teacher data reports take class size into account as a significant limiting factor in how much schools can get student test scores to improve, the progress reports do not.

There are lots more problems with the school grading system, including the fact that they are primarily based upon state exams that we know are themselves completely unreliable. As MIT professor Doug Ariely recently wrote about the damaging nature of value-added teacher pay, because of the way they are based on highly unreliable measurements:

…What if, after you finished kicking [a ball] somebody comes and moves the ball either 20 feet right or 20 feet left? How good would you be under those conditions? It turns out you would be terrible. Because human beings can learn very well in deterministic systems, but in a probabilistic system—what we call a stochastic system, with some random error—people very quickly become very bad at it.

So now imagine a schoolteacher. A schoolteacher is doing what [he or she] thinks is best for the class, who then gets feedback. Feedback, for example, from a standardized test. How much random error is in the feedback of the teacher? How much is somebody moving the ball right and left? A ton. Teachers actually control a very small part of the variance. Parents control some of it. Neighborhoods control some of it. What people decide to put on the test controls some of it. And the weather, and whether a kid is sick, and lots of other things determine the final score.

So when we create these score-based systems, we not only tend to focus teachers on a very small subset of [what we want schools to accomplish], but we also reward them largely on things that are outside of their control. And that's a very, very bad system.”

Indeed. The invalid nature of the school grades are just one more indication of the fundamentally dishonest nature of the Bloomberg/Klein administration, and yet another reason for the cynicism, frustration and justifiable anger of teachers and parents.

Also be sure to check out this Aaron Pallas classic: Could a Monkey Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education?

Tying tenure to test scores: not ready for prime time

Lots of interesting letters to the Times today, deploring the Mayor's proposal to base tenure decisions on test scores. [see “Mayor to Link Teacher Tenure to Test Scores” ]

In the same vein, Aaron Pallas has a column in Gotham schools, Teacher Education in New York State: A skoolboy’s-Eye View, in which he lucidly explains how the evaluation of teachers based on value-added student test scores is not ready for prime time. Pallas recently appeared on a panel at Teachers College with David Steiner, new NY Commissioner of State Education, (photo to the right), and Merryl Tisch, head of the Board of Regents. (You can see a webcast of this event here.)

In his column, Pallas urges Steiner and Tisch to start working on improving the state exams, which have gotten radically easier over time, before beginning to consider a system that would base decision-making on their results. He also points out how the long-standing practice of having high schools score their own Regents exams is a system ripe for abuse.

As part of the state's "Race to the Top" proposal, Commissioner Steiner recently also proposed that they expand the awarding of teaching degrees -- allowing providers other than institutions of higher learning to offer teacher preparation programs, with the Board of Regents granting Master’s degrees to candidates who "graduate" from these programs.

There is so much lacking in terms of the state's current oversight -- of district spending practices, of cheating, of "credit recovery", of the proper reporting of graduation rates, of whether schools are even providing the minimal services to kids that they are entitled to by law.

Given the awful mess at State Ed which Steiner has not yet begun to clean up, I would hate to see him allow further abuses to occur by deregulating the awarding of teaching degrees -- which could easily make a teaching certificate as meaningless as passing the Regents exam is now.

New book on Bloomberg/Klein record

Check out our new book on the Bloomberg/Klein educational regime: "NYC Schools Under Bloomberg and Klein: What Parents, Teachers and Policymakes Need to Know."

With chapters by contributors to this blog like Diane Ravitch, Steve Koss, and Patrick Sullivan, and by other experts like Debbie Meier, Hazel Dukes of the NAACP, Udi Ofer of the NYCLU, Aaron Pallas of Columbia University and Jennifer Jennings (A/K/A Eduwonkette), it is must-read for anyone concerned about the future of our schools in this city -- and indeed the nation.

Our findings go behind the headlines to present an inside view on how the Mayor's unfettered authority has affected students, families, teachers, and communities -- you can purchase a copy now or download one for free at the Lulu website here.