"Negative learning" and statistical malpractice at the Panel on Educational Policy

At last week’s meeting of the Panel on Education Policy at Tweed, Jim Liebman’s performance in attempting to defend the indefensible – the school grading system that he designed -- was breathtaking in its ignorance.

Liebman, the current DOE accountability “czar,” is a former criminal attorney, currently on leave from the Columbia law school, with no training or experience in education policy, statistics or testing, and yet the entire educational focus of the DOE is now based upon his faulty theories and expensive initiatives, including the $80 million supercomputer called ARIS, assigning letter grades to all schools primarily on the basis of one year’s worth of test scores, devoting millions of more dollars and hours of precious classroom time to interim standardized assessments, and the creation of “data inquiry teams” in all schools – all in the effort to “differentiate instruction” which in the end will be impossible without smaller classes.

At the PEP meeting, in order to justify the school grading system, he fastened on the “F” that PS 35 in Staten Island received, a school in which 98% of its students are on grade level in math, and 86% in ELA. Why did this exemplary school receive an “F”? Because last year, only 35% of its students improved their scores over the year before in reading, and only 23% in math – though research shows that a large part of annual variations in test scores are based on chance alone and are statistically unreliable. (For more on this, see my Daily News oped and a previous posting, Ten reasons to distrust the new accountability system.)

During the discussion, Liebman compared PS 35 to one of its “peer” schools – the Anderson school, a citywide Gifted and Talented school that accepts students on the basis of their high IQ and high test scores. When Patrick Sullivan pointed out the unfairness of comparing PS 35 to a selective school like Anderson, Liebman said it didn’t matter how the kids got there, they should all make the same annual gains. He failed to mention, however, that elementary schools are grouped with other schools according to only the roughest measures of demography –and that no statistician would compare the performance of a school that selects its students on the basis of test scores with a neighborhood school, like PS 35, that has to admit every child in its zone.

There was an abundance of statistical malpractice on display that night -- between Liebman’s presentation and the talk given by the DOE testing “expert”, Jennifer Bell-Elwanger, who tried to convince the panel that the city’s lack of significant progress on the NAEPs since 2003 was indeed real progress. Both of these individuals would have flunked an elementary course in statistics if they had tried to make these arguments in a college exam.

When asked wouldn’t it better to have separate grades for achievement and progress, rather than collapse all these categories into one grade, even if he were convinced that the lack of one year’s progress in test scores was significant (which it isn’t) Liebman replied that the good thing about giving a single grade is that it gets people’s attention (or something like that.) One could say the same about threatening to cut off the hands of someone accused of theft, or even capital punishment, which doesn’t mean it’s a remotely fair practice or even useful.

More recently, in response to questions about class size from parents in Manhattan and Queens, Liebman has insisted that the reason the DOE refuses to reduce class size is that classes would have to shrink to below 15 students to improve instruction and/or student achievement. In other words, lowering class size from 30 to 20 would make absolutely no difference.

Not only is such a statement absurd to anyone who has actually spent any time teaching in the public schools or observing classrooms, it is completely unsupported by research. Instead, it is simply another lame excuse that opponents of reducing class size like to throw up as a smokescreen in order to discourage such efforts.

Here is a comment sent to me from Chuck Achilles, a principal investigator of the famed STAR experiment in Tennessee and a professor of at Eastern Michigan University and at Seton Hall University. Chuck is also one of the premier class size researchers in the world:

“Hi Leonie:

I thought that the “below 15” idea (archaic) had faded. Anyone who says that is uninformed and ought to be asked (challenged) publicly to defend the assertion. It came once from one meta-analysis (Glass & Smith, 1988) that was very limited in its n of observations (77, of which some were for physical skills like hitting a tennis ball against a wall.) Just in STAR, we had more than 1300 observations in the range of 12-28 students. We typically analyzed reading outcomes, but sometimes we did math (giving us 2600 comparisons) and could have used other academic (test) outcomes… I’ve faxed some pages to show the linear effect: About a correlation of -.35 for each student added to a class. Because STAR used the class average as the unit of analysis, this means (approximately) the addition of each student to a class in the n=12-28 range reduces the class average score (about .1 of a month per year.) Later analyses show that it is cumulative.

Chuck A.”

Here is a fact sheet with numerous citations, showing there is no threshold in terms of reducing class size; and that the increase in achievement in relation to the decrease in class size is roughly linear.

Liebman reminds me of a phenomenon called “negative learning” ---in layman’s terms, a little learning is a dangerous thing. One would think that someone who got his reputation by writing about the high error rate in capital punishment would have a little humility and understand the possibility of human fallibility in making absolute judgments, but no such luck.