Why the school grading system, and Joel Klein, still deserve a big "F"

Amidst all the hype and furor of the release of today’s NYC school "progress reports", everyone should remember how the grades are not to be trusted. By their inherent design, the grades are statistically invalid, and the DOE must be fully aware of this fact. Why?

See this Daily News oped I wrote in 2007, in which all the criticisms still hold true, “Why parents and teachers should reject the new grades”.

In part, this is because 85% of each school’s grade depends on one year’s test scores alone – which according to experts, is highly unreliable. Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

(source: Thomas Kane, Douglas O. Staiger, “The Promise and Pitfalls of Using Imprecise School Accountability Measures, The Journal of Economic Perspectives, Autumn, 2002.)

Now Jim Liebman admitted this fact, that one year’s test score data was inherently unreliable, in testimony to the City Council, and to numerous parent groups, including to CEC D2, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read.” In responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results, he admitted:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

Nevertheless, the DOE and Liebman have refused to comply with this promise, which reveals a basic intellectual dishonesty. This is what Suransky emailed me about the issue, a couple of weeks ago, when I asked him about it before our NY Law school “debate.”

“We use one year of data because it is critical to focus schools’ attention on making progress with their students every year. While we have made gains as a system over the last 9 years, we still have a long way to reach our goal of ensuring that all students who come out of a New York City school are prepared for post-secondary opportunities. Measuring multiple years’ results on the Progress Report could allow some schools to “ride the coattails” of prior years’ success or unduly punish schools that rebound quickly from a difficult year.”

Of course, this is nonsense. No educators would “coast” on a prior year’s “success”, but they would be far more confident in a system that didn’t give them an inherently inaccurate rating.

Given the fact that that school grades bounce up and down each year, most teachers, administrators and even parents have long figured out how they should be discounted, and justifiably believe that any administration that would punish or reward a school based on such invalid measures is not to be trusted.

That DOE has changed the school grading formula in other ways every year for the last three years also doesn’t give one any confidence….though they refuse to change the most fundamental flaw. Yet another major problem is while the teacher data reports take class size into account as a significant limiting factor in how much schools can get student test scores to improve, the progress reports do not.

There are lots more problems with the school grading system, including the fact that they are primarily based upon state exams that we know are themselves completely unreliable. As MIT professor Doug Ariely recently wrote about the damaging nature of value-added teacher pay, because of the way they are based on highly unreliable measurements:

…What if, after you finished kicking [a ball] somebody comes and moves the ball either 20 feet right or 20 feet left? How good would you be under those conditions? It turns out you would be terrible. Because human beings can learn very well in deterministic systems, but in a probabilistic system—what we call a stochastic system, with some random error—people very quickly become very bad at it.

So now imagine a schoolteacher. A schoolteacher is doing what [he or she] thinks is best for the class, who then gets feedback. Feedback, for example, from a standardized test. How much random error is in the feedback of the teacher? How much is somebody moving the ball right and left? A ton. Teachers actually control a very small part of the variance. Parents control some of it. Neighborhoods control some of it. What people decide to put on the test controls some of it. And the weather, and whether a kid is sick, and lots of other things determine the final score.

So when we create these score-based systems, we not only tend to focus teachers on a very small subset of [what we want schools to accomplish], but we also reward them largely on things that are outside of their control. And that's a very, very bad system.”

Indeed. The invalid nature of the school grades are just one more indication of the fundamentally dishonest nature of the Bloomberg/Klein administration, and yet another reason for the cynicism, frustration and justifiable anger of teachers and parents.

Also be sure to check out this Aaron Pallas classic: Could a Monkey Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education?

The absurd, silly, ridiculous school grades

See articles on this year’s elementary and middle school grades, or “progress reports”: Times, NY Post, Daily News, NY1, and this piece in Gotham schools (the best).

They are replete with examples of the absurd results, with schools such as PS 82 in the Bronx jumping from an “A” to an “F” and back this year to an “A”. PS 8 in Brooklyn which got an “F” last year and this year got an “A”, with the principal saying they did nothing different. The three schools that Klein tried to close last year to put charter schools in their place-- P.S. 194, 241 and 150 – all got “As”.

In fact, 84% of the schools got an “A” this year, and only two schools out of more than one thousand got “F”s. This is grade inflation that would put any human being other than Joel Klein to hide his head in shame.

And yet, according to the NY Times, “he clearly took pride in the results. “If you’re asking whether I would rather see less A’s,” he said, “the answer is no.”

Meanwhile, 87% of principals said in a recent survey that their schools were unable to provide a quality education because of excessive class sizes.

The absurdity of the grades this year derive from two profound flaws: First, 85% of the grade is based on one year’s gains or losses in test scores, which experts have found to be statistically unreliable and extremely erratic.

And two, the state tests have become so much easier and their scoring so lax that students can pass them without reading the questions – as long as they manage to fill in a few bubbles along the way. (For more on this scandal, well-reported everywhere except the Times, see the Daily News here and here, Gotham Schools and this NY Post column by Diane Ravitch.)

I wrote an oped for the Daily news about the new grading system when it was first announced in the fall of 2007: “Why parents & teachers should reject new school grades.” It starts out this way: “The new school grading system unveiled this week by Chancellor Joel Klein is a fiasco.”

But what do you expect when you had two guys in charge, Liebman and Klein, who are ignorant as to statistics, the unreliability of test scores, and even the larger goals of education?

Perhaps the best result is where we have now arrived: hopefully everyone realizes that the emperor has no clothes and they should ignore these silly grades, as they should have in the first place.

A warning to Australians: don't let your kids suffer, as have ours!

This week, Joel Klein is visiting Australia to promote his educational policies and perspective. To alert Aussies to the disinformation campaign he will likely be spreading, we have deconstructed a recent interview on Australian TV with Klein, carried out by journalist Kerry O'Brian, and aired last month on Australian TV.

[The full video of the Klein interview is here and the transcript here.]

Question: "You can't have a system corrupted or distorted by a principal under pressure or a principal that wants to boost his or her status by encouraging teachers at the school to go soft on the students in tests to help them cheat in effect?"

JOEL KLEIN: "It's happened on a handful of occasions since I’ve been here and we’ve terminated principals, terminated a couple of teachers at one point, but I don't think systematically that's a powerful explanation.

And even before our system there was a Federal system here in the US under "no child left behind." So schools are under a certain amount of pressure, but I don't think somebody is going to imperil their livelihood by taking the risk of trying to have students cheat on exams."

In reality, there is now rampant test prep in NYC schools – as well as many serious allegations of cheating – few of which are ever followed up by this administration, even when the allegations are based on reliable evidence from people on the ground. In fact, many principals have been promoted and given bonuses whose schools have seen large and unexplained boosts in test scores. See for example this article, detailing allegations about cheating at one school, where test scores jumped 30 points in one year, where Chancellor Joel Klein spoke at the school's graduation while wearing a "Best School in the Universe" T-shirt.

In fact, Klein doesn’t appear to care what’s causing test scores to rise, as long as they do. And as pointed out by independent experts like Diane Ravitch, as well as in the NY Times and elsewhere, test scores on state exams have not been matched by NYC's results on the more reliable national assessments called the NAEPs. Indeed, an analysis of NAEP scores between 2003-2007 reveals that NYC came in 10^th out of 11 urban school districts across the nation in changes in test score changes over the course of this administration.

JOEL KLEIN: "The class sizes have reduced…”

Actually, Joel Klein has stubbornly refused to reduce class size, the top priority of NYC parents, even when it means ignoring the law. As a result, our class sizes remain the largest in the state and among the largest in the nation, despite hundreds of millions of dollars allocated by the state to reduce class size.

In a 2006 audit, the State Comptroller found that only 20 additional classes were formed in NYC schools over the baseline figure, despite $89 million in annual state funds specifically allocated to reduce class size. The audit concluded that “while the [NYC] DoE was receiving State funding… it was reducing its own support for early grade class size reduction and using it for other purposes.” The NY State Comptroller proposed a number of recommendations to improve compliance with the law– all of which the Chancellor rejected.

Then again, this past September, the NY State Education Department found that despite commitments by the city to use millions of dollars in new state aid to reduce class size, instead “53.9% of New York City schools reported that either class size or pupil-to-teacher ratio increased in 2007-08.”

Accordingly, over 80% of NYC parents say that there has been no improvement in class size under this administration, and 86% of NYC principals say that they are unable to provide a quality education to their students because of excessive class sizes at their schools.

JOEL KLEIN: "The President of our [teachers] union, who's also the President of the National Union, she said on the first day of school this year in September that she had been traveling around the US and no place, in any urban area, was comparable to the work that we're doing. In fact he word she used was that New York City had become a beacon."

Among NYC teachers, there has been overwhelming disapproval of the leadership and policies of Joel Klein. Last year, the NYC teacher’s union released its own survey, showing that 85% of its members believe that Klein has refused to provide the support and resources they need to succeed; 85% disagreed with his emphasis on high-stakes testing; and 83% said that he had put other interests above the learning needs of the children.

According to Klein’s top spokesperson, only two out of twenty of the top administrators at the Department of Education are long-term educators; instead, he has largely staffed the Department with corporate consultants and attorneys.

In the past few weeks, thousands of teachers in NYC and elsewhere have signed petitions, warning President-elect Obama not to appoint Joel Klein to his administration, because of his misguided policies.

Check out this petition, which has been signed by nearly 4200 educators in NYC and nationwide:

"The NYC Department of Education under Joel Klein has been run like a ruthless dictatorship -- with no input from parents or educations. Teachers have not been respected, consulted, nor listened to. And little thought has been devoted to how the policies he has imposed on our schools have been destructive to the children and their futures...While focusing on test scores, he has consistently ignored the crisis of overcrowding in NY schools. Thousands of children are being given special services in hallways or in closets."

See also this petition, signed by more than 3600 teachers and academic experts nationwide, in opposition to Joel Klein's "vision of privatized, corporatized, and anti-democratic schools."

QUESTION: "How do you answer the critics who say your school statistics are flawed because of wild statistical fluctuations or results from year to year, which suggest fundamental flaws?"

JOEL KLEIN: "I don't think that's a fair criticism. I've actually studied the statistics. There are some year to year fluctuations that are significant, but that's because schools that weren't making progress refocused and decided that the risk to them was they were gonna close.

"And so they decided that they would focus much more effectively on promoting student performance. But all of these things will come out over the years, meaning this is not a one year or two year experiment.”

Some year-to-year fluctuations? School grades have widely varied between one year to the next, which is not surprising considering they are primarily based on one year's worth of test score gains or losses, which experts have shown to be 30-80% random.

See this analysis by Daniel Koretz, an expert on testing and statistics from Harvard, pointing out that three quarters of NYC schools that received an “F” last year received an “A’” or “B” this year. As Koretz writes:

“It strains credulity to believe that if these schools were really “failing” last year, three-fourths of them improved so markedly in a mere 12 months that they deserve grades of A or B ...This instability is sampling error and measurement error at work. It does not make sense for parents to choose schools, or for policymakers to praise or berate schools, for a rating that is so strongly influenced by error.”

Aaron Pallas and Jennifer Jennings of Columbia University have concluded that “The progress measure…is a fruitless exercise in measuring error rather than the value that schools themselves add to students.”

As another of their summaries suggest, “a Monkey [could] Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education."

In short, NYC parents and teachers have this advice for Australians: Do not be fooled by either Klein or his claims.

Achievement has not significantly improved under his leadership. Instead, his administration has consistently misused millions of dollars meant to reduce class size that would have provided children with a better chance to learn, while spending millions more in taxpayer funds on expanding the bureaucracy with high-paid corporate consultants, no-bid contracts, and more testing in our schools.

Along the way, he has stubbornly refused to listen to parents, teachers and academic experts who have pointed out the destructive impact of his policies.

Take heed -- and reject these policies that will cause your kids to needlessly suffer -- as have ours.

Forewarned, Australia is ready for Joel Klein!

According to the Age, the Australian teachers union and parent advocates are already rising up in protest against the highly unreliable NYC school grading system, which Joel Klein intends to promote on a visit down under next month, after being invited by Julia Gillard, their Education Secretary:

The federal president of the Australian Education Union, Angelo Gavrielatos, said the Australian Government should not be importing "flawed" approaches from the US, a nation that was consistently outperformed by countries such as Finland which did not publicly rank schools.

Canberra-based public education advocacy group Save Our Schools last week called on Ms Gillard to release the details of her performance reporting plan to ensure it did not reproduce the problems of the New York system, which it said had led to league tables and dissimilar schools being compared with each other.

See also the following postings from Save our Schools – which has studied the analyses of the NYC school grades by Diane Ravitch, Eduwonkette, and heard from the parents on our NYC education listserv about the numerous problems with the priorities of the Bloomberg/Klein administration.

The comments of NYC public school parent Ann Kjellberg are featured on Testimony From New York on Klein’s School Reforms:

Pretty much anyone in New York who has a kid in the system or who works in the system outside of “central office” hates these policies. It’s a triumph of public relations. There are so many things to get enraged about it’s hard to know where to begin.

See also: Gillard Should Come Clean on School Reports

Secrecy and avoidance of public debate are characteristic of how Klein has implemented change in New York’s schools. There too, teacher and parent organisations were excluded from the process.

Gillard Should Repudiate Klein’s Skulduggery

Gillard should know that Klein is widely derided by educationalists, teachers and parents in New York. She should also know that Klein’s response to their criticisms involves much skulduggery which she should repudiate.

School Accountability Discredited in New York

Conservative educationalist, Diane Ravitch, is at it again – highlighting the failures of the New York City school accountability model which Julia Gillard wants to emulate in Australia.

More Questions Raised About the New York School Reporting Model

More questions are being raised about the validity and reliability of the New York system of reporting school performance, which Julia Gillard wants to introduce in Australian schools.

Charter School Report Cards: Apples, Oranges, and DNA

Yesterday’s glowing announcement of the “pilot test” release of School Progress Reports for 20 charter schools should hardly come as a surprise to anyone who has followed the DOE’s steady, five-year march toward privatization. “Charter Schools Outshine Others as They Receive Their First Report Cards,” hails Jennifer Medina’s report in the New York Times. Medina quotes the DOE’s Michael Duffy that, “As a group, they skew higher,” as he simultaneously took a backhanded slap at public schools by stating that accountability in charter schools is “baked into their DNA.”

The DOE’s pilot test Progress Reports rated 20 schools – eleven elementary or K-8 schools and nine middle schools. One school, Opportunity Charter, originally received an F, but the DOE has since retracted their Progress Report grade. They assigned letter grades to thirteen of the remaining schools: five A’s (middle schools all), five B’s (elementary or K-8 all), two C’s, and one F. Of the remaining six, their total scores as computed from the reported components (School Environment, Student Performance, Student Progress, and Closing the Achievement Gap (Additional Credit) suggest approximate letter grades of two more A’s, one B, one C, and two D’s. Overall, roughly 68% A’s and B’s, 16% C’s, and 16% D’s or F’s.

Leaving aside bias factors in the DOE’s demonstrated predisposition toward charter schools and the lack of class size data that might well correlate with and explain some of the skewing, a slightly deeper look at the DOE’s Progress Report data reveals several interesting facts.

-- School Environment comprises 15% of each school’s score, but for the charter schools, this measure consists entirely of Attendance. As a result, in seven of the 20 piloted schools, attendance accounted for over 20% of their cumulative scores, and in three of those schools, it accounted for well over 30%.

-- In seven of the nine middle schools, their peer indexes (the average of their students’ Grade 4 NYS ELA and Math exam scores) exceeded 3.0, and an eighth school came in at 2.97 (the lowest, Opportunity Charter, had a peer index of 2.54 and also happened to rate an F grade before it was retracted). Overall, these nine charter middle schools had an average peer index of over 3.13 (3.21 if we also drop Opportunity Charter out as the DOE did), suggesting that these schools succeed in part by siphoning away stronger students from other schools.

-- The four middle schools that rated letter grades of A had the following peer indexes: 3.45, 3.39, 3.36, and 3.34 (the fifth was 2.97). By comparison, the Clinton, Wagner, and Sun Yat-Sen middle schools in Manhattan had peer indexes of 3.53, 3.58, and 3.30, respectively, and the five middle school only (Grades 6 – 8) schools in District 4 had peer indexes of 2.82, 2.86, 2.87, 3.20 (Isaac Newton), and 3.65 (Manhattan East, on East 100th Street).

-- In what seems a near statistical miracle, the three KIPP schools each reported Attendance rates of 98.0%.

Given the obvious selectivity of the charter middle schools, the lack of intake information about charter elementary and K-8 schools, the absence of reporting on many charter schools, and the inaccessibility of charter school class size data, it seems disingenuous at best for the DOE to claim that charters “skew higher” from a sample of just 13 schools that the DOE itself chose for letter grade assignment.

All this is not to say that charter schools do not do good work in motivating and educating their students. Just once, however, it would be nice if the DOE would release enough data in format and content for independent voices to evaluate its public relations claims and provide a real basis for comparing apples to apples. Sadly, nothing in the past several years suggests that all those well-paid DOE executives recognize the differences between apples and oranges. Apparently, it’s all just fruit salad to them.

Update: here is an excel file with the charter school progress reports and their components.

Tweed's consultation with parents, or lack thereof

Mike Meenan of NY1 had an excellent story last night about the Chancellor’s failure to consult with any Community Education Councils before closing schools in their districts – contrary to the language and the spirit of state law, which says the following:

"The chancellor shall consult with the affected community district education council before... substantially expanding or reducing such an existing school or program within a community district.”

Yet Klein never bothered to do so in this case – nor in previous years. James Liebman said that these decisions were made because of long-term educational failure over many years -- yet several of the schools being closed are in good standing with both the state and federal government.

Liebman also claimed in his testimony to the City Council that he had conferred with many experts, union officials, and other “educational and community leaders” who provided “valuable feedback” during the development of the school grading system. Several of those cited in his testimony contradicted him later that same the day, including Ernest Logan, the President of the the principal’s union, Amanda Gentile, a VP of the UFT, and Ann Cook from the Consortium for Performance Standards.

His testimony also included the statement that his office had “directly reached over 20,000 parents in conversation about explanation of the Progress report and associated accountability tools.”

Yet according to an article in today's NY Sun, these sessions included visits to Laundromats and trips on the subway! It turns out that DOE employees had handed out flyers about the parent surveys at many of these places -- but there's no evidence that any real “consultations” or “conversations” about the school progress reports took place, and it’s not clear how many of the 20,000 people they supposedly contacted were actually parents.

Check out this list, provided by the DOE, which include not only visits to laundromats, but also a dentist office (each time supposedly involving 50 parents), and assorted trips on the #1,#2, #3, #4 trains – and just so the IND didn’t feel left out, also the E, F, G, V, and the R trains.

Many of these direct “conversations” appear to have occurred during trainings of parent coordinators – who after all, are not necessarily parents themselves.

In short, another incredible product of the unaccountable Accountability office at Tweed.

Voices in opposition to the school grading system grow louder

Diane Ravitch has an oped in today's NY Sun about the new school grades:

Is the grading system accurate and reliable? Did the grading system identify the worst schools? Is the closure of the lowest-performing schools likely to improve public education? Could the Department have taken other actions that might have been more effective than closing schools?

The answers to all of these questions, she suggests, is no. Diane also provides an important critique of the whole notion that simply closing schools is the best way to make significant progress:

Nor is it enough to turn out the lights. Schools are not a franchise operation. They are deeply embedded community institutions. They should be improved with additional resources, smaller classes, and additional training for educators. The starting point in reforming schools is to have a valid evaluation system that correctly identifies the schools that need extra help. It may not be easy to transform the schools that are in trouble, but if we want a good public education system, there really is no alternative.

Indeed, this is an essential element of the school reform process for which Tweed no longer feels accountable -- their responsibility to provide the support and resources schools need to improve.

See the show on PBS about the NYC school grading controversy, including parents and principals at some of the schools that got low marks, and one that got high marks, talking about the meaning and impact of these grades. The show also includes an interview with the Chancellor, in which he attempts to explains the "F" that PS 35, the Staten Island neighborhood school received, despite having 98% of students at grade level in math, by comparing it unfavorably to Anderson School – a highly selective gifted and talented school.

The interviewer, Rafael Pi Roman points out that William Sanders, the father of value-added accountability systems, told him that the sort of one year’s test score gains that the NYC grades are based upon are not meaningful. Klein responds that nevertheless, the school grade is a positive motivational factor in getting schools to work harder on improving test scores.

You can also listen to audio clips from the City Council hearings on the school grades from December 10, now posted on You Tube:

Public Advocate Betsy Gotbaum , who says out that closing schools unilaterally, as the Chancellor has done, without first consulting Community Education Councils is potentially illegal.

City Council Education Chair Robert Jackson (Part 1, Part 2, and Part 3), who aggressively questions James Liebman on many issues, including whether the DOE reached out to parents sufficiently.

Council Member Lew Fidler of Brooklyn, who flunks the school grades for their lack of transparency. (Part 1 and Part 2.)

And Council Member John Liu , who is masterful in showing that these grades are derived primarily from the results of only two tests -- though Liebman keeps trying to argue that these are really "multiple assessments" given out over "multiple days." (Part 1 and Part 2.)

Finally, watch the Channel 2 news segment featuring the hearings and showing Liebman fleeing from parents, now also posted on YouTube.

UPDATE: see also this article in City Limits: PARENTS, COUNCIL STILL ANGRY ABOUT SCHOOL GRADES

James Liebman on the run

On Monday, at the City Council hearings on the school grades, James Liebman, the chief accountability czar and former law professor, faced fierce criticism from Council Members. No wonder; his testimony was evasive, full of misleading statements and outright errors.

Liebman went on to make many questionable statements, among them, that a school at which "hundreds of children on average lost 10 percent of a proficient level in a year almost certainly has a significant problem."

Instead, experts say that one year's gain or losses in test scores at the school level is 34-80% random, and unrelated to the amount of learning taking place.

Liebman also claimed that factors related to overcrowding and class size were taken into account when devising the grades, when they clearly weren't.

In his testimony and power point, he claimed that he had consulted with many groups and experts, including the United Federation of Teachers, the Council of Supervisors and Administrators (the Principal's Union), CPAC, Community Education Councils and the NY Performance Standards Consortium in devising these grades.

Ann Cook, the co-chair of the Consortium, later testified to the fact that this was untrue. Her group had asked for and gotten a meeting about the interim assessments, but the topic of the school grades never even came up.

Ernest Logan, President of the CSA also denied that he had ever been consulted, and laughed when Jackson asked him this question. (See this letter from Logan to the Chancellor, about the many flaws in the school grades.) The UFT VP, Aminda Gentile, said they had “conversations” with DOE about the school grades, but there was no consultation.

Betsy Gotbaum, the Public Advocate, also criticized the unreliability of the school grades, and said that the Chancellor's decision to close schools without consulting first with Community Education Councils is against the law. She cited the state law, (2590-h) , which says that the Chancellor has the authority to:

Establish, control and operate new schools or programs…or…discontinue any such schools and programs as he or she may determine; provided however, that the chancellor shall consult with the affected Community District Education Council before substantially expanding or reducing such an existing school or program within a community district. (The law is posted here.)

Yet, she added, this has not happened in this case. "And the truth is, I can't think of an example where it has happened."

When asked by the chair of the Education committee, Robert Jackson, Liebman admitted that CECs had not been consulted before the announcement to close schools. Instead, they had been consulted afterwards, "entirely consistent with the process that has applied for the last several years."

Did he believe that parents should be consulted? Liebman said that the process that was used "was sufficient and adequate and very comprehensive."

Jackson said this response was "totally unacceptable", and if this was the direction the chancellor is going, he is in "big trouble." Liebman also claimed that the method he used was very "transparent" with very "clear rules" and that the results of the Quinnipiac polls showed that parents understood the methods used. (!!)

Liebman kept returning to the results of this poll in his defense, though it turns out that only 143 public school parents were polled.

City Council Member Lou Fidler was concerned that stigmatizing schools with failing grades will likely accelerate the decline of these schools, rather than helping them improve. Melinda Katz said it best: In her 14 years as an elected official, she’s never seen an agency so sure they’re right, when all the parents she has spoken to believe they’re wrong.

John Liu was very effective, asking Liebman repeatedly if the 85% of each school's grade was not just based upon a single measure, the results of a test taken once a year. Liebman kept on evading the issue, saying these grades were not based on one measure but actually "many measures" from a "series of assessments" that take place over a "series of daysm" and that each assessment "cuts across many hundreds of different items, and many skill areas." Liu pointed out the fact that its still only one test!

Finally, Liebman blurted out, "Life is one test" and everyone booed. Liu concluded that not only was Liebman trying to obfuscate, but that that his entire testimony was an obfuscation.

At the end of Liebman's three-hour testimony, the Chair, Robert Jackson, politely requested that he step outside the hearing room to receive petitions from Time Out from Testing and Class Size Matters, signed by nearly 7,000 parents, calling for a halt to the school grades. (Thanks so much to those of you who signed.)

In preparation, we filed out in an orderly fashion, (see above photo from the NY Times) but rather than have to confront us directly, Liebman slipped out a side door, out the back exit of City Hall, and ran away from us like a thief in the night, as we tried to catch up. He then entered the private gates to Tweed, but refused to let us in.

Liebman’s flight from parents was captured on video on many of the nightly news shows. As Lisa Donlan was quoted as saying in the Daily News, all this is symbolic of DOE’s arrogant and dismissive attitude. "He wouldn't even stay to hear our questions ... after we sat for three hours and listened to his testimony."

Here is an excerpt from today’s Times story, “Defending School Report Cards, Over a Chorus of Boos”:

“Mr. Liebman, whose title is chief accountability officer of the Education Department, ducked out a side door, leaving parents to chase him out the back of City Hall to behind the Education Department’s headquarters at Tweed Courthouse.

There, several education officials ran in circles for several minutes to avoid Jane Hirschmann, the director of Time Out From Testing, an advocacy group, as well as parents and reporters.”

Later in a phone interview, Liebman claimed to Times reporter that “he had not deliberately avoided the parents.” This claim is about as trustworthy as the school grades themselves.

See also article in Daily News, Escape from NY parents, the CBS newsclip here and NY1 here.

The CBS story repeats the erroneous statement that Liebman has met with Time out from Testing “many times”; in fact, according to Jane Hirschmann, head of the group, he has refused to ever meet with them.

I also gave testimony posted here about how unfair, inaccurate and destructive these school grades are, and entered into the record the comments criticizing the school grades from many of you, including parents, teachers, and at least one retired principal, that were posted online at our petition.

Update: Erin Einhorn of the NY Daily News pointed out today in Only in NY schools can get an 'A' & 'F' that of the 26 SURR schools on the state failing list, nine got As or Bs.

"The city can do whatever they want to do, but at the end of the day, I think the public deserves better," said Merryl Tisch, the vice chancellor of the state Board of Regents and a longtime supporter of Mayor Bloomberg.

Sign our petition against the new school grades and for smaller classes today!

Class Size Matters has a new online petition, asking that the school grading system be terminated, and all the hundreds of millions of dollars being spent on the DOE’s ill-conceived overemphasis on standardized testing and test scores be redirected towards reducing class size and expanding the capital plan.

Our online petition is posted here. Please go sign it now! We’d like as many signatures as possible by the City Council hearings on the new school grades, scheduled for Dec. 10.

(If you’d like more info on why the grading system is unfair, arbitrary and will hurt rather than help our schools, check out my Daily news oped, Why Parents and teachers should reject the new school grading system, posted here (as a word doc) or on the News website here. Also: Ten reasons to distrust the new accountability system and "Negative learning" and statistical malpractice at the Panel on Educational Policy for more.)

North Carolina, a state that over ten years ago pioneered the move towards more testing, is about to reverse course. See N. Carolina begins to turn away from testing.

“We're testing more but we're not seeing the results," said Sam Houston, the commission's chairman. "We're not seeing graduation rates increasing. We're not seeing remediation rates decreasing. Somewhere along the way testing isn't aligning with excellence."

Here in NYC, rather than be satisfied with all the existing state tests, and the new interim assessments now being given five to six times each year, the administration want to start adding new standardized tests in science and perhaps other subjects next year, as well as yet additional tests for grades K-2.

Rather than go down that same road that North Carolina is now rejecting, Tweed needs to be woken up from its delusion that more testing means more learning, as quickly as possible, before things get even worse.

Please sign our petition today! The full text is below.

We are vehemently opposed to the new DOE school grading system. These grades are unfair, simplistic and arbitrary, are based on statistically unreliable measures, and will hurt rather than help our schools.

By awarding each school a grade from A to F, the progress report trivializes the complexity of teaching, and will drive schools towards even more test prep and less learning, as well as further deprive our children of art, music, and physical education.

We demand that the energy, focus, personnel and millions of dollars that have been spent on devising this system, as well as the entire data collection system known as ARIS, interim assessments, financial incentives for high test scores, and “data inquiry teams” in the name of “differentiated instruction” be instead invested in reducing class size and expanding the capital plan, so that all NYC children can be provided with smaller classes and an equitable and adequate chance to learn.

"Negative learning" and statistical malpractice at the Panel on Educational Policy

At last week’s meeting of the Panel on Education Policy at Tweed, Jim Liebman’s performance in attempting to defend the indefensible – the school grading system that he designed -- was breathtaking in its ignorance.

Liebman, the current DOE accountability “czar,” is a former criminal attorney, currently on leave from the Columbia law school, with no training or experience in education policy, statistics or testing, and yet the entire educational focus of the DOE is now based upon his faulty theories and expensive initiatives, including the $80 million supercomputer called ARIS, assigning letter grades to all schools primarily on the basis of one year’s worth of test scores, devoting millions of more dollars and hours of precious classroom time to interim standardized assessments, and the creation of “data inquiry teams” in all schools – all in the effort to “differentiate instruction” which in the end will be impossible without smaller classes.

At the PEP meeting, in order to justify the school grading system, he fastened on the “F” that PS 35 in Staten Island received, a school in which 98% of its students are on grade level in math, and 86% in ELA. Why did this exemplary school receive an “F”? Because last year, only 35% of its students improved their scores over the year before in reading, and only 23% in math – though research shows that a large part of annual variations in test scores are based on chance alone and are statistically unreliable. (For more on this, see my Daily News oped and a previous posting, Ten reasons to distrust the new accountability system.)

During the discussion, Liebman compared PS 35 to one of its “peer” schools – the Anderson school, a citywide Gifted and Talented school that accepts students on the basis of their high IQ and high test scores. When Patrick Sullivan pointed out the unfairness of comparing PS 35 to a selective school like Anderson, Liebman said it didn’t matter how the kids got there, they should all make the same annual gains. He failed to mention, however, that elementary schools are grouped with other schools according to only the roughest measures of demography –and that no statistician would compare the performance of a school that selects its students on the basis of test scores with a neighborhood school, like PS 35, that has to admit every child in its zone.

There was an abundance of statistical malpractice on display that night -- between Liebman’s presentation and the talk given by the DOE testing “expert”, Jennifer Bell-Elwanger, who tried to convince the panel that the city’s lack of significant progress on the NAEPs since 2003 was indeed real progress. Both of these individuals would have flunked an elementary course in statistics if they had tried to make these arguments in a college exam.

When asked wouldn’t it better to have separate grades for achievement and progress, rather than collapse all these categories into one grade, even if he were convinced that the lack of one year’s progress in test scores was significant (which it isn’t) Liebman replied that the good thing about giving a single grade is that it gets people’s attention (or something like that.) One could say the same about threatening to cut off the hands of someone accused of theft, or even capital punishment, which doesn’t mean it’s a remotely fair practice or even useful.

More recently, in response to questions about class size from parents in Manhattan and Queens, Liebman has insisted that the reason the DOE refuses to reduce class size is that classes would have to shrink to below 15 students to improve instruction and/or student achievement. In other words, lowering class size from 30 to 20 would make absolutely no difference.

Not only is such a statement absurd to anyone who has actually spent any time teaching in the public schools or observing classrooms, it is completely unsupported by research. Instead, it is simply another lame excuse that opponents of reducing class size like to throw up as a smokescreen in order to discourage such efforts.

Here is a comment sent to me from Chuck Achilles, a principal investigator of the famed STAR experiment in Tennessee and a professor of at Eastern Michigan University and at Seton Hall University. Chuck is also one of the premier class size researchers in the world:

“Hi Leonie:

I thought that the “below 15” idea (archaic) had faded. Anyone who says that is uninformed and ought to be asked (challenged) publicly to defend the assertion. It came once from one meta-analysis (Glass & Smith, 1988) that was very limited in its n of observations (77, of which some were for physical skills like hitting a tennis ball against a wall.) Just in STAR, we had more than 1300 observations in the range of 12-28 students. We typically analyzed reading outcomes, but sometimes we did math (giving us 2600 comparisons) and could have used other academic (test) outcomes… I’ve faxed some pages to show the linear effect: About a correlation of -.35 for each student added to a class. Because STAR used the class average as the unit of analysis, this means (approximately) the addition of each student to a class in the n=12-28 range reduces the class average score (about .1 of a month per year.) Later analyses show that it is cumulative.

Chuck A.”

Here is a fact sheet with numerous citations, showing there is no threshold in terms of reducing class size; and that the increase in achievement in relation to the decrease in class size is roughly linear.

Liebman reminds me of a phenomenon called “negative learning” ---in layman’s terms, a little learning is a dangerous thing. One would think that someone who got his reputation by writing about the high error rate in capital punishment would have a little humility and understand the possibility of human fallibility in making absolute judgments, but no such luck.

What grade would you give Joel Klein?

Yesterday, the DOE let the legal deadline for class size reporting lapse. Not entirely surprising, given the fact that they have refused to tell us a single school where class sizes have supposedly been reduced this year. Their official excuse is that they are still working on cleaning up the data.

Interesting that they felt they had enough accurate data to assign grades to every school; but they are still working on providing accurate class size information – as required by a law passed by the City Council nearly two years ago.

Tweed talks a lot about data-driven instruction, and is spending millions of dollars putting together data inquiry teams in every school to pump up test scores. But when it comes to the most basic, most critical data of all – how large are the classes our kids are sitting in – they remain officially at sea. Or else they are simply trying to hide the truth. Which is worse, I can’t say.

• The NAEP (national) test scores were released yesterday. These are the most reliable indicators we have for spotting long-term trends. NYC made no significant improvements in three out of the four categories since 2003, when the administration put into effect its Children First reforms.

No improvement in 4th or 8th grade reading, no improvement in 8th grade math. Only in 4th grade math have there been increases. Below is an article from today’s Times; and here is Diane Ravitch’s summary on our blog.

Also see the NY Post story, in which the Chancellor is quoted as saying that they are going to launch another “comprehensive study” to see what works in terms of raising achievement in middle schools. Parents and teachers could tell him plenty about how our middle school class sizes are not conducive to learning, in which 79% of students are in classes of 25 or more, and 40% in classes of 30 or more, far above the state, national and OECD averages, but I doubt he’d be interested in listening.

Clearly, the fact that this fall, NYC received the Broad award for most improved school district in the country was undeserved. As I said at the time, it was based largely on manipulated graduation data and inflated state test scores.

What grade would you give Joel Klein and this administration, not only for failing to improve student achievement in three of four categories, but in other areas? Say, transparency, communication, fairness? Or pick any category you like.

Our blog has a new poll (see right hand corner.) Please vote, and then put your comments on right here as to why you give the administration the grade you did.

NAEP scores released: mostly bad news for NYC by Diane Ravitch

The NAEP scores for 11 big cities were released on November 15 and there was a little good news for New York City and quite a lot of bad news.

The good news is that fourth grade students made significant gains in mathematics on the 2007 NAEP test. These gains were spread across all groups of students of every race and ethnicity.

The bad news is that NAEP showed no significant gains between 2003 and 2007 for New York City students in fourth grade reading, eighth grade reading, or eighth grade mathematics. In these areas, there were no gains for students of any race or ethnicity.

NAEP has been releasing test results for certain big cities since 2002. The big-city testing program started at the request of the Council for Great City Schools, which wanted a clear measure of how the urban districts were doing compared to states and the nation.

The baseline for Children First begins with 2003 data, as the test was given in the spring of 2003, before the reforms were implemented in September 2003. There was a significant increase in scores from 2002-2003, but that was the year prior to the implementation of Children First.

Here is what the NAEP reports (http://nationsreportcard.gov) say:

• In fourth grade reading: the overall score was unchanged from 2003. The only significant gain was made between 2002 and 2003, before the Children First regime was launched.

For NYC fourth-graders in 2007, lower-income students showed "no significant change in the average score compared to 2003 and 2005."

In addition, there was "no significant change in the average scores for White, Hispanic, and Asian/Pacific Islander students compared to all previous assessments"; for Black students, "there was no significant change compared to 2003 and 2005."

The report said that there was "no significant change" in the percentage of students at or above Basic since 2003, and "no significant change in the percentage at or above Proficient compared to all previous assessments." (p. 50).

• For NYC eighth-graders in reading, "The overall score was not significantly different from 2003 and 2005."Results for lower-income students showed "no significant difference in the average score compared to 2003 and 2005."

Results for racial/ethnic groups showed "no significant change in the average scores for White, Black, Hispanic, and Asian/Pacific Islander students compared to 2003 and 2005."

Achievement-level results showed "no significant change in the percentage at or above Basic compared to 2003 and 2005" and "no significant change in the percentage at or above Proficient compared to 2003 and 2005." (p. 51)

• Fourth-grade mathematics was the one bright spot for NYC students. The overall score was higher in 2007 than in 2003 and 2005. Lower-income students showed a "higher average score" than in either 2003 or 2005. (p.50)

Results for racial/ethnic groups showed "higher average scores for White, Black, Hispanic, and Asian/Pacific Islander students compared to 2003 but no significant change compared to 2005."

Achievement level results showed "an increase in the percentage at or above Basic compared to 2003 and 2005, and an increase in the percentage at or above Proficient compared to 2003 and 2005."

• In eighth-grade mathematics, "the overall score was not significantly different from 2003 and 2005." Results for lower-income students showed a higher average score compared to 2003, but "no significant change compared to 2005." (p. 51)

Results for racial/ethnic groups showed: "No significant change in the average scores for White, Black, Hispanic, and Asian/Pacific Islander students compared to 2003 and 2005."
There was "no significant change in the percentage at or above Basic compared to 2003 and 2005."

There was "no signficant change in the percentage at or above Proficient compared to 2003 and 2005."

The NAEP scores may be seen as a report card for the Department of Education for its reforms from 2003 to 2007. In one area, fourth grade mathematics, the DOE gets an A. On three other areas -- fourth grade reading, eighth grade reading, and eighth grade mathematics--the DOE showed no progress at all.

Not a good report card, though I will leave it to others to assign a letter grade.

NAEP is widely considered the gold standard of educational testing. The federal government has invested heavily to ensure its reliability and validity.

-- Diane Ravitch