Nightline on test prep & the gifted exams: more "choices" for parents or magnifying social inequities?

The results of the Gifted and Talented exams are in, and according to the NY Times, more than half of the children tested in wealthier districts like District 2 and District 3 were found to be "gifted", while only six children made the grade in District 7 in the South Bronx.  Why the disparity?

Are these tests merely a way of sorting children by race and class, as Debbie Meier pointed out in 2007, when Klein first proposed to base all admissions to gifted programs on the basis of high stakes exams, or do the results really reflect children's inherent abilities?  And does the proliferation of G and T programs across the city help or hinder the goal of equity and systemic reform?

We have written often about the severe problems with the way this program has been implemented in NYC, including how we believe it works to magnify inequities, as well as about the continued pro-administration bias of reporting on this issue here.

Check out these segments from a recent Nightline investigation, aired April 14, including interviews with anxious NYC parents enrolling their four-year-olds in arduous test prep programs, because they believe that if they can get their kids into these programs they will be set for life, as well as an interview with Chancellor Walcott, who expresses no concern that some  parents may be paying four or five thousand dollars to prep their children for the test, because he says this gives them more "choices." Walcott also evinces a surprising lack of skepticism, given the extreme racial disparity in the results, that these exams  test actual giftedness rather than economic and social privilege.

A warning to Australians: don't let your kids suffer, as have ours!

This week, Joel Klein is visiting Australia to promote his educational policies and perspective. To alert Aussies to the disinformation campaign he will likely be spreading, we have deconstructed a recent interview on Australian TV with Klein, carried out by journalist Kerry O'Brian, and aired last month on Australian TV.

[The full video of the Klein interview is here and the transcript here.]

Question: "You can't have a system corrupted or distorted by a principal under pressure or a principal that wants to boost his or her status by encouraging teachers at the school to go soft on the students in tests to help them cheat in effect?"

JOEL KLEIN: "It's happened on a handful of occasions since I’ve been here and we’ve terminated principals, terminated a couple of teachers at one point, but I don't think systematically that's a powerful explanation.

And even before our system there was a Federal system here in the US under "no child left behind." So schools are under a certain amount of pressure, but I don't think somebody is going to imperil their livelihood by taking the risk of trying to have students cheat on exams."

In reality, there is now rampant test prep in NYC schools – as well as many serious allegations of cheating – few of which are ever followed up by this administration, even when the allegations are based on reliable evidence from people on the ground. In fact, many principals have been promoted and given bonuses whose schools have seen large and unexplained boosts in test scores. See for example this article, detailing allegations about cheating at one school, where test scores jumped 30 points in one year, where Chancellor Joel Klein spoke at the school's graduation while wearing a "Best School in the Universe" T-shirt.

In fact, Klein doesn’t appear to care what’s causing test scores to rise, as long as they do. And as pointed out by independent experts like Diane Ravitch, as well as in the NY Times and elsewhere, test scores on state exams have not been matched by NYC's results on the more reliable national assessments called the NAEPs. Indeed, an analysis of NAEP scores between 2003-2007 reveals that NYC came in 10^th out of 11 urban school districts across the nation in changes in test score changes over the course of this administration.

JOEL KLEIN: "The class sizes have reduced…”

Actually, Joel Klein has stubbornly refused to reduce class size, the top priority of NYC parents, even when it means ignoring the law. As a result, our class sizes remain the largest in the state and among the largest in the nation, despite hundreds of millions of dollars allocated by the state to reduce class size.

In a 2006 audit, the State Comptroller found that only 20 additional classes were formed in NYC schools over the baseline figure, despite $89 million in annual state funds specifically allocated to reduce class size. The audit concluded that “while the [NYC] DoE was receiving State funding… it was reducing its own support for early grade class size reduction and using it for other purposes.” The NY State Comptroller proposed a number of recommendations to improve compliance with the law– all of which the Chancellor rejected.

Then again, this past September, the NY State Education Department found that despite commitments by the city to use millions of dollars in new state aid to reduce class size, instead “53.9% of New York City schools reported that either class size or pupil-to-teacher ratio increased in 2007-08.”

Accordingly, over 80% of NYC parents say that there has been no improvement in class size under this administration, and 86% of NYC principals say that they are unable to provide a quality education to their students because of excessive class sizes at their schools.

JOEL KLEIN: "The President of our [teachers] union, who's also the President of the National Union, she said on the first day of school this year in September that she had been traveling around the US and no place, in any urban area, was comparable to the work that we're doing. In fact he word she used was that New York City had become a beacon."

Among NYC teachers, there has been overwhelming disapproval of the leadership and policies of Joel Klein. Last year, the NYC teacher’s union released its own survey, showing that 85% of its members believe that Klein has refused to provide the support and resources they need to succeed; 85% disagreed with his emphasis on high-stakes testing; and 83% said that he had put other interests above the learning needs of the children.

According to Klein’s top spokesperson, only two out of twenty of the top administrators at the Department of Education are long-term educators; instead, he has largely staffed the Department with corporate consultants and attorneys.

In the past few weeks, thousands of teachers in NYC and elsewhere have signed petitions, warning President-elect Obama not to appoint Joel Klein to his administration, because of his misguided policies.

Check out this petition, which has been signed by nearly 4200 educators in NYC and nationwide:

"The NYC Department of Education under Joel Klein has been run like a ruthless dictatorship -- with no input from parents or educations. Teachers have not been respected, consulted, nor listened to. And little thought has been devoted to how the policies he has imposed on our schools have been destructive to the children and their futures...While focusing on test scores, he has consistently ignored the crisis of overcrowding in NY schools. Thousands of children are being given special services in hallways or in closets."

See also this petition, signed by more than 3600 teachers and academic experts nationwide, in opposition to Joel Klein's "vision of privatized, corporatized, and anti-democratic schools."

QUESTION: "How do you answer the critics who say your school statistics are flawed because of wild statistical fluctuations or results from year to year, which suggest fundamental flaws?"

JOEL KLEIN: "I don't think that's a fair criticism. I've actually studied the statistics. There are some year to year fluctuations that are significant, but that's because schools that weren't making progress refocused and decided that the risk to them was they were gonna close.

"And so they decided that they would focus much more effectively on promoting student performance. But all of these things will come out over the years, meaning this is not a one year or two year experiment.”

Some year-to-year fluctuations? School grades have widely varied between one year to the next, which is not surprising considering they are primarily based on one year's worth of test score gains or losses, which experts have shown to be 30-80% random.

See this analysis by Daniel Koretz, an expert on testing and statistics from Harvard, pointing out that three quarters of NYC schools that received an “F” last year received an “A’” or “B” this year. As Koretz writes:

“It strains credulity to believe that if these schools were really “failing” last year, three-fourths of them improved so markedly in a mere 12 months that they deserve grades of A or B ...This instability is sampling error and measurement error at work. It does not make sense for parents to choose schools, or for policymakers to praise or berate schools, for a rating that is so strongly influenced by error.”

Aaron Pallas and Jennifer Jennings of Columbia University have concluded that “The progress measure…is a fruitless exercise in measuring error rather than the value that schools themselves add to students.”

As another of their summaries suggest, “a Monkey [could] Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education."

In short, NYC parents and teachers have this advice for Australians: Do not be fooled by either Klein or his claims.

Achievement has not significantly improved under his leadership. Instead, his administration has consistently misused millions of dollars meant to reduce class size that would have provided children with a better chance to learn, while spending millions more in taxpayer funds on expanding the bureaucracy with high-paid corporate consultants, no-bid contracts, and more testing in our schools.

Along the way, he has stubbornly refused to listen to parents, teachers and academic experts who have pointed out the destructive impact of his policies.

Take heed -- and reject these policies that will cause your kids to needlessly suffer -- as have ours.

More statistical malpractice from Tweed: Joel Klein and his claims do not measure up

Check out Elizabeth Green’s article today in the NY Sun, in which she asked three academics to scrutinize the validity of the administration’s claim of narrowing the achievement gap between ethnic and racial groups.

See, for example, Bloomberg’s recent testimony before Congress, in which he said that “over the past six years, we’ve done everything possible to narrow the achievement gap – and we have. In some cases, we’ve reduced it by half.”

Yet the evidence for this is weak to non-existent. On the national tests called the NAEPs, there has been no narrowing of the achievement gap in any area since the Bloomberg/Klein reforms were instituted:

An analysis by the National Center for Education Statistics, the research arm of the federal Education Department, concludes that no achievement gaps have narrowed at all in New York City between 2003 and 2007. The only gap that moved in any significant direction is the one between poor students and the rest of the population, which widened slightly, that analysis said. The National Center for Education Statistics also concludes that upward trends in the reading scores of black and Hispanic fourth-graders lauded by Mr. Klein are not statistically significant.

In the article, Joel Klein reveals his statistical illiteracy:

“Those are just confidence levels. Nobody is saying this is a science," Mr. Klein said. He added: "If three points is flat, and four points is statistically significant, then what you're doing is, you're playing something of a game."

Chief press officer David Cantor called the memo from NCES "a politicized gloss.”

Instead, it is the DOE who insists on playing games – and politicizing the issue, by continuing to slander experts as somehow biased when they provide objective evidence that the non-stop PR spin issuing from Tweed has no basis in reality.

This is hardly the first time the DOE has revealed such statistical malpractice. Jim Liebman, law professor and head of the DOE accountability office, is a repeat offender. I recall one episode in particular when Liebman, testifying before the City Council, insisted that he wasn't basing school grades primarily on the results of a few tests, since each test was really "multiple assessments" given out over "multiple days," resulting in "multiple measures" of proficiency.

Another instance of this was the presentation of Jennifer Bell-Elwanger, head of testing for DOE, who was giving a power point presentation to the Panel for Educational Policy in late November, following the release of the NAEP results. She continually pointed out gains that, according to the NCES, were not statistically significant. Patrick Sullivan, Manhattan rep to the PEP and fellow blogger here, who seems to understand data better than anyone currently employed by the DOE, questioned her closely, saying, "But by definition these are insignificant gains, no?" Which she, of course, fervently denied.

Indeed, the memo from the NCES was prepared in response to a highly misleading email that Klein sent to nearly every NYC resident after NAEP results were first reported. In his email, he falsely claimed that the results showed “good progress that is consistent with the overall picture” and said that the NAEP showed a narrowing of the gap in nearly all areas, when the data itself revealed quite the opposite.

According to NYC’s results on the state exams, the situation is more complicated. The achievement gap is narrowing in some areas when one looks at “proficiency” levels, that is whether a student is at a level 1, 2, 3, etc., but not in terms of the actual scale scores.

Some testing experts consider proficiency levels less meaningful than scale scores, as they can be arbitrary, subjective and easy to manipulate. Daniel Koretz, a professor at Harvard and a national expert on testing, has just published the must-read book of the summer, Measuring Up: What Educational Testing Really Tells Us. Here is what Koretz has to say about proficiency levels:

….the percents deemed proficient [on state tests] are largely unrelated to states’ actual levels of student achievement…are also often inconsistent across grades or among subjects in a grade....[This system] obscures a great deal of information [because] of the coarseness of the resulting scale.

Even more importantly, why should we trust the state scores at all, when we know that the gains that they purport to show are not reflected in more trustworthy measures like the NAEPs?

The best thing about the Koretz’ book is his lucid explanation of why “test score inflation” inevitably occurs when you attach high-stakes to exams, and how this undermines the integrity and validity of the results; this has increasingly been the case throughout the nation as a result of NCLB, but even more here in NYC, as a result of the increasingly high-stakes policies of the Bloomberg/Klein administration.

Steve Koss has written about this eloquently on our blog, in relation to Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

I have tried to explain this phenomenon to many elected officials, staff, and reporters over the years, apparently with little success. I certainly don’t know a single NYC media outlet that has ever mentioned it, though Campbell’s Law was cited in some recent letters to the NY Times in response to the administration’s experiment to pay students for high scores.

I recall a lengthy discussion of this issue several years back with NY Times reporter David Herszenhorn while he was still on the education beat, to explain my opposition to the Mayor’s newly-proposed 3rd grade retention policy. One of the reasons I so vociferously opposed this policy, and still do, was not just that it was unfair to the student to base such a life-altering decision on the basis of one single, fallible test score, with such large margins of error; and not just that retention has been shown to have a racially-disparate impact and hurt rather than help most low-performing students.

My opposition was also due to the fact that the more significant consequences are attached to any test, the less its results can be trusted as a reliable gauge of real learning.

Since then, of course, the administration has piled on more and more high-stakes consequences -- for students, teachers, and schools – by adding fifth and seventh grade retention, awarding principals, teachers and students monetary rewards for high scores, and threatening to close down schools if scores don’t improve fast enough. The scores themselves have been rendered entirely meaningless as a result, as excessive test prep, teaching to the test, cheating, and other strategies to “game” the system has totally overtaken our schools.

Here is what Koretz has to say about this:

"One might expect that with the huge increase in the amount of testing in recent years, we would know more…Ironically, the reverse is true. While we have far more data now than we did twenty of thirty years ago, we have fewer sources of data that we can trust. The reason is simple: the increasing in testing has been accompanied by a dramatic upsurge in the consequences attached to scores. This is turn has created incentives to take shortcuts --- various forms of inappropriate test preparation, including outright cheating – that can substantially inflate test scores, rending trends seriously misleading or even meaningless.”

To the administration, all this appears acceptable, because as long as test scores go up, this justifies their policies; in truth, they don’t seem to care if the increases are meaningful or not.

Their laissez-faire attitude is revealed by the total lack of interest evinced in following-up on even well-documented cases of cheating. (See for example this story in the NY Sun, which though it says the DOE is “investigating” this will likely lead nowhere, as such stories have in the past.)

This “anything goes” attitude is also reflected in Klein’s remarks in a recent interview in the NY Post:

Q: What about complaints about the report-card grades for schools?

A: The report cards were probably one of the noisy periods. But . . . I can't tell you how many principals said to me, 'You know, chancellor, I didn't get the right grade but I promise you I won't get the same one next year,' so I think that had a big impact.

Here, Klein implicitly acknowledged that even while principals do not accept the fairness of these grades, based primarily on one-year gains in scores, he is content as long as they guarantee that they will get these scores to rise in the future, by any means possible.