Wednesday, November 25, 2009

Yet Another Nail in the NYS Regents Exam Coffin

On November 19, the Office of the NY State Comptroller released a report of its findings from an audit of local district scoring of high school Regents exams. The results, while not surprising to those closest to high school education in NY State, was nevertheless stunning in its confirmation of just how badly skewed the entire Regents examination system has become. Equally startling was the mainstream press’s utter failure to note the findings, virtually all of which agreed with by the Regents themselves. (Note: only Maura Walz at the Gotham Schools website seems to have reported on this so far.)

The audit team randomly selected 200 NY State high schools and, using a team of experienced high school teachers, rescored nine non-multiple-choice questions on one 2005 subject area exam (identified only as Exam A) and thirteen non-multiple-choice questions on another 2005 subject area exam (Exam B). In total, the Review team rescored almost 2,400 Exam A papers and over 3,200 Exam B papers, looking only at questions where local school exam graders has discretion over how many points to award their students’ answers. Their findings in summary:

“…a significant tendency for local school districts to award full credit on questions requiring scorer judgment even when the exam answers were vague, incomplete, inaccurate, or insufficiently detailed.”

That sentence euphemistically recaps the much more disturbing details of their findings:

1. For Exam B, the locally reported total scores of the thirteen questions were higher than the Review Team’s re-scored total on 80% of the examination papers reviewed (totals were the same on 15% of the papers).

2. For Exam A, the locally reported total scores on the nine questions were higher than the Review Team’s re-scored total on 58% of the examination papers reviewed (totals were the same on 32% of the papers).

3. For Exam B, the locally reported total scores were at least three raw score points higher (or lower) on 34% of the exam papers re-scored by the Review Team. Three raw score points can easily scale to ten or more points on the student’s final, converted score. While not detailed in the report, one can well imagine that “bubble students’” tests were most prone to this higher level of score inflation to ensure they passed the raw score hurdle to receive a converted score of 65 or more.

4. For Exam A, the locally reported total scores were at least three raw score points higher (or lower) on 17% of the exam papers re-scored by the Review Team.

5. Exam B contained two five-point essay questions. The locally reported scores on these two questions were higher (or lower) than the Review Team’s re-scoring in 47% and 43%, respectively, of the exam papers reviewed.

6. Eighteen of 192 selected schools failed altogether to submit their requested Exam A papers, and 20 of 205 did not submit their Exam B papers. Even the Comptroller’s audit report suggests that these compliance failures might be attempts, as they put it, “to avoid scrutiny.”

7. Review of SED’s procedures for follow-up on privately-lodged complaints of scoring fraud or irregularity found no evidence that twelve of them had ever been investigated. Thus, even an honest teacher who whistle-blows on scoring fraud has virtually no guarantee that SED will conduct any investigation whatsoever. The door for cheating, fraud, or just looking the other way on exam grading is wide open and seemingly encouraged by SED’s actions and lack thereof.

Combine this pattern of fraudulently inflated grading with the persistent dumbing down of Regents exams and the concomitant lowering of the raw score needed for a passing scaled score grade, and the end result is an examination system that is utterly meaningless as a measure of knowledge or understanding. Even worse, as the Comptroller’s report makes clear, SED has failed completely to follow up on any of these issues, even having in hand another report from 2003/2004 detailing almost exactly the same problems.

What has become clear in the past five or six years (noticeably since the advent of NCLB), is that the NY State Regents examination system, once a moderately respectable measure of academic achievement, is now broken almost beyond repair. As long as the numbers are up, everyone rests easy; nobody seems to care that they are meaningless, as witnessed by the high levels of remediation required of first-year college student products of our state education system. As usual, the losers in this breakdown are the students and their sadly unaware parents.

It seems clear as well that the time has come for a major investigation and overhaul of SED and the Regents system. Governor Paterson and others in Albany, when will you wake up and start doing what’s right for the children in your state?


Leonie Haimson said...

Excellent summary, Steve. But two additional points:

1- the fact that HS teachers score the Regents exams of students at their own schools (and principals are allowed to change their scores) is a system ripe for abuse. Particularly now when the future employment of NYC HS principals and teachers depend upon improved graduation rates each year. Until a blind scoring system is implemented, with teachers randomly assigned student exams from throughout the state, the credibility of the results will be nil.

2. Paterson is not responsible for this mess. Instead, the Regents govern our schools, along with their hand-picked Commissioner, and the Democrats in the Assembly select the Regents.

So far, Merryl Tisch, the head of the Regents has been open in saying the standards on the state testes have to be "raised" -- statements that Bloomberg and Klein have recently echoed. Yet not one of them to my knowledge has admitted how much the standards have been lowered.

Unknown said...

I am so glad that an officially-sanctioned team of reviewers was able to demythologize the "tough new standards". I saw many barely intelligible papers "passed" but couldn't do anything about it when I was a high school teacher unless I wanted to get fired for breaching test security. George Morris

Anonymous said...

Anyone who has ever "followed Richard Mill's career knows that the state before NJ dismantled his garbage effrots right after he left, why? Because he has no standard he has smok mirrors and subtrafuge. This idiot, and I would say it to his face, had An Attorney replace hi ac Acting when he was ill. Who in the right mind has an Attorney run anything. Thery advise , they know law, they are not trained administrators, just see, NYC. Again, no one in their right mind permits an attorney to run things, they advise and if the do more than advise you are not a good/effective/knowledgable administrator. You are imprerssed by knowledge of law, I have worked extensively with good solid attorneys but I know their limits and their training, Chancellor Acting Comissioner, show you the idiot at the top.

Anonymous said...

"now broken almost beyond repair."

I don't think so. I think it is beyond repair.

Lousy tests.
Measuring? Hmm, no one seems to know.
Deform and distort instruction.
Encourage cheating.
Make cheating easy.

Honestly, why would anyone even try to fix these?


Lynne Bailey said...

@Leonie Haimson.

I particularly concur that there is way too much riding on test scores these days. Is it "too broke to fix?" What do you fix it with? Blind grading is one good idea. But imagine, too, that teachers may get paid a bonus if everyone passes, or grades increase from one year to the next. Our government is wrong to ask school systems to tie teaching salaries to tests passed.

It seems to me that the more we corporatize the education system, and neglect to acknowledge that there is more than one way to learn something, the less succesful our society is at educating our youth.[See the previous blog post!] Perhaps we should award parents those bonuses to encourage a family/ society ethic that values a good education, and that it starts at home.

While testing is an important part of overall assessment, it is outrageous that such huge amounts of funding and wealth distribution should rely soley on this kind of measurement.

Anonymous said...

To continue on scoring, not that long ago it was not such a big issue that schools scored their own exams. When I started teaching (97), I never heard a comment or a question.

But --

1. Regents became graduation exams (Albany)

2. and teachers and schools are being evaluated by regents scores. (Bloomberg, but other districts as well)


Anonymous said...

It is completely broken. I started teaching in 97 and I had 20 kids in a class. it was heaven. I could get to each and every one of them I could put them into small groups and work closely with one group while the others were busy at work. I could give them long writing projects to work on and sit and conference with them over their work. I now have 31 students in a room. It is chaos. Complete chaos. I'm told if I were a better teacher I would be able to engage each and every one of them in learning. I have kids with huge literacy problems and kids who need to be pushed further because they are right on grade level. I can accomplish very little because there are so many kids in the room. Ask any teacher. No one will tell you the schools have improved under our petty little dictator.