Tuesday, August 9, 2011

NYC test scores; small and unreliable gains

Yesterday, the state finally released school test scores; for NYC schools they are posted here.   Individual student test scores will only be made available August 17 – through the ARIS system, for which you will need your child’s OSIS number. 
Although the city showed gains of a few percentage points, the results were nothing to write home about: only 43.9 percent of city students in grades 3-8 met the standards in reading and 57.3 percent in math.
Though the  Mayor predictably claimed the city's gains of 1.5 percentage points in reading and 3.3 points in math showed great progress, actually the results are very mixed.  Only 35% of 8th graders were proficient in reading.  Moreover, I believe the results overall are still highly unreliable.  Why? 
  • The high stakes attached to test scores  in city schools will tend to lead to gains, because of excessive test prep, narrowing of the curriculum, and even cheating, rather than real learning.  (For more on this see our blog.)
  • The same testing company is still writing them and the same “experts” are in charge at the NY State Education Department as in previous years, when there was tremendous test score inflation (the state intends to switch vendors next year.)
  • Even if the exams were perfectly constructed and scaled, the city's gains are so small as to be likely statistically meaningless.
How else do we know the tests are still flawed?  Only 3.5% of students statewide received 4’s (or advanced) on the ELA exam, and only 2.7% in NYC. This is clearly a test which cannot distinguish performance at the upper levels.
Howard Everson, consultant to SED, claimed otherwise to GothamSchools, saying that the gains under the new standards were small, they can be viewed as statistically significant because of the sheer number of students tested. He also said he trusted the state’s ability to track score trends even as the tests’ length, composition, and proficiency standards change.”    
Yet Everson told the New York Times in 2009 that the state tests were "about as good as we can build them," right before the test score inflation bubble burst and after it was clear to most objective observers quite the opposite. 
In addition, this year, some city schools, according to the NY Post, saw suspicious gains of up to 25 percentage points in both subjects.   Yet the  Mayor in his press conference said the precautions to prevent and check for cheating that were in place before he took office were too expensive to implement. He also said there was “no evidence of widespread cheating” (actually, in a Freudian slip, he said “no evidence of widespread teaching” !) 
Unfortunately, NYSED is no longer releasing the test questions, which will prevent anyone from discerning whether they were poorly or ambiguously worded as has occurred quite frequently in past exams; the state claims that this change “helps to ensure that preparation for the tests goes much deeper than simply reviewing past exams.”    But other states and reputable testing companies like the College Board still release the questions on their exams; why can’t NY State do the same?  The lack of transparency can only further diminish public confidence in the results.

There's a short clip of my views in last night's Fox-TV news story here.


Lee Barrios said...

Your comment that the low percentage of Advanced scores shows that the test "cannot distinguish performance at the upper levels" is telling.

I taught gifted 7th graders for years. Their standardized test scores should have topped out as their Duke University SAT/ACT scores were extremely high and garnered scholarships for many. That was not, however, the case. That and comparing each student's classroom performance with their std. test scores was evidence enough for me that the tests are invalid for their purpose.

Alexandra Miletta said...

Yes the Everson "good as we can build them" echoes what Steiner and Tisch have said about imperfect measures and the idea that less than perfect is better than nothing. But I have to say, Bloomberg is right to note "no evidence of widespread teaching" because all anyone is allowed to do these days is teach to the test and how to take the test. These dismal results show that is getting us absolutely nowhere.

For our Future said...

The items in the test I scored themselves seemed fair,overall, while the writing piece could have been confusing (in the way students were prompted after what they had read). I can tell you that there were some interesting things in the "anchor papers"-supposed student responses to the test questions to be used as exemplars for each "level" or score on a particular piece. It seemed in several cases that an adult (not any elementary aged child)had done the writing, and even had gone through the trouble to disguise their handwriting on separate pieces by adding ovious little swoops and swooshes in places that only a calligrapher would put them. Putting in blatant misspellings that were inconsistent in the same piece, advanced language use and vocabulary then obvious attempts to "sound" young in isolated places (like me saying "Fer sure, dude"-just ain't right). Of course this year, ALL MATERIALS were collected and destroyed. Test security, or destruction of evidence? We are in a time where the plan is to evaluate schools, students and teachers based on tests that are in development, and a gradind system that has always been nebulous. If you were an industry that relies on shools, teachers and students appearing "not good enough", what would your approach be?

Anonymous said...

We need to do more protests, protest and MORE PROTESTS!! Its the only way to show our elected officials what we want and do not want to see.

(By the way, I am also linking this to my blog. I hope you do not mind. The points were clearly stated, as usual).

Unitymustgo! said...

Tried to look up my schools scores. Can the DOE make it an more difficult? Their spending how much on technology this year? Half a billion? How about something that lets me type in my school # and you show me my schools results. Instead you get to download a billion page excel spreadsheet. That suxs.

Bill Gunlocke, a city reader said...

Check out my blog on city schools and reading:


35% of 8th graders proficient in reading? Parents should be marching.

proofoflife said...

To Unity Must Go! You are so right! I spent a half hour scrolling down to find my school! Millions and billions on technology and this is what we get!

Leonie Haimson said...

For our future: please contact me off line at leonie@att.net very interested in following up re destruction of testing materials. thanks!

Anonymous said...

The test scores in no way reflect a child's real ability . In most cases they are higher than what the children's actually ability is. The lack of level 4 shows the test is even incapable of distinguishing the gifted from the average child.The way the tests are scored,many times relying on score's judgment for parts, makes the validity of the test questionable. The emphasis put on these test scores makes them downright dangerous.The amount of money and time wasted on test prep in our schools is scary. The lack of learning meaningful life time skills in our system is the most dangerous thing of all.HELP!!

Anonymous said...
This comment has been removed by a blog administrator.