More bloviating from NYS and NYC on test scores, all over again

Update: see the statement from NYSUT here:"the State Education Department has failed to adjust the troublesome proficiency benchmarks to take into account changes in the state tests. “The state’s measuring stick for determining ‘proficiency’ remains badly flawed, painting a misleading and harmful picture of our public education system. In simple terms, they do not add up. The benchmarks must be fixed before any more students or schools are incorrectly labeled.”

Additional doubts expressed about the reliability of these results from Aaron Pallas of Columbia University in the Daily News: 

 "This week, we are seeing 2018 test results; but the state Education Department has still not released the technical report documenting how the scores for the 2016 tests, administered in April of that year, were calculated. The tests have been used for both high- and low-stakes purposes, but we still don’t know much about them.

Tests can be useful indicators of how individual students, schools and districts are doing, and whether they are improving, holding steady or declining over time. But the imperfections of the state’s testing system urge caution in making too much of any single test administration. It’s best to place the test scores alongside other performance indicators, and not to treat them as more precise or accurate than they really are."

 

So once again, in an eternal Groundhog day, the Mayor trumpeted the new state test scores, claiming great progress: 

Mayor de Blasio and Chancellor Carranza Announce NYC Students Continue to Outperform NY State on English Exams and Continue to Close Gap on Math Exams

“We came into office to shake the foundation of a system that neglected too many students, parents, teachers and schools,” said Mayor Bill de Blasio. “We now have a school system that is steadily improving before our eyes. We’ve seen steady gains across our students’ State math and English exams, proving that equity and excellence go hand in hand. I salute our students on their progress.”

And yet as clear as clear as day, on nearly every NYSED slide the following statement is made: 

"Due to the new two-session test design and performance standards, the 2018 Grades 3-8 ELA and Math results cannot be compared with prior results."

No comparisons can be made, because the exams were fundamentally changed, shortened to two days instead of three.  Also computerized testing was used in much of the state, the opt out rates remained high at 18%, and the entire system of scaling was altered.  Now the maximum score is about 651-661 in ELA and 644-656 in math instead of previously at the 402-428 in ELA and 387-401 in math. 

Why else can't we trust any claims of progress?  Because the trend lines on the more reliable national assessments called the NAEPs for the last four years have not matched the trend lines on the state tests.  While scores and proficiency levels on the state exams keep on going up, for both the state and the city, they are stagnant or falling on the NAEPs -- known as the "audit test" because its questions, scoring,  testing time, and scaling do not change much from year to year.  

The NAEPs are only given every two years, in 4th and 8th grade reading and math, so we can only look at the comparisons for those data points. In the charts below, instead of the apparent gains the city and state made between 2013 and 2017 on the state test scores, results in both 4th grade reading and math actually dropped since the Mayor took office. (NYC scores are in bold vs NYS scores; and the NAEPs are in orange below the state test scores in green.)

Also see this from the City Hall press release: ‘"New York City students outperformed the State on State English exams for the first time ever in 2016 and have now done so for three years in a row.”

But see above.  According to the more reliable NAEPs, NYC has never matched NY State in either average test scores or proficiency levels, and not in 2015 and 2017 when this happened in the 4th grade state reading scores, nor 2013 when it happened in 4th grade math.  (If you don't trust my NAEP graphs, check out the ones on DOE's website here.)

Just as during the days of Bloomberg/Klein, City Hall and Tweed prefer to inhabit an illusory bubble produced by state test score inflation & enabled by an incompetent and/or dishonest State Education Department. 

Why do I blame the state as well?  Because in many of the very same slides in which NYSED cautioned that no year-to-year comparisons can be made, they also claim the results show improvements in test scores compared to the previous year.  

See  slide 50 for example, in which the two sentences are included one after another.  

After saying, "Cannot compare 2018 scores with prior years because of switch from 3 to 2 -day testing", they add, "All race and ethnicity groups made progress, continuing to slowly close the achievement gap."    

Juxtaposing these two sentences is oxymoronic, and beggars belief.

Here are some additional questions that I would have asked the Commissioner and/or the Mayor if I'd had the chance:

1. How can NYSED or DOE or mayor claim progress has been made, if as clearly stated that as a result in the change in the tests, this year’s scores aren’t comparable to previous years?
2.  Why did they so radically change the scoring range, from a maximum of about 428 to about 651 this year?
3. Why does the state no longer report scale scores in its summaries, rather than proficiency levels which are notoriously easy to manipulate?
4. Where are the NYSED technical reports for 2016, 2017, and 2018 that could back up the reliability of the scoring and the scaling?
5. Why was the public release of the scores delayed though schools have had student level scores t for a month?
6. How were the state vs the city comparisons affected by the fact that opt out rates in the rest of the state averaged more than 18% while they were only about 4% here?
7. Finally, how can either the state or the city claim that these tests are reliable or valid, when neither the scoring nor the trends have been matched on the NAEPs, in which NYC scores have NEVER equaled the state in any category and results for the state & city have fallen in 4^th grade math and reading since 2013?

Though the Mayor apparently tempered his tone at this afternoon's press conference, according to Twitter he apparently claimed that he expects next year's scores to show significant gains because those 3rd graders will have had the benefit of Universal preK.
 

Sorry to say I won't trust the state test results next year either.  We will have take those scores with several handfuls of salt too -- and wait for the 2019 NAEP scores to judge their reliability.