Wednesday, September 26, 2018

More bloviating from NYS and NYC on test scores, all over again


Update: see the statement from NYSUT here:"the State Education Department has failed to adjust the troublesome proficiency benchmarks to take into account changes in the state tests. “The state’s measuring stick for determining ‘proficiency’ remains badly flawed, painting a misleading and harmful picture of our public education system. In simple terms, they do not add up. The benchmarks must be fixed before any more students or schools are incorrectly labeled.”

Additional doubts expressed about the reliability of these results from Aaron Pallas of Columbia University in the Daily News: 

 "This week, we are seeing 2018 test results; but the state Education Department has still not released the technical report documenting how the scores for the 2016 tests, administered in April of that year, were calculated. The tests have been used for both high- and low-stakes purposes, but we still don’t know much about them.
Tests can be useful indicators of how individual students, schools and districts are doing, and whether they are improving, holding steady or declining over time. But the imperfections of the state’s testing system urge caution in making too much of any single test administration. It’s best to place the test scores alongside other performance indicators, and not to treat them as more precise or accurate than they really are."
 
So once again, in an eternal Groundhog day, the Mayor trumpeted the new state test scores, claiming great progress: 
“We came into office to shake the foundation of a system that neglected too many students, parents, teachers and schools,” said Mayor Bill de Blasio. “We now have a school system that is steadily improving before our eyes. We’ve seen steady gains across our students’ State math and English exams, proving that equity and excellence go hand in hand. I salute our students on their progress.”
And yet as clear as clear as day, on nearly every NYSED slide the following statement is made: 
"Due to the new two-session test design and performance standards, the 2018 Grades 3-8 ELA and Math results cannot be compared with prior results."
No comparisons can be made, because the exams were fundamentally changed, shortened to two days instead of three.  Also computerized testing was used in much of the state, the opt out rates remained high at 18%, and the entire system of scaling was altered.  Now the maximum score is about 651-661 in ELA and 644-656 in math instead of previously at the 402-428 in ELA and 387-401 in math
Why else can't we trust any claims of progress?  Because the trend lines on the more reliable national assessments called the NAEPs for the last four years have not matched the trend lines on the state tests.  While scores and proficiency levels on the state exams keep on going up, for both the state and the city, they are stagnant or falling on the NAEPs -- known as the "audit test" because its questions, scoring,  testing time, and scaling do not change much from year to year.  
The NAEPs are only given every two years, in 4th and 8th grade reading and math, so we can only look at the comparisons for those data points. In the charts below, instead of the apparent gains the city and state made between 2013 and 2017 on the state test scores, results in both 4th grade reading and math actually dropped since the Mayor took office. (NYC scores are in bold vs NYS scores; and the NAEPs are in orange below the state test scores in green.)
Also see this from the City Hall press release: "New York City students outperformed the State on State English exams for the first time ever in 2016 and have now done so for three years in a row.”
But see above.  According to the more reliable NAEPs, NYC has never matched NY State in either average test scores or proficiency levels, and not in 2015 and 2017 when this happened in the 4th grade state reading scores, nor 2013 when it happened in 4th grade math.  (If you don't trust my NAEP graphs, check out the ones on DOE's website here.)
Just as during the days of Bloomberg/Klein, City Hall and Tweed prefer to inhabit an illusory bubble produced by state test score inflation & enabled by an incompetent and/or dishonest State Education Department. 
Why do I blame the state as well?  Because in many of the very same slides in which NYSED cautioned that no year-to-year comparisons can be made, they also claim the results show improvements in test scores compared to the previous year.  
See  slide 50 for example, in which the two sentences are included one after another.  
After saying, "Cannot compare 2018 scores with prior years because of switch from 3 to 2 -day testing", they add, "All race and ethnicity groups made progress, continuing to slowly close the achievement gap."    
Juxtaposing these two sentences is oxymoronic, and beggars belief.


Here are some additional questions that I would have asked the Commissioner and/or the Mayor if I'd had the chance:

  1. How can NYSED or DOE or mayor claim progress has been made, if as clearly stated that as a result in the change in the tests, this year’s scores aren’t comparable to previous years?
  2.  Why did they so radically change the scoring range, from a maximum of about 428 to about 651 this year?
  3. Why does the state no longer report scale scores in its summaries, rather than proficiency levels which are notoriously easy to manipulate?
  4. Where are the NYSED technical reports for 2016, 2017, and 2018 that could back up the reliability of the scoring and the scaling?
  5. Why was the public release of the scores delayed though schools have had student level scores t for a month?
  6. How were the state vs the city comparisons affected by the fact that opt out rates in the rest of the state averaged more than 18% while they were only about 4% here?
  7. Finally, how can either the state or the city claim that these tests are reliable or valid, when neither the scoring nor the trends have been matched on the NAEPs, in which NYC scores have NEVER equaled the state in any category and results for the state & city have fallen in 4th grade math and reading since 2013?
Though the Mayor apparently tempered his tone at this afternoon's press conference, according to Twitter he apparently claimed that he expects next year's scores to show significant gains because those 3rd graders will have had the benefit of Universal preK.
 
Sorry to say I won't trust the state test results next year either.  We will have take those scores with several handfuls of salt too -- and wait for the 2019 NAEP scores to judge their reliability.  


Parents to Improve School Transportation (PIST NYC) Statement on the Current Busing Crisis

Check out our previous blog post on the current busing problems here.  

We in PIST NYC have been saying for eight years that school transportation is run by the wrong people. The fact that the Chancellor has denounced one company and fired the top Office of Pupil Transportation executive is refreshing, but our children’s safe travel and access to education remain at the mercy of companies and executives whose first priority is money.  
Any true overhaul would not just change some faces, but would change the structure and empower the real stakeholders in school busing: the disability community; families of riders of all income levels, races, and languages; the workers who drive and attend to them; educators.  Each of these already have organizations that represent us.
For years, these stakeholders, along with civil rights lawyers, have stated practical methods for fixing everyday school bus route problems. Parents to Improve School Transportation and Amalgamated Transit Union local 1181-1061 have been advocating for a School Bus Bill of Rights to prevent route problems systematically.  
These suggestions are all on record; we have documentation from various City Council hearings, proposals raised to the now-Mayor and now-Public Advocate by Attorney Norman Siegel, notes from multiple parent discussions with Office of Pupil Transportation Director Alexandra Robinson, and a study paid for by the Department of Education itself. Further, at a 2013 mayoral candidates’ forum, Bill DeBlasio agreed to: “an independent commission on school busing, with representation from disability advocates, unions, and parent groups, on standards for bidders, routes, safety, training, and fair labor practices.”  Was Chancellor Carranza advised of any of this?
What is missing from most coverage of the scandal is this: The sad state of New York City school busing got worse after Bloomberg’s administration undermined the Employee Protection Provisions (EPP) that the union had won fair and square.  The companies are impelled to use strict cost-cutting to lower their bids for DOE/OPT contracts; they offer worse pay/benefits and conditions to school bus drivers, attendants and mechanics; hundreds were made jobless or found retirement more appealing; the result is a shortage of trained, experienced bus professionals. [Indeed, a cohort of school bus companies has been in court lately, trying to eliminate EPP altogether against the testimony of parents, workers, and one or two other companies. If they win, we expect more of the same. Even if they lose, these proceedings stall the start of new route contract bids; so OPT recently extended contracts that in theory might not have been renewed otherwise.]
No wonder school bus routes are doubled-up and miserable this Fall! Consider it on a small scale: Any time there is only one driver willing to tolerate a low wage job--where there used to be two drivers enjoying a secure career, that’s when students get picked up late or not at all, have routes which stop at four schools instead of one or two, get yelled at, or worse. When there is little to no investment in climate control or upgrading of vehicles, that’s when children get dehydrated, or their car seats or wheelchairs do not attach properly, or the bus breaks down.
Multiply this by thousands, and you will understand our doubt that merely removing CEO Eric Goldstein or Grandpa’s Bus would make all 150,000 riders safer (We also think the figures and emphasis on background checks are overstated to sensationalize rather than solve the problem).
Any true overhaul of school busing would have a long term goal of public ownership of the bus yards, to stop a chunk of our tax money in the busing budget from going straight to profit for individual companies--to spend it instead on upgrading and “greening” the vehicles. We look forward to civil service-type standards for regulating employee recruitment, training, and retention.
In the meantime:    

  • New York City students deserve a skilled workforce that is compensated with EPP, and is listened to about ways to improve the situation.
  • We believe that unionized bus crews are the buffer against employer moves that harm young people, whose lives and education matter!   
  • We want workers' and parents' grievances about long, overcrowded routes to be expedited. 
  • We want a mass way to enforce and strengthen existing regulations on travel time limits, bus temperatures, and other conditions, rather than leaving it up to individual political appointees or company owners.

As always, we urge all NYC communities and labor to get involved in our campaign for a School Bus Bill of Rights.  For more information, please write to pistnyc@gmail.com
For information in fluent Spanish, please contact https://www.facebook.com/groups/221709251868819/
--  Parents to Improve School Transportation