Thursday, August 24, 2017

State test scores increase again; but is it real? Opt out rates remain high


NYC average proficiency rates in 3rd-8th grade 2013-2017
The NY state and city test scores were released this week.  Proficiency rates statewide increased again though by a smaller amount than last year. In English Language Arts, the percentage of students in grades 3-8 who scored at proficient levels increased by an average of 1.9 percentage points; from 37.9% in 2016 to 39.8%.  In math, the students who scored at proficiency rose to 40.2%, up 1.1 points from 29.1% last year.  

In NYC the increases were a little larger: a gain of more than two points in ELA proficiency to 40.6% and 1.4 points to 37.8% proficiency  in math.  

Commissioner Elia, Mayor de Blasio and Chancellor Farina claimed that the increase in proficiency since 2013 was strong evidence that our students and schools are making progress.

Yet the reality is that the trends over the last 15 years have not matched any of the trends on the more reliable national test called the NAEPs, for either NYC or the state as a whole.  


In fact, the NY State Education Department has appeared unable since 2002 to produce a reliable test and score it consistently enough to allow one to assess if there’s been any sort of improvement in our schools.  Instead, Commissioners and their staff have repeatedly changed cut scores and set proficiency rates to make political points.


There are many ways to show increases in proficiency -- a metric notoriously easy to manipulate -- including making the tests easier, shorter, giving them untimed, and/or changing the scoring by lowering the raw scores to scale scores or the cut scores need for proficiency.  The state has used all these tricks over time.  

From at least 2002- 2008, under Commissioner Mills, there were years of test score inflation, until the questions and scoring became so easy a student could randomly guess their way into a Level 2.  
NYC average proficiency rates in 3rd-8th grade 2002-2008

In New York City, the rapid growth in proficiency rates – in 2009 with 82 percent of city students proficient in math and 69 percent in English, up sharply from 42 and 38 percent, respectively, in 2002 -- helped Michael Bloomberg renew control over the schools and cruise to an unprecedented third term as mayor.  (Here are the charts showing gains up to 2008- I can no longer find the ones from 2009.)



Yet the test score inflation became so embarrassingly obvious to nearly everyone that David Steiner, Mill’s replacement as Commissioner, was forced to reset the cut scores to more reasonable levels in 2010, and statewide proficiency rates fell dramatically from 86% to 61% in math, and from 77% to 53% in ELA.  


In NYC proficiency rates fell even more – in math, proficiency rates dropped 28 points to 54% and reading scores dropped 27 points to 42%.


When Steiner was pushed out by Merryl Tisch in April 2011, she replaced him with John King, who aligned the new exams to the Common Core standards – and proceeded to rate more than two thirds of the student in the state as below proficient.  To this end, King made the exams longer and more difficult, and raised the cut scores much higher.  He determined that less than one third of the state’s students would be deemed proficient, by matching the rate with the percent of NY students achieving arbitrarily high marks on the SATs of 540 in math and 560 in reading, far higher than the score of 500 that the College Board has defined as “college-ready.”   


Parents protested, opt out rates soared, and in December 2014, King left in disgrace.   Test scores began to increase once again. Commissioner Elia was appointed in July 2015, and revamped the tests to make them a bit shorter and allowed students to take them untimed.  The state also lowered the level of raw scores equated with proficiency in 11 out of 12 exams last year.


For more on the history of NYS test score inflation see our blog here:
The state tests are still too long, full of abstruse vocabulary, reading passages above grade level, and too many ambiguous questions.  And last year the opt out rate remains high at 19 percent. 
In 2017, 92% of public school districts had opt out rates of more than 5% in math, and 91% of districts had opt out rates more than 5% in English Language Arts– making the results even more unreliable to assess any progress.   Most experts say you need at least 95% test participation rate on which to base any reliable judgements.  (I used the NYSED test refusal file here; and counted NYC as one district and removed charter schools.)

The NYSED press release and presentation emphasized that of “the total test refusals statewide, the most are from average and low need districts.” Similarly, the pro-testing organizations funded by the Gates organization have repeatedly claimed that this is primarily a phenomenon of middle-class suburban parents.  But if you look within districts instead of across them, there is a very different story to be told:  65% of NYS districts had a HIGHER or equal percent of their economically disadvantaged students opting out of the ELA exams than their district-wide averages in 2017; and 75% of districts in math.  Moreover, several high-poverty urban districts had substantial opt out rates, including 12.7% in Rochester, 20.3% in Albany and 15.2% in Buffalo.

There are other anomalies in both the ELA and the math results suggesting that the tests and/or the proficiency levels may be unreliable.  In math, the NYC mean scale score didn’t increase from last year though the average proficiency level did – and the proportion of level one students remains very high, with more students testing at level one in math than any other level in grades 5-8. 

In ELA, the pattern of increases doesn’t make much sense.  Usually proficiency declines with higher grades; while in NYC, the percent of students testing at proficient levels is much larger in 7th and 8th grades than 5th and 6th grades.  Also highly suspect is the fact that the technical report for this year’s state exams hasn’t been released – no less the exams given over a year ago in 2016.
Mayor de Blasio press conference Aug. 22, 2017 

In any case, just like the previous Mayor, Bill de Blasio held a press conference to trumpet the results and explain how the rise in proficiency levels proved his reforms are working. (Video here.) 

He evinced amnesia when he claimed that for the first time, most kids in NYC are near proficiency, saying that at the end of the previous administration only 30% of kids were reading at grade level and that low achievement was "tolerated" by earlier mayors. 
In fact, no Mayor that I remember ever “tolerated” low levels of achievement, and Bloomberg took great credit in 2008 and 2009 when the proficiency rates soared far higher than they are today --  peaking at 69% in ELA and 82% in math, before the bubble burst.  
De Blasio also claimed that no previous administration had really made any effort to close achievement gap "because we didn't start where we could make the most impact which is early childhood education."  

Yet despite the Mayor’s words, early childhood education doesn't end at preK, but according to most experts lasts at least to 2nd grade.  De Blasio deserves credit for vastly expanding preK, but when Rudy Crew was Chancellor, he put focus on expanding preK and reducing class size in grades K-3– both reforms critical for improving learning opportunities and narrowing the achievement gap.  Crew also invested special funding into reducing class size to even smaller levels in low-performing schools, called the Chancellor’s district. 
Number of NYC 1st-3rd grade students in classes of 30 or more 2007-2016
In contrast, this administration has failed to make any effort into reducing class size anywhere – even the struggling Renewal schools.  Accordingly, class sizes remain very high, with more than 300,000 students in classes of 30 or more citywide. 

The number of students in very large classes in the early grades has soared.  More than 20,000 Kindergarten students were in classes of 25 or more last fall, and more than 43,000 1st-3rd graders in classes of 30 or larger – the latter an increase of 4000% since 2007.  The failure of de Blasio to reduce class size as he promised and as the state law requires prompted Class Size Matters,  the Public Advocate and nine NYC parents to file a legal complaint against the DOE to the State Education Department last month.
Will the last two years of state test score gains prove to be as illusory as those which occurred during the last period of test score inflation of 2002-2009?  The NAEPs are only given once every two years, and I will remain skeptical until the 2017 NAEPs are released in January and February of next year. 
All we know is that the recent rise in proficiency and test scores since 2013 at the state and the city level haven’t yet been matched on the NAEPs– which have instead been flat or slightly declining, as shown below.    

NAEP scores for NYS & NYC vs. state scores 2013-2015

As Karl Marx wrote, history repeats itself, first as tragedy, second as farce.

 

Tuesday, August 15, 2017

Please vote now to send Parent Coalition for Student Privacy and other grassroots privacy activists to SXSWEdu!

Each spring, thousands of edtech entrepreneurs, and advocates funded by the edtech industry, descend upon Austin’s SXSWEdu conference to promote their products and publicize their point of view.

For example, it’s where Bill Gates launched inBloom Inc. in 2013, to push the expansion of data-mining student personal information and online learning. Rarely do you bump into any classroom teachers, parent leaders or grassroots education activists attending, much less find one on stage. We hope to change that in 2018.

Please help us by voting for our panel proposal “Shielding data privacy and resisting online learning.” Our panelists include attorney and privacy activist Bradley Shear, creator of the  annual National Student Data Deletion Day, research associate Faith Boninger, who has just co-authored a terrific new report on student privacy for the National Education Policy Center, Marla Kilfoyle, the Executive Director of Badass Teachers Association (BATS)  and me.


The BATs NEA Caucus supported, and pushed for several pro-privacy and anti-online learning resolutions that were adopted at the recent NEA Representative Assembly.

To submit your vote:
1.     Simply fill out this form.
2.     Sign in and select “PANELPICKER.”
3.     Search for our panel name “Shielding data privacy and resisting online learning.”
4.     Click on “Vote Up.”

The more votes we get, the better our chances of being selected – so please vote today and ask your friends to vote too! The voting window closes Friday, August 25th.

Background

In spring of 2013, Bill Gates took the stage at SXSWEdu to unveil inBloom, his foundation’s $100 million student data collection project which was being piloted in nine states and school districts across the nation. inBloom surfaced in every corner of the conference that year with parties, meet-ups, and a code-a-thon where cash “bounties” were awarded to teams who developed the best apps using the inBloom data store.

At the same time, parents whose children’s data was going to be ensnared in the project were raising their voices in opposition – concerned about how inBloom threatened student privacy and could accelerate the use of online learning. After considerable parental backlash, inBloom shuttered its doors in 2014. Shortly after, parents involved in the fight formed the Parent Coalition for Student Privacy.

Since then, student data privacy has gained national attention and SXSWEdu has featured countless workshops and presentations on the subject – including on the reasons for inBloom’s demise, without inviting a single parent involved in the fight to explain their opposition. We want to change that next year by going directly to the belly of the beast, and inviting others who are active in the resistance movement on the ground to protect student privacy and prevent the expansion of online learning to join with us. You can help by casting your vote in support of our panel here!

Monday, August 14, 2017

Devastating audit of DOE's contracting process, which violates the City Charter and state laws



Last week I wrote about the problems with the new $670,000 DOE contract for Joel Rose's blended/computerized math instruction program, which was originally promised to be provided free to NYC schools in perpetuity since Rose developed it with about a million dollars of city funds while at DOE. The Panel for Educational Policy is due to vote on this contract on August 23.

Yet since his company New Classrooms went private, NYC has been charged many hundreds of thousands of dollars for this program with a highly questionable educational value.  Subsequently, Gary Rubinstein also wrote about the program's low quality, at different times called School of One or Teach to One.

All this cronyism is not an anomaly, sadly, but is consistent with the scandalously lax oversight of DOE contracts -- including a $1.1 billion contract proposed in 2015 for a contractor who had been involved in a kickback scheme just a few years ago --later thankfully rescinded by the Mayor's office and rebid for savings of between $163 million and $727 million.

On Friday, the NYC Comptroller Stringer released a follow-up audit showing how the DOE continues to violate  state law and its own rules when it comes to contracting.  More on this from the New York Daily News, New York Post and Chalkbeat.

From the Daily News:

In fiscal year 2016 the city school system handed out more than 500 contracts worth $2.7 billion without competitive bidding or required protections, Stringer's investigation found.
That’s 64% of the agency's total contract spending of more than $4 billion for that year....the city agency didn’t bother to evaluate vendors or keep track of their work — or to register contracts with Stringer’s office for review.

From the Comptroller's summary: 

A review of 521 limited competition contracts and contract actions revealed:
  • DOE directed 442 of the vendors — approximately 85 percent — to begin work before the contract was even registered with the Comptroller’s Office, in violation of the New York State Education Law, the City Charter and DOE’s own rules;
  • DOE submitted 302 new contracts and renewals to the Comptroller’s Office, on average, 258 days after their start dates, while 140 extensions of existing contracts were submitted on average 137 days after work began;
  • In the most egregious case, a contract was submitted to the Comptroller’s Office 910 days — nearly two and a half years — after the vendor began work; and
  • DOE submitted dozens of contracts to the Comptroller’s Office after the contract’s end date.
I myself discovered that of the 27 contracts approved at last month's Panel for Educational Policy meeting , 13 were retroactive—meaning at least some or all of the vendor's work had already started –  rendering the vote of the PEP pretty much irrelevant.  Contracts are only submitted and registered by the Comptroller after the PEP has voted. 

For the audit they specifically looked at how many contracts were registered retroactively.  Of 358 contracts and contract renewals, 84% of the vendors started work before registration,  an average of eight months prior to submission.  More than 10% of the contracts were submitted after the work had already ended.  In another data set of 163 contract extensions, 86% had started before submission,  an average of 137 days prior.


The Comptroller’s audit also found these continuing problems:
  • DOE still does not provide sufficient guidance for contract assignments, and, in particular, it fails to ensure that a proposed new vendor’s capacity and history are adequately assessed. In addition, assignment files lack sufficient information justifying the assignments, detailing prior performance and documenting adequate insurance coverage, among other things.
  • DOE has not developed written procedures or guidelines to help staff detect the warning signs of possible collusion when awarding contracts via actions other than fully competitive solicitations.
  • While general guidance to contract managers appears to be available now in DOE’s contract management training manual, it does not offer the contract managers specific instruction about handling their responsibilities. For example, the training manual is silent on the performance evaluations that must be conducted throughout the contract period and on the frequency of such evaluations.
  • Despite DOE’s representation in response to the previous audit that it would implement a standard format with standard criteria and ratings for evaluating vendor performance and a schedule for conducting such evaluations, we found that no such implementation has taken place.
  • DOE still does not conduct required performance evaluations of contractors seeking renewals or extensions.
  • DOE still does not adequately monitor its contract managers to ensure that contract monitoring and performance evaluations are conducted in accordance with its PPP manual.
  • DOE does not ensure that limited-competition and non-competitive contracts are registered with the Comptroller prior to vendors beginning performance. As a result, vendor payments are delayed, which could potentially decrease competition and raise prices.
David Ross
The question of evaluations is key; many DOE contracts are renewed several times without any evidence that any assessments had been done of the quality of their previous work.  Even when an evaluation was done, in the case of New Classrooms for example, which showed no significant gains  through a randomized experiment, this study was unmentioned in the DOE Request for Authorization document, which asked the PEP for approval.

All these problems are not new, but occurred during the previous administration as well, under the aegis of David Ross.  Ross spent 12 years at the DOE as chief procurement officer,  and is now going to oversee contracts at the MTA – another troubled government agency that cannot afford to waste a single penny of public funds. Will the new contract head at DOE clean up this horrible mess?  We will have to see, but from the weak responses to the audit from DOE, I would not bet on it.

The person ultimately in charge however is Chancellor Fariña.  We've had four Chancellors in a row who were lax in terms of allowing the waste of  millions of dollars in taxpayer money, while more than 300,000 NYC children continue to languish in classes of 30 or more.  Will Fariña stick around after the election?  And if not, who will replace her?