Disappointing 2017 results released from literacy coach initiative- but is DOE hiding even more recent findings?

On Thursday, Leslie Brody of the Wall Street Journal reported that the first year results of the DOE literacy coach program were a bust, as the 2^nd graders at schools with these full time teacher coaches tested no better after nearly a year than similar students in other schools with no coaches. 

And yet the DOE plans to keep on expanding this program, and to spend nearly $90 million on it next year:

 A major push by New York City to help poor children in public schools learn to read by assigning literacy coaches to their teachers had no impact on second-graders’ progress, according to a study of its first year.

The city Department of Education conducted the evaluation, but its officials said Thursday it was too early to judge the initiative. They said they would strengthen the program while boosting annual funding to $89 million, from $75 million….

This evaluation tested second-graders in schools that had literacy coaches, and compared their results with peers in similar city schools that had no coaches. The report found that both groups of students were behind in skills in October 2016 and fell further behind expectations by May 2017.

Each group gained an average of four months of skills, when they should have gained seven months. At the end of second grade, students in schools with coaches on average performed at the level expected in the second month of second grade, on a measure known as the Gates-MacGinitie Reading Test. It covered decoding skills, word knowledge and comprehension.

This program is really the only major DOE initiative focused on boosting learning in elementary schools, aside from the expansion of preK, as the mayor went back on his campaign promises to reduce class size.

Like the Renewal program for struggling schools, this initiative was launched by Chancellor Farina with great fanfare, based on her theory of education that the only thing that really matters is providing more professional development to teachers.  Teachers cannot have too much training, according to Farina, and the Renewal program featured more and more PD for teachers and principals, with the total program (along with wrap-around services) costing about $187 million last year, according to the Independent Budget Office, with disappointing results.  Many schools on the Renewal list have now closed, without ever getting a chance to reduce class size.  Those that did feature small classes saw a significantly better chance of success, and many of them have left the program, as I pointed out in my testimony on Renewal schools last year.

Similarly, for all schools, Farina altered the use of the 20 minutes a day that teachers had devoted to providing small group instruction to struggling learners to yet more teacher PD.

Farina's near exclusive focus on PD was accompanied by a belief, as she often expressed it at Town hall meetings when parents complained about the huge size of their kids’ classes, that her concern was not that classes were too large, but they were too small. Perhaps as a result of this single-minded devotion in favor of spending time and funds on PD to the exclusion of reducing class size, NYC NAEP scores have stalled in most areas, and actually declined in 4th grade math since.

I previously wrote about the literacy program last year here; and predicted it wouldn’t work, based on the past record of an expensive program in which Chancellor Klein hired literacy coaches in all elementary schools during the first phase of Children’s First, presumably at the advice of Carmen Farina who was Deputy Chancellor at the time.  This program similarly led to little or no gains in reading, according to the NAEPs, and was given up after a few years. 

Now with Farina gone, and with these mediocre results, one would hope for a new direction at DOE, yet Josh Wallack, the Deputy Chancellor, is quoted as saying full speed ahead:

Department of Education Deputy Chancellor Josh Wallack said he had confidence in the coaches, their training and principal buy-in. He noted that some schools showed real improvements.

“We think we are on the right track,” he said. “We know we have a lot of work to do.”

Skeptics of the initiative have long argued it would be better to reduce class size, add services for the disabled and require a stronger focus on phonics, which teaches children to sound out letters as a primary way to identify words.

The department has expanded the literacy initiative yearly, and will dispatch about 500 coaches this fall,  [out of 661 elementary schools] with every elementary school getting a coach or additional attention.

 After I read the Wall Street Journal article, I asked for and received a copy of this evaluation from DOE which is now posted here. I was surprised to discover that it is not really a formal evaluation, but a mere eleven slide power point, larded with inspirational quotes and with only one slide devoted to the test results. 

That slide shows that the 2^nd graders tested in November and May in both “current and future ULit schools [those with literacy coaches] started behind in reading and fell further behind” . 

There are also survey results showing that while coaches reported that they spent the greatest amount of time with classroom teachers “co-planning”, teachers themselves wanted instead “More support in working with struggling readers.”

The DOE deck ends this way:

 “We hope to see reading gains in the coming years

• ·       As coaches deepen their knowledge base and craft

• ·       As teachers and schools benefit from multiple years of coaching.”

(Strangely, this comment omits mention of the students who one would hope would be the main beneficiaries of the program.)

What I hadn’t originally noticed from the Wall St. Journal story was the comparative analysis of test score gains (or losses) was based on assessments that these students took more than a year ago, in October 2016 and then in May of 2017.  Why would DOE wait till August 2018 to release this analysis? 

More importantly, where are the more recent results?  Surely there must be new data including the scores of  2^nd and 3^rd graders from October 2017 and May of 2018.  A statistical analysis could be done in a matter of  hours or at most days to see if there was any evidence of improved results, especially as they already have a pre-selected comparison group of students at similar schools.  

One has to wonder if the folks at DOE waited to see the latest scores, but since they didn’t show better results omitted them from this summary and will wait for yet another year, hoping for better news to provide from the third year of the initiative.  If they wait until the fourth year of the program to deliver those results, it will have already cost city taxpayers about $300 million on a cumulative basis.

I have FOILed for any more thorough evaluation of the program, as well as more recent data from the schools with literacy coaches, and/or any analyses based on them.  Let’s see what data,  if any, DOE provides.

State test scores increase again; but is it real? Opt out rates remain high

NYC average proficiency rates in 3rd-8th grade 2013-2017

The NY state and city test scores were released this week.  Proficiency rates statewide increased again though by a smaller amount than last year. In English Language Arts, the percentage of students in grades 3-8 who scored at proficient levels increased by an average of 1.9 percentage points; from 37.9% in 2016 to 39.8%.  In math, the students who scored at proficiency rose to 40.2%, up 1.1 points from 29.1% last year.  

In NYC the increases were a little larger: a gain of more than two points in ELA proficiency to 40.6% and 1.4 points to 37.8% proficiency  in math.  

Commissioner Elia, Mayor de Blasio and Chancellor Farina claimed that the increase in proficiency since 2013 was strong evidence that our students and schools are making progress.

Yet the reality is that the trends over the last 15 years have not matched any of the trends on the more reliable national test called the NAEPs, for either NYC or the state as a whole.  

In fact, the NY State Education Department has appeared unable since 2002 to produce a reliable test and score it consistently enough to allow one to assess if there’s been any sort of improvement in our schools.  Instead, Commissioners and their staff have repeatedly changed cut scores and set proficiency rates to make political points.

There are many ways to show increases in proficiency -- a metric notoriously easy to manipulate -- including making the tests easier, shorter, giving them untimed, and/or changing the scoring by lowering the raw scores to scale scores or the cut scores need for proficiency.  The state has used all these tricks over time.  

From at least 2002- 2008, under Commissioner Mills, there were years of test score inflation, until the questions and scoring became so easy a student could randomly guess their way into a Level 2.  

NYC average proficiency rates in 3rd-8th grade 2002-2008

In New York City, the rapid growth in proficiency rates – in 2009 with 82 percent of city students proficient in math and 69 percent in English, up sharply from 42 and 38 percent, respectively, in 2002 -- helped Michael Bloomberg renew control over the schools and cruise to an unprecedented third term as mayor.  (Here are the charts showing gains up to 2008- I can no longer find the ones from 2009.)

Yet the test score inflation became so embarrassingly obvious to nearly everyone that David Steiner, Mill’s replacement as Commissioner, was forced to reset the cut scores to more reasonable levels in 2010, and statewide proficiency rates fell dramatically from 86% to 61% in math, and from 77% to 53% in ELA.  

In NYC proficiency rates fell even more – in math, proficiency rates dropped 28 points to 54% and reading scores dropped 27 points to 42%.

When Steiner was pushed out by Merryl Tisch in April 2011, she replaced him with John King, who aligned the new exams to the Common Core standards – and proceeded to rate more than two thirds of the student in the state as below proficient.  To this end, King made the exams longer and more difficult, and raised the cut scores much higher.  He determined that less than one third of the state’s students would be deemed proficient, by matching the rate with the percent of NY students achieving arbitrarily high marks on the SATs of 540 in math and 560 in reading, far higher than the score of 500 that the College Board has defined as “college-ready.”   

Parents protested, opt out rates soared, and in December 2014, King left in disgrace.   Test scores began to increase once again. Commissioner Elia was appointed in July 2015, and revamped the tests to make them a bit shorter and allowed students to take them untimed.  The state also lowered the level of raw scores equated with proficiency in 11 out of 12 exams last year.

For more on the history of NYS test score inflation see our blog here:

2007 http://nycpublicschoolparents.blogspot.com/2007/09/here-are-numbers-everything-is.html

2009 https://nycpublicschoolparents.blogspot.com/2009/08/ny-times-artlcle-falls-in-line-with.html

2010 https://nycpublicschoolparents.blogspot.com/2010/10/in-its-account-of-test-score-inflation.html

2016 https://nycpublicschoolparents.blogspot.com/2016/08/evidence-grows-we-are-entering-new-era.html

The state tests are still too long, full of abstruse vocabulary, reading passages above grade level, and too many ambiguous questions.  And last year the opt out rate remains high at 19 percent. 

In 2017, 92% of public school districts had opt out rates of more than 5% in math, and 91% of districts had opt out rates more than 5% in English Language Arts– making the results even more unreliable to assess any progress.   Most experts say you need at least 95% test participation rate on which to base any reliable judgements.  (I used the NYSED test refusal file here; and counted NYC as one district and removed charter schools.)

The NYSED press release and presentation emphasized that of “the total test refusals statewide, the most are from average and low need districts.” Similarly, the pro-testing organizations funded by the Gates organization have repeatedly claimed that this is primarily a phenomenon of middle-class suburban parents.  But if you look within districts instead of across them, there is a very different story to be told:  65% of NYS districts had a HIGHER or equal percent of their economically disadvantaged students opting out of the ELA exams than their district-wide averages in 2017; and 75% of districts in math.  Moreover, several high-poverty urban districts had substantial opt out rates, including 12.7% in Rochester, 20.3% in Albany and 15.2% in Buffalo.

There are other anomalies in both the ELA and the math results suggesting that the tests and/or the proficiency levels may be unreliable.  In math, the NYC mean scale score didn’t increase from last year though the average proficiency level did – and the proportion of level one students remains very high, with more students testing at level one in math than any other level in grades 5-8. 

In ELA, the pattern of increases doesn’t make much sense.  Usually proficiency declines with higher grades; while in NYC, the percent of students testing at proficient levels is much larger in 7^th and 8^th grades than 5^th and 6^th grades.  Also highly suspect is the fact that the technical report for this year’s state exams hasn’t been released – no less the exams given over a year ago in 2016.

Mayor de Blasio press conference Aug. 22, 2017 

In any case, just like the previous Mayor, Bill de Blasio held a press conference to trumpet the results and explain how the rise in proficiency levels proved his reforms are working. (Video here.) 

He evinced amnesia when he claimed that for the first time, most kids in NYC are near proficiency, saying that at the end of the previous administration only 30% of kids were reading at grade level and that low achievement was "tolerated" by earlier mayors. 

In fact, no Mayor that I remember ever “tolerated” low levels of achievement, and Bloomberg took great credit in 2008 and 2009 when the proficiency rates soared far higher than they are today --  peaking at 69% in ELA and 82% in math, before the bubble burst.  

De Blasio also claimed that no previous administration had really made any effort to close achievement gap "because we didn't start where we could make the most impact which is early childhood education."  

Yet despite the Mayor’s words, early childhood education doesn't end at preK, but according to most experts lasts at least to 2^nd grade.  De Blasio deserves credit for vastly expanding preK, but when Rudy Crew was Chancellor, he put focus on expanding preK and reducing class size in grades K-3– both reforms critical for improving learning opportunities and narrowing the achievement gap.  Crew also invested special funding into reducing class size to even smaller levels in low-performing schools, called the Chancellor’s district. 

Number of NYC 1st-3rd grade students in classes of 30 or more 2007-2016

In contrast, this administration has failed to make any effort into reducing class size anywhere – even the struggling Renewal schools.  Accordingly, class sizes remain very high, with more than 300,000 students in classes of 30 or more citywide. 

The number of students in very large classes in the early grades has soared.  More than 20,000 Kindergarten students were in classes of 25 or more last fall, and more than 43,000 1^st-3^rd graders in classes of 30 or larger – the latter an increase of 4000% since 2007.  The failure of de Blasio to reduce class size as he promised and as the state law requires prompted Class Size Matters,  the Public Advocate and nine NYC parents to file a legal complaint against the DOE to the State Education Department last month.

Will the last two years of state test score gains prove to be as illusory as those which occurred during the last period of test score inflation of 2002-2009?  The NAEPs are only given once every two years, and I will remain skeptical until the 2017 NAEPs are released in January and February of next year. 

All we know is that the recent rise in proficiency and test scores since 2013 at the state and the city level haven’t yet been matched on the NAEPs– which have instead been flat or slightly declining, as shown below.    

NAEP scores for NYS & NYC vs. state scores 2013-2015

As Karl Marx wrote, history repeats itself, first as tragedy, second as farce.