John Pane of RAND writes to correct my post on Teach to One and my response

About two weeks ago, I posted a history of the program Teach to One (TtO): how it had first been developed in NYC Department of Education as a blended learning math program called School of One, how after it had spun off from DOE as a separate company called New Classrooms, the developer Joel Rose had promised never to charge NYC schools a fee to use it, instead granting them with a “royalty free, perpetual, non-exclusive license”, but then how the company has continued to charge a license fee to NYC schools anyway. The main focus of the piece was to describe how the huge hype surrounding the Teach to One program and the suppression of the findings of negative or null evaluations of its results has allowed it to expand to more schools, despite disappointing  results and a 60 percent school attrition rate.

In a single paragraph towards the end of this rather lengthy post,  I summarized the findings of a RAND report on the Next Generation Learning Challenge (NGLC) schools, assuming that schools using Teach to One were part of the evaluation, since TtO is a grantee of the NGLC program.  Diane Ravitch subsequently ran excerpts of my blog on hers.

John Pane, the lead researcher on the Rand report, wrote to Diane that New Classrooms / Teach to One was not one of the programs included in this evaluation.  I have posted a correction on that matter on my original blog post.   

He also critiqued the way I reported his remarks to Education Week about “personalized learning” schools in general, that “the evidence base is very weak at this point,” and said that the paragraph in which I described the results of the Rand report had “numerous false and misleading statements,” including my summary of survey results that suggest that the students at NGLC schools “were more likely to feel alienated and unsafe compared to matched students at similar schools.”

He has granted his permission to quote his letter in full below, which I have done, along with my response to the points in his letter.   

On Thu, Mar 7, 2019 at 12:17 PM Pane, John <jpane@rand.org> wrote:

Dear Diane, 

On March 4, 2018 you published this blog entry, “Leonie Haimson: Reality Vs. Hype in “Teach to One” Program,” excerpting from Leonie Haimson’s blog. Your excerpt included this paragraph about my own research (with colleagues) and my public statements:

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, and their students were more likely to feel alienated and unsafe compared to matched students at similar schools. The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“

 This paragraph by Haimson has numerous false and misleading statements. Here I summarize my critique, excerpting the original paragraph: 

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, …”

None of the schools in our sample reported using Teach to One (TtO) among the 194 education technology products they mentioned. Our sample includes schools in the Next Generation Learning Challenges (NGLC) wave IIIa and wave IV programs, a subset of all the NGLC initiatives. Haimson points to blog posts by NGLC about Summit and TtO, but that does not mean our study included them.

“…included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, …”

Our conclusions were about the whole sample of schools, and did not single out any particular schools as is implied by juxtaposing “Summit and Teach to One” with “these schools.” Our concluding remarks related to achievement did not say “small and mostly insignificant.” What we actually said was, “Students in NGLC schools experienced positive achievement effects in mathematics and reading, although the effects were only statistically significant in mathematics. On average, students overcame gaps relative to national norms after two years in NGLC schools. Students at all levels of achievement relative to grade-level norms appeared to benefit. Results varied widely across schools and appeared strongest in the middle grades.” 

 “… and their students were more likely to feel alienated and unsafe compared to matched students at similar schools”

This was not a conclusion of our report. In a supplemental appendix we did compare results from our sample (again, the whole sample of schools in the study, none of which reported using TtO) to a national sample. Our method did not use “matched students at similar schools.” Given data limitations, we were able to make the student samples similar (through weighting) only on grade level, gender, and broad classifications of geographic locale (e.g., urban vs. suburban). Even after weighting, we suspect the high-minority, high-poverty schools in the NGLC sample may be located in more distressed communities than the national survey counterparts, and that this could be related to feelings of safety. Indeed, fewer NGLC students (78 vs. 82 percent) agreed that “I feel safe in this school,” but this small difference cannot be attributed to personalized learning and has no direct relevance to TtO. None of our survey items or reports used the word “alienated.” Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” and “I feel good about being in this school,” 76 percent agreed that “I care about this school” and 72 percent agreed “I am an important part of my school community.” 

The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“ 

This EdWeek article clearly states that it is about “what K-12 educators and policymakers need to know about the research on personalized learning” broadly. Quoting accurately, “RAND has found some positive results, including modest achievement gains in some of the Gates-funded personalized-learning schools. But overall, ‘the evidence base is very weak at this point, Pane said.” There is no justification for Haimson to insert “[for these schools]” into my quoted remark. It appears as though Haimson is attempting to give a misleading impression that I was specifically talking about Summit and TtO rather than the entire body of personalized learning research. 

I find it very unfortunate that you accepted Haimson’s claims without fact checking, and increased their visibility and attention through your own platform. 

I am requesting that you please issue a correction in a way that previous readers of your March 4 post will likely notice. You may include this letter if you wish.

 With regards,

 John Pane, RAND Corporation

____

My response:  

I have now posted a correction on my blog post about the NGLC report – which  was only a small part of my post on TTO which is here .

 It is unfortunate that the names of the specific online program that were evaluated were left out of the RAND evaluation.  I had wrongly assumed that  TTO was included, since it is one of the most heavily funded and promoted by Gates and others of the Next Generation Learning Challenge “personalized learning” programs. 

I would also like to point out that the following survey statistics that John Pane includes in his letter about the NGLC schools omit the results from the comparison schools, as cited in the appendix of the Rand  report:

Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” [compared to 86% of the national sample] and “I feel good about being in this school,” [vs. 89% of the national sample] 76 percent agreed that “I care about this school” [vs. 87% of the national sample] and 72 percent agreed “I am an important part of my school community.” [compared to 79% of the national sample.]

In addition, the  students at the NGLC personalized learning schools were more likely to say that that “their classes do not keep their attention, and they get bored” compared to the national sample (30% to 23%). Only 35% of students at the NGLC schools said that “learning is enjoyable” compared to 45% of the national sample. With results like this it is difficult to see what was wrong with my statement that students at these schools were more likely to feel alienated and unsafe.

Now we know that TtO students aren’t included in these surveys, but there is no reason to assume that their responses would be significantly different until and unless New Classrooms releases their own survey results.  And we do know that from the survey of students at the Mountain View school, which used TtO, showed a 413% increase in the number of students who said they hated math as a result.

 Nor does John Pane’s response relate to the larger question which I discussed in my post, about how problematic it is to use MAP scores to evaluate these programs, especially scores from students  that aren’t disaggregated by race or economic status.  One might expect that with all the data that NWEA has by now they would have done that by now.

Finally, it is extremely unfortunate that Gates, Zuckerberg etc. haven’t bothered to commission any truly randomized  small-scale evaluation of Summit Learning, TtO or any of the other personalized learning programs that they so heavily fund and promote before expanding their reach and subjecting hundreds of thousands of students to them.   Summit has rejected  any independent evaluation of its results.  One can only speculate why.

Devastating audit of DOE's contracting process, which violates the City Charter and state laws

Last week I wrote about the problems with the new $670,000 DOE contract for Joel Rose's blended/computerized math instruction program, which was originally promised to be provided free to NYC schools in perpetuity since Rose developed it with about a million dollars of city funds while at DOE. The Panel for Educational Policy is due to vote on this contract on August 23.

Yet since his company New Classrooms went private, NYC has been charged many hundreds of thousands of dollars for this program with a highly questionable educational value.  Subsequently, Gary Rubinstein also wrote about the program's low quality, at different times called School of One or Teach to One.

All this cronyism is not an anomaly, sadly, but is consistent with the scandalously lax oversight of DOE contracts -- including a $1.1 billion contract proposed in 2015 for a contractor who had been involved in a kickback scheme just a few years ago --later thankfully rescinded by the Mayor's office and rebid for savings of between $163 million and $727 million.

On Friday, the NYC Comptroller Stringer released a follow-up audit showing how the DOE continues to violate  state law and its own rules when it comes to contracting.  More on this from the New York Daily News, New York Post and Chalkbeat.

From the Daily News:

In fiscal year 2016 the city school system handed out more than 500 contracts worth $2.7 billion without competitive bidding or required protections, Stringer's investigation found.
That’s 64% of the agency's total contract spending of more than $4 billion for that year....the city agency didn’t bother to evaluate vendors or keep track of their work — or to register contracts with Stringer’s office for review.

From the Comptroller's summary: 

A review of 521 limited competition contracts and contract actions revealed:

• DOE directed 442 of the vendors — approximately 85 percent — to begin work before the contract was even registered with the Comptroller’s Office, in violation of the New York State Education Law, the City Charter and DOE’s own rules;
• DOE submitted 302 new contracts and renewals to the Comptroller’s Office, on average, 258 days after their start dates, while 140 extensions of existing contracts were submitted on average 137 days after work began;
• In the most egregious case, a contract was submitted to the Comptroller’s Office 910 days — nearly two and a half years — after the vendor began work; and
• DOE submitted dozens of contracts to the Comptroller’s Office after the contract’s end date.

I myself discovered that of the 27 contracts approved at last month's Panel for Educational Policy meeting , 13 were retroactive—meaning at least some or all of the vendor's work had already started –  rendering the vote of the PEP pretty much irrelevant.  Contracts are only submitted and registered by the Comptroller after the PEP has voted. 

For the audit they specifically looked at how many contracts were registered retroactively.  Of 358 contracts and contract renewals, 84% of the vendors started work before registration,  an average of eight months prior to submission.  More than 10% of the contracts were submitted after the work had already ended.  In another data set of 163 contract extensions, 86% had started before submission,  an average of 137 days prior.


The Comptroller’s audit also found these continuing problems:

• DOE still does not provide sufficient guidance for contract assignments, and, in particular, it fails to ensure that a proposed new vendor’s capacity and history are adequately assessed. In addition, assignment files lack sufficient information justifying the assignments, detailing prior performance and documenting adequate insurance coverage, among other things.
• DOE has not developed written procedures or guidelines to help staff detect the warning signs of possible collusion when awarding contracts via actions other than fully competitive solicitations.
• While general guidance to contract managers appears to be available now in DOE’s contract management training manual, it does not offer the contract managers specific instruction about handling their responsibilities. For example, the training manual is silent on the performance evaluations that must be conducted throughout the contract period and on the frequency of such evaluations.
• Despite DOE’s representation in response to the previous audit that it would implement a standard format with standard criteria and ratings for evaluating vendor performance and a schedule for conducting such evaluations, we found that no such implementation has taken place.
• DOE still does not conduct required performance evaluations of contractors seeking renewals or extensions.
• DOE still does not adequately monitor its contract managers to ensure that contract monitoring and performance evaluations are conducted in accordance with its PPP manual.
• DOE does not ensure that limited-competition and non-competitive contracts are registered with the Comptroller prior to vendors beginning performance. As a result, vendor payments are delayed, which could potentially decrease competition and raise prices.

David Ross

The question of evaluations is key; many DOE contracts are renewed several times without any evidence that any assessments had been done of the quality of their previous work.  Even when an evaluation was done, in the case of New Classrooms for example, which showed no significant gains  through a randomized experiment, this study was unmentioned in the DOE Request for Authorization document, which asked the PEP for approval.

All these problems are not new, but occurred during the previous administration as well, under the aegis of David Ross.  Ross spent 12 years at the DOE as chief procurement officer,  and is now going to oversee contracts at the MTA – another troubled government agency that cannot afford to waste a single penny of public funds. Will the new contract head at DOE clean up this horrible mess?  We will have to see, but from the weak responses to the audit from DOE, I would not bet on it.

The person ultimately in charge however is Chancellor Fariña.  We've had four Chancellors in a row who were lax in terms of allowing the waste of  millions of dollars in taxpayer money, while more than 300,000 NYC children continue to languish in classes of 30 or more.  Will Fariña stick around after the election?  And if not, who will replace her?