John Pane of RAND writes to correct my post on Teach to One and my response

About two weeks ago, I posted a history of the program Teach to One (TtO): how it had first been developed in NYC Department of Education as a blended learning math program called School of One, how after it had spun off from DOE as a separate company called New Classrooms, the developer Joel Rose had promised never to charge NYC schools a fee to use it, instead granting them with a “royalty free, perpetual, non-exclusive license”, but then how the company has continued to charge a license fee to NYC schools anyway. The main focus of the piece was to describe how the huge hype surrounding the Teach to One program and the suppression of the findings of negative or null evaluations of its results has allowed it to expand to more schools, despite disappointing  results and a 60 percent school attrition rate.

In a single paragraph towards the end of this rather lengthy post,  I summarized the findings of a RAND report on the Next Generation Learning Challenge (NGLC) schools, assuming that schools using Teach to One were part of the evaluation, since TtO is a grantee of the NGLC program.  Diane Ravitch subsequently ran excerpts of my blog on hers.

John Pane, the lead researcher on the Rand report, wrote to Diane that New Classrooms / Teach to One was not one of the programs included in this evaluation.  I have posted a correction on that matter on my original blog post.   

He also critiqued the way I reported his remarks to Education Week about “personalized learning” schools in general, that “the evidence base is very weak at this point,” and said that the paragraph in which I described the results of the Rand report had “numerous false and misleading statements,” including my summary of survey results that suggest that the students at NGLC schools “were more likely to feel alienated and unsafe compared to matched students at similar schools.”

He has granted his permission to quote his letter in full below, which I have done, along with my response to the points in his letter.   

On Thu, Mar 7, 2019 at 12:17 PM Pane, John <jpane@rand.org> wrote:

Dear Diane, 

On March 4, 2018 you published this blog entry, “Leonie Haimson: Reality Vs. Hype in “Teach to One” Program,” excerpting from Leonie Haimson’s blog. Your excerpt included this paragraph about my own research (with colleagues) and my public statements:

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, and their students were more likely to feel alienated and unsafe compared to matched students at similar schools. The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“

 This paragraph by Haimson has numerous false and misleading statements. Here I summarize my critique, excerpting the original paragraph: 

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, …”

None of the schools in our sample reported using Teach to One (TtO) among the 194 education technology products they mentioned. Our sample includes schools in the Next Generation Learning Challenges (NGLC) wave IIIa and wave IV programs, a subset of all the NGLC initiatives. Haimson points to blog posts by NGLC about Summit and TtO, but that does not mean our study included them.

“…included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, …”

Our conclusions were about the whole sample of schools, and did not single out any particular schools as is implied by juxtaposing “Summit and Teach to One” with “these schools.” Our concluding remarks related to achievement did not say “small and mostly insignificant.” What we actually said was, “Students in NGLC schools experienced positive achievement effects in mathematics and reading, although the effects were only statistically significant in mathematics. On average, students overcame gaps relative to national norms after two years in NGLC schools. Students at all levels of achievement relative to grade-level norms appeared to benefit. Results varied widely across schools and appeared strongest in the middle grades.” 

 “… and their students were more likely to feel alienated and unsafe compared to matched students at similar schools”

This was not a conclusion of our report. In a supplemental appendix we did compare results from our sample (again, the whole sample of schools in the study, none of which reported using TtO) to a national sample. Our method did not use “matched students at similar schools.” Given data limitations, we were able to make the student samples similar (through weighting) only on grade level, gender, and broad classifications of geographic locale (e.g., urban vs. suburban). Even after weighting, we suspect the high-minority, high-poverty schools in the NGLC sample may be located in more distressed communities than the national survey counterparts, and that this could be related to feelings of safety. Indeed, fewer NGLC students (78 vs. 82 percent) agreed that “I feel safe in this school,” but this small difference cannot be attributed to personalized learning and has no direct relevance to TtO. None of our survey items or reports used the word “alienated.” Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” and “I feel good about being in this school,” 76 percent agreed that “I care about this school” and 72 percent agreed “I am an important part of my school community.” 

The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“ 

This EdWeek article clearly states that it is about “what K-12 educators and policymakers need to know about the research on personalized learning” broadly. Quoting accurately, “RAND has found some positive results, including modest achievement gains in some of the Gates-funded personalized-learning schools. But overall, ‘the evidence base is very weak at this point, Pane said.” There is no justification for Haimson to insert “[for these schools]” into my quoted remark. It appears as though Haimson is attempting to give a misleading impression that I was specifically talking about Summit and TtO rather than the entire body of personalized learning research. 

I find it very unfortunate that you accepted Haimson’s claims without fact checking, and increased their visibility and attention through your own platform. 

I am requesting that you please issue a correction in a way that previous readers of your March 4 post will likely notice. You may include this letter if you wish.

 With regards,

 John Pane, RAND Corporation

____

My response:  

I have now posted a correction on my blog post about the NGLC report – which  was only a small part of my post on TTO which is here .

 It is unfortunate that the names of the specific online program that were evaluated were left out of the RAND evaluation.  I had wrongly assumed that  TTO was included, since it is one of the most heavily funded and promoted by Gates and others of the Next Generation Learning Challenge “personalized learning” programs. 

I would also like to point out that the following survey statistics that John Pane includes in his letter about the NGLC schools omit the results from the comparison schools, as cited in the appendix of the Rand  report:

Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” [compared to 86% of the national sample] and “I feel good about being in this school,” [vs. 89% of the national sample] 76 percent agreed that “I care about this school” [vs. 87% of the national sample] and 72 percent agreed “I am an important part of my school community.” [compared to 79% of the national sample.]

In addition, the  students at the NGLC personalized learning schools were more likely to say that that “their classes do not keep their attention, and they get bored” compared to the national sample (30% to 23%). Only 35% of students at the NGLC schools said that “learning is enjoyable” compared to 45% of the national sample. With results like this it is difficult to see what was wrong with my statement that students at these schools were more likely to feel alienated and unsafe.

Now we know that TtO students aren’t included in these surveys, but there is no reason to assume that their responses would be significantly different until and unless New Classrooms releases their own survey results.  And we do know that from the survey of students at the Mountain View school, which used TtO, showed a 413% increase in the number of students who said they hated math as a result.

 Nor does John Pane’s response relate to the larger question which I discussed in my post, about how problematic it is to use MAP scores to evaluate these programs, especially scores from students  that aren’t disaggregated by race or economic status.  One might expect that with all the data that NWEA has by now they would have done that by now.

Finally, it is extremely unfortunate that Gates, Zuckerberg etc. haven’t bothered to commission any truly randomized  small-scale evaluation of Summit Learning, TtO or any of the other personalized learning programs that they so heavily fund and promote before expanding their reach and subjecting hundreds of thousands of students to them.   Summit has rejected  any independent evaluation of its results.  One can only speculate why.

New Classrooms, crony contracts at DOE, and David Ross' new job

In the Daily News this week, Ben Chapman reported on a proposed contract for Joel Rose’s New Classrooms (formerly School of One) program which uses algorithms and computers to deliver so-called “personalized learning”.  This contract for as much as $670,000 is to be voted on at the August 23 PEP meeting. The DOE’s RA (Request for Authorization) says the company won a competitive bid, though the cost was higher than the other bidders – whose identities are not revealed: 

“While New Classrooms is a higher priced outlier for their license fees, they provided documentation demonstrating that this license fee encompasses a more extensive package of services than what other vendors have offered. On this basis, pricing can be determined to be fair and reasonable."

I have been tracking this program ever since Joel Rose was working at DOE and started School of One in a few NYC public schools in the summer of 2009.  In 2010-2011 School of One was implemented as a full-time mathematics program in three NYC middle schools, costing at least $3.3 million, with about a million dollars from the NYC Department of Education, and the rest provided by venture philanthropists, including the Robin Hood Foundation, the Dell Foundation, and the Gates Foundation.  Rose’s company was named “New Classrooms” and the program renamed “Teach to One” when the venture spun off as a separate non-profit company and introduced in other districts around the country.

I previously wrote about the numerous conflict of interest issues pertaining to this school, as well as some negative evaluations of its quality here, here and here, including a visit I took myself to a School of One classroom in Chinatown in 2012, which featured students listlessly answering multiple questions randomly until they got them right.  

Yet before it was even tried out as actual school-year program, Time magazine touted “the School of One” as one of the 50 top innovations of 2009.  It quickly garnered positive media in The Wall Street Journal, New York Times and Education Next.  Bill Gates repeatedly has praised the value of this program, most recently in a April 2016 speech in which he said that New Classrooms “represents the future not only of math, but a number of subjects.”  

The ability of Rose to take his program private and form a new company that would then contract with DOE was controversial, to say the least, given the conflict of interest laws.  When the first NYC contract was awarded in Jan. 2012, it was at no cost to the city, as the company promised that the DOE would be granted “joint ownership or a royalty-free perpetual non-exclusive license to the platform … for use in NYC schools."   (See p. 37 for this language in the  2012 RA document.) 

Even so, the awarding of the contract in 2012 generated negative comment, as noted by Rachael Monahan in the Daily News:

Under the agreement, the city will share with Rose’s groups the licenses for the School of One program, which uses computer-based learning to individually tailor math classes for students at three middle schools…“This is exactly the type of thing that raises eyebrows and causes people to question” the Education Department, said Michael Loughran, a spokesman for city Controller John Liu, whose office will review the contract....

 “We believe this zero-cost contract is a smart move for the city, potentially saving millions of dollars,” said spokesman Matt Mittenthal, defending the deal that aims to expand School of One to 50 schools.” 

Yet in 2014, as previously reported, DOE proposed a new, one year contract for $420,750 for New Classrooms – ignoring the fact that they had previously been promised a perpetual free license.  Nor did the DOE ever claim joint ownership. 

Since then, a new study has been released of the New Classrooms program in NYC  schools.  Jonah Rockoff and a team of researchers at Columbia Business School conducted an experimental randomized study – the gold standard --of the program in NYC schools.  The study concluded that “School of One had no statistically significant effects on student achievement—positive or negative–relative to traditional math instruction.” This evaluation is not mentioned by New Classrooms anywhere on its website, nor on the US Department of Education i3 website, which helped fund it.    

Instead, another study is trumpeted on the New Classrooms website – a non-randomized study by Doug Ready that found modest test score gains but included this caveat:

 

 “It is important to stress again that these findings cannot be attributed to TTO [Teach to One] without the use of experimental or quasi-experimental designs. In other words, we cannot state definitively that TTO caused the above-average achievement gains noted above.”     

Meanwhile, EdWeek reports that more than a quarter of the 53 schools that tried New Classrooms program are no longer using it. I would guess that the number is higher. Two out of the three NYC schools that were the first to try it in 2011-2012 school year, MS 131 in Manhattan and IS 339 in the Bronx, dropped it.   In Oct. 2014, the Daily News reported that six city school were using  the program,  by the 2016-2017 school year, only two schools were still involved: I.S. 228 and J.H.S. 88 in Brooklyn,  according to the New Classrooms website – the latter in one of its three academies.  This suggests that at least two-thirds of NYC schools that have tried the program have ditched it. 

Schools where New Classrooms operated 2016-2017

Nevertheless, with the backing of Bill Gates and an aggressive PR campaign, New Classrooms has aggressively expanded to nine other states – though the expansion has not been incident-free. In January 2017, two of the middle schools from the Mountain View Whisman School District in Silicon Valley abandoned the program, after having spent four months and more than half a million dollars on it – due to “avalanche” of parent opposition and poor interim test scores, according to the local paper.

A letter signed by 180 parents of Mountain View fifth and sixth graders observed that “topics are taught in an incoherent and seemingly random order, are riddled with mistakes and outright wrong answers, and students are frequently given math problems that are better-suited for ninth-graders and beyond.”   A survey revealed that 61% of the parents "said they do not believe the program matches the needs of their children," and the number of students who said they “hate math” sharply increased, from 7% to 29%. For more on what happened in Mountain View, see these articles in EdSurge, the Wall Street Journal and a parent blog here.  

 At the same time, it was revealed that the Mountain View district’s contract with New Classrooms had a “non-disparagement” clause, forbidding teachers or district officials from publicly criticizing it – something that was apparently banned from contracts in California in 2014 and by the federal government in 2017.

It would be very interesting to see if DOE’s contract or the contract that NYC schools signs with New Classrooms has a similar clause. In the past, New York City reporters who have tried to get comment from principals at the schools that have ceased using the program have been unable to do.  

All in all, it appears that DOE has continued paying for this program despite its disappointing results and conflicts of interest for no particular reason.  In the Daily News, I called these continued “crony contracts…inexplicable.”

Another dysfunctional company that spun off from DOE during the Bloomberg/Klein years is the highly problematic Leadership Academy , which still has a contract to provide coaching to principals through 2019, and which recently received a scathing audit by the NY State Comptroller, who found that the Academy had been paid without any evidence that the services had been rendered.

And all these problems with shady and wasteful contracts are symptomatic of an even larger problem at DOE – lax oversight.  For twelve years as chief procurement officer at DOE, David Ross has overseen hundreds of millions of education dollars that have been wasted on vendors who have engaged in fraud and corruption, including some of those described in my 2011 testimony before the City Council.

Ross’s career at DOE was capped off by the incredibly inflated five-year contract for $1.1 billion proposed for Computer Consultant Specialists in 2015 for internet wiring, extendable four more years at $2 billion—hurriedly reduced to $637 million overnight after news broke that this huge amount was to be awarded a company that had engaged in a fraudulent kickback scheme just a few years before. 

Friedlander and Ross at PEP meeting

See the video and the account on my blog where David Ross and Hal Friedlander, Chief Information Officer for DOE reassured the PEP members that this was the best possible contract at the best possible price, only to have it later kicked back by City Hall because of the resulting scandal, and rebid to other vendors for savings of between $163 million and $727 million – depending on what baseline is used.

As a result of the controversy provoked by this contract, DOE promised to the City Council to post full details of contracts within their Requests for Authorization (RAs) at least 30 days before every PEP vote, as Juan Gonzalez reported in the Daily News: “Tweed will even post information on all bids on its website 30 days before the scheduled vote by the panel, and has committed to do the same with other contracts.”

Yet in a May 2017 letter to the City Council Speaker Melissa Mark-Viverito, Ursulina Ramirez of DOE and Laura Anglin of the Mayor’s office said they would post this information online at least four weeks prior to the vote, “except where the DOE determines it is in the best interests of the DOE and the City to provide less notice…”  This is a significant backtrack from the DOE’s earlier promise. 

After 12 years, David Ross is finally leaving – but sadly, now going to head contracts at MTA, where he will likely waste many more millions, while Hal Friedlander, the other DOE official who pushed so relentlessly for the outrageous CCS contract and claimed they had gotten the best possible price for the contract, has started a new organization called Technology for Education Consortium.  This Gates-funded organization is focused on- get this – how to save school districts money on technology.  You just can’t make this stuff up.