John Pane of RAND writes to correct my post on Teach to One and my response

About two weeks ago, I posted a history of the program Teach to One (TtO): how it had first been developed in NYC Department of Education as a blended learning math program called School of One, how after it had spun off from DOE as a separate company called New Classrooms, the developer Joel Rose had promised never to charge NYC schools a fee to use it, instead granting them with a “royalty free, perpetual, non-exclusive license”, but then how the company has continued to charge a license fee to NYC schools anyway. The main focus of the piece was to describe how the huge hype surrounding the Teach to One program and the suppression of the findings of negative or null evaluations of its results has allowed it to expand to more schools, despite disappointing  results and a 60 percent school attrition rate.

In a single paragraph towards the end of this rather lengthy post,  I summarized the findings of a RAND report on the Next Generation Learning Challenge (NGLC) schools, assuming that schools using Teach to One were part of the evaluation, since TtO is a grantee of the NGLC program.  Diane Ravitch subsequently ran excerpts of my blog on hers.

John Pane, the lead researcher on the Rand report, wrote to Diane that New Classrooms / Teach to One was not one of the programs included in this evaluation.  I have posted a correction on that matter on my original blog post.   

He also critiqued the way I reported his remarks to Education Week about “personalized learning” schools in general, that “the evidence base is very weak at this point,” and said that the paragraph in which I described the results of the Rand report had “numerous false and misleading statements,” including my summary of survey results that suggest that the students at NGLC schools “were more likely to feel alienated and unsafe compared to matched students at similar schools.”

He has granted his permission to quote his letter in full below, which I have done, along with my response to the points in his letter.   

On Thu, Mar 7, 2019 at 12:17 PM Pane, John <jpane@rand.org> wrote:

Dear Diane, 

On March 4, 2018 you published this blog entry, “Leonie Haimson: Reality Vs. Hype in “Teach to One” Program,” excerpting from Leonie Haimson’s blog. Your excerpt included this paragraph about my own research (with colleagues) and my public statements:

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, and their students were more likely to feel alienated and unsafe compared to matched students at similar schools. The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“

 This paragraph by Haimson has numerous false and misleading statements. Here I summarize my critique, excerpting the original paragraph: 

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, …”

None of the schools in our sample reported using Teach to One (TtO) among the 194 education technology products they mentioned. Our sample includes schools in the Next Generation Learning Challenges (NGLC) wave IIIa and wave IV programs, a subset of all the NGLC initiatives. Haimson points to blog posts by NGLC about Summit and TtO, but that does not mean our study included them.

“…included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, …”

Our conclusions were about the whole sample of schools, and did not single out any particular schools as is implied by juxtaposing “Summit and Teach to One” with “these schools.” Our concluding remarks related to achievement did not say “small and mostly insignificant.” What we actually said was, “Students in NGLC schools experienced positive achievement effects in mathematics and reading, although the effects were only statistically significant in mathematics. On average, students overcame gaps relative to national norms after two years in NGLC schools. Students at all levels of achievement relative to grade-level norms appeared to benefit. Results varied widely across schools and appeared strongest in the middle grades.” 

 “… and their students were more likely to feel alienated and unsafe compared to matched students at similar schools”

This was not a conclusion of our report. In a supplemental appendix we did compare results from our sample (again, the whole sample of schools in the study, none of which reported using TtO) to a national sample. Our method did not use “matched students at similar schools.” Given data limitations, we were able to make the student samples similar (through weighting) only on grade level, gender, and broad classifications of geographic locale (e.g., urban vs. suburban). Even after weighting, we suspect the high-minority, high-poverty schools in the NGLC sample may be located in more distressed communities than the national survey counterparts, and that this could be related to feelings of safety. Indeed, fewer NGLC students (78 vs. 82 percent) agreed that “I feel safe in this school,” but this small difference cannot be attributed to personalized learning and has no direct relevance to TtO. None of our survey items or reports used the word “alienated.” Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” and “I feel good about being in this school,” 76 percent agreed that “I care about this school” and 72 percent agreed “I am an important part of my school community.” 

The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“ 

This EdWeek article clearly states that it is about “what K-12 educators and policymakers need to know about the research on personalized learning” broadly. Quoting accurately, “RAND has found some positive results, including modest achievement gains in some of the Gates-funded personalized-learning schools. But overall, ‘the evidence base is very weak at this point, Pane said.” There is no justification for Haimson to insert “[for these schools]” into my quoted remark. It appears as though Haimson is attempting to give a misleading impression that I was specifically talking about Summit and TtO rather than the entire body of personalized learning research. 

I find it very unfortunate that you accepted Haimson’s claims without fact checking, and increased their visibility and attention through your own platform. 

I am requesting that you please issue a correction in a way that previous readers of your March 4 post will likely notice. You may include this letter if you wish.

 With regards,

 John Pane, RAND Corporation

____

My response:  

I have now posted a correction on my blog post about the NGLC report – which  was only a small part of my post on TTO which is here .

 It is unfortunate that the names of the specific online program that were evaluated were left out of the RAND evaluation.  I had wrongly assumed that  TTO was included, since it is one of the most heavily funded and promoted by Gates and others of the Next Generation Learning Challenge “personalized learning” programs. 

I would also like to point out that the following survey statistics that John Pane includes in his letter about the NGLC schools omit the results from the comparison schools, as cited in the appendix of the Rand  report:

Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” [compared to 86% of the national sample] and “I feel good about being in this school,” [vs. 89% of the national sample] 76 percent agreed that “I care about this school” [vs. 87% of the national sample] and 72 percent agreed “I am an important part of my school community.” [compared to 79% of the national sample.]

In addition, the  students at the NGLC personalized learning schools were more likely to say that that “their classes do not keep their attention, and they get bored” compared to the national sample (30% to 23%). Only 35% of students at the NGLC schools said that “learning is enjoyable” compared to 45% of the national sample. With results like this it is difficult to see what was wrong with my statement that students at these schools were more likely to feel alienated and unsafe.

Now we know that TtO students aren’t included in these surveys, but there is no reason to assume that their responses would be significantly different until and unless New Classrooms releases their own survey results.  And we do know that from the survey of students at the Mountain View school, which used TtO, showed a 413% increase in the number of students who said they hated math as a result.

 Nor does John Pane’s response relate to the larger question which I discussed in my post, about how problematic it is to use MAP scores to evaluate these programs, especially scores from students  that aren’t disaggregated by race or economic status.  One might expect that with all the data that NWEA has by now they would have done that by now.

Finally, it is extremely unfortunate that Gates, Zuckerberg etc. haven’t bothered to commission any truly randomized  small-scale evaluation of Summit Learning, TtO or any of the other personalized learning programs that they so heavily fund and promote before expanding their reach and subjecting hundreds of thousands of students to them.   Summit has rejected  any independent evaluation of its results.  One can only speculate why.

Bob Hughes, now at Gates & formerly New Visions, provides $14M to New Visions & $2M to Jim Liebman for "evaluation"

Last week, Bob Hughes, appointed director of the Gates Foundation K12 division in 2016, made his first big move.   

He announced $92 million in grants for the new Networks of School Improvement initiative to be given to 19 organizations, collaboratives and districts, out of 532 applications submitted.   

New Visions, the NYC-based organization that Hughes ran before coming to the Foundation, received the second largest grant at $14 million – to work with 75 NYC schools, as yet unidentified.

The grant was more than the amount given to the entire Baltimore system of public schools – despite New Visions’ spotty record.

Though Hughes admitted that there’s not much evidence behind the theory of network improvement, he’s determined to push forward nonetheless:

 “I don’t think the research base is fully developed, and that’s one reason we’re making these investments,” said Hughes.

Asked by EdWeek reporter Steven Sawchuk how the results of this new initiative would be evaluated,

 Hughes replied that “ the foundation is still formulating its research approach.” And:  

"We don't have details for you, but we remain deeply committed to a third party evaluation of all our work and transparency about the results of those evaluations so we can enable the field to understand what we do well and what we don't do well," he said. 

Yet it appears that The Center for Public Research and Leadership (CPRL) at Columbia Law School has already been chosen by Hughes to evaluate the networks initiative.

As the CPRL website notes, “In January 2018, CPRL received a two and one-half year grant to report on the research underlying the NSI [Networks for School Improvement] initiative and to use the research to design and conduct a formative evaluation of the initiative’s initial implementation.”

(Sure enough, the Gates Foundation lists a grant for $1.9 million over 31 months to be awarded “Columbia University”  to "support evaluation”.)

The first Gates-funded CPRL study was a literature review of network impacts.  The findings were described by Sawchuk this way:

A Gates-commissioned review of the research on the topic from Columbia University's Center for Public Research and Leadership noted that there are more studies on the norms and conditions needed to support healthy networks than on how they affect K-12 outcomes; most of the 34 studies were case studies or qualitative, rather than quasi-experimental designs that sought to answer cause-and-effect questions.

CPRL is headed by Columbia Law professor James Liebman, who was appointed head of the NYC Department of Education’s Accountability Office under Joel Klein, despite the fact that he had no K12 education experience either as a teacher, administrator or researcher.   

Liebman made a mess of the School Progress Reports at DOE, instituting  a volatile, unstable system in which school grades wildly veered from year to year.  A blog post by Professor Aaron Pallas in Edweek was memorably entitled, “Could a Monkey Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education?”  Under Liebman’s direction, DOE efforts were statistically inept and I would not trust his ability to undertake a credible  evaluation.

 Liebman also commissioned the  expensive ARIS data system, which lived up to none of its promises.  It was rarely used by parents or teachers and was finally ditched in 2015 after costing the city $95 million.

In any case, I hope the Gates Foundation has not decided against commissioning an evaluation from more experienced, credible organization like RAND.  

RAND recently released a highly critical analysis of the results of the Gates-funded Teacher Evaluation Initiative and before that, a skeptical evaluation of the Gates-funded NextGeneration Learning Challenge schools, those  that feature  “personalized [online] learning.”

John F. Pane, senior scientist at RAND and the chief author of the latter study frankly pointed out to Ed Week, the evidence base for personalized learning is still "very weak."

Hughes himself doesn’t have the greatest reputation for transparency.  In 2005, he tried to suppress a Gates-funded research study that contained negative findings about the  New Visions Gates-funded small schools initiative in New York City, a study that was subsequently leaked to the NY Times .  

In 2007, it was revealed that New Visions threatened these small schools that they would not receive their full Gates grants unless they chose New Visions as their DOE “partnership support network” and paid the organization a fee in return.

"I thought, 'Oh, my God, what a huge conflict of interest,'" a principal said. "We have to join their PSO and pay them for support in order to get this grant that we qualified for?"

Only time will tell, but the hints of insular cronyism in these decisions by Hughes to award grants to New Visions and to Jim Liebman’s outfit do not bode well for the future.