John Pane of RAND writes to correct my post on Teach to One and my response

About two weeks ago, I posted a history of the program Teach to One (TtO): how it had first been developed in NYC Department of Education as a blended learning math program called School of One, how after it had spun off from DOE as a separate company called New Classrooms, the developer Joel Rose had promised never to charge NYC schools a fee to use it, instead granting them with a “royalty free, perpetual, non-exclusive license”, but then how the company has continued to charge a license fee to NYC schools anyway. The main focus of the piece was to describe how the huge hype surrounding the Teach to One program and the suppression of the findings of negative or null evaluations of its results has allowed it to expand to more schools, despite disappointing  results and a 60 percent school attrition rate.

In a single paragraph towards the end of this rather lengthy post,  I summarized the findings of a RAND report on the Next Generation Learning Challenge (NGLC) schools, assuming that schools using Teach to One were part of the evaluation, since TtO is a grantee of the NGLC program.  Diane Ravitch subsequently ran excerpts of my blog on hers.

John Pane, the lead researcher on the Rand report, wrote to Diane that New Classrooms / Teach to One was not one of the programs included in this evaluation.  I have posted a correction on that matter on my original blog post.   

He also critiqued the way I reported his remarks to Education Week about “personalized learning” schools in general, that “the evidence base is very weak at this point,” and said that the paragraph in which I described the results of the Rand report had “numerous false and misleading statements,” including my summary of survey results that suggest that the students at NGLC schools “were more likely to feel alienated and unsafe compared to matched students at similar schools.”

He has granted his permission to quote his letter in full below, which I have done, along with my response to the points in his letter.   

On Thu, Mar 7, 2019 at 12:17 PM Pane, John <jpane@rand.org> wrote:

Dear Diane, 

On March 4, 2018 you published this blog entry, “Leonie Haimson: Reality Vs. Hype in “Teach to One” Program,” excerpting from Leonie Haimson’s blog. Your excerpt included this paragraph about my own research (with colleagues) and my public statements:

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, and their students were more likely to feel alienated and unsafe compared to matched students at similar schools. The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“

 This paragraph by Haimson has numerous false and misleading statements. Here I summarize my critique, excerpting the original paragraph: 

“The most recent RAND analysis of schools that used personalized learning programs that received funding through the Next Generation Learning initiative, which have included both Summit and Teach to One, …”

None of the schools in our sample reported using Teach to One (TtO) among the 194 education technology products they mentioned. Our sample includes schools in the Next Generation Learning Challenges (NGLC) wave IIIa and wave IV programs, a subset of all the NGLC initiatives. Haimson points to blog posts by NGLC about Summit and TtO, but that does not mean our study included them.

“…included both Summit and Teach to One, concluded there were small and mostly insignificant gains in achievement at these schools, …”

Our conclusions were about the whole sample of schools, and did not single out any particular schools as is implied by juxtaposing “Summit and Teach to One” with “these schools.” Our concluding remarks related to achievement did not say “small and mostly insignificant.” What we actually said was, “Students in NGLC schools experienced positive achievement effects in mathematics and reading, although the effects were only statistically significant in mathematics. On average, students overcame gaps relative to national norms after two years in NGLC schools. Students at all levels of achievement relative to grade-level norms appeared to benefit. Results varied widely across schools and appeared strongest in the middle grades.” 

 “… and their students were more likely to feel alienated and unsafe compared to matched students at similar schools”

This was not a conclusion of our report. In a supplemental appendix we did compare results from our sample (again, the whole sample of schools in the study, none of which reported using TtO) to a national sample. Our method did not use “matched students at similar schools.” Given data limitations, we were able to make the student samples similar (through weighting) only on grade level, gender, and broad classifications of geographic locale (e.g., urban vs. suburban). Even after weighting, we suspect the high-minority, high-poverty schools in the NGLC sample may be located in more distressed communities than the national survey counterparts, and that this could be related to feelings of safety. Indeed, fewer NGLC students (78 vs. 82 percent) agreed that “I feel safe in this school,” but this small difference cannot be attributed to personalized learning and has no direct relevance to TtO. None of our survey items or reports used the word “alienated.” Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” and “I feel good about being in this school,” 76 percent agreed that “I care about this school” and 72 percent agreed “I am an important part of my school community.” 

The overall results caused John Pane, the lead RAND researcher, to say to Ed Week that ‘the evidence base [for these schools] is very weak at this point.’“ 

This EdWeek article clearly states that it is about “what K-12 educators and policymakers need to know about the research on personalized learning” broadly. Quoting accurately, “RAND has found some positive results, including modest achievement gains in some of the Gates-funded personalized-learning schools. But overall, ‘the evidence base is very weak at this point, Pane said.” There is no justification for Haimson to insert “[for these schools]” into my quoted remark. It appears as though Haimson is attempting to give a misleading impression that I was specifically talking about Summit and TtO rather than the entire body of personalized learning research. 

I find it very unfortunate that you accepted Haimson’s claims without fact checking, and increased their visibility and attention through your own platform. 

I am requesting that you please issue a correction in a way that previous readers of your March 4 post will likely notice. You may include this letter if you wish.

 With regards,

 John Pane, RAND Corporation

____

My response:  

I have now posted a correction on my blog post about the NGLC report – which  was only a small part of my post on TTO which is here .

 It is unfortunate that the names of the specific online program that were evaluated were left out of the RAND evaluation.  I had wrongly assumed that  TTO was included, since it is one of the most heavily funded and promoted by Gates and others of the Next Generation Learning Challenge “personalized learning” programs. 

I would also like to point out that the following survey statistics that John Pane includes in his letter about the NGLC schools omit the results from the comparison schools, as cited in the appendix of the Rand  report:

Possibly related, 77 percent of NGLC students agreed that “at least one adult in this school knows me well” [compared to 86% of the national sample] and “I feel good about being in this school,” [vs. 89% of the national sample] 76 percent agreed that “I care about this school” [vs. 87% of the national sample] and 72 percent agreed “I am an important part of my school community.” [compared to 79% of the national sample.]

In addition, the  students at the NGLC personalized learning schools were more likely to say that that “their classes do not keep their attention, and they get bored” compared to the national sample (30% to 23%). Only 35% of students at the NGLC schools said that “learning is enjoyable” compared to 45% of the national sample. With results like this it is difficult to see what was wrong with my statement that students at these schools were more likely to feel alienated and unsafe.

Now we know that TtO students aren’t included in these surveys, but there is no reason to assume that their responses would be significantly different until and unless New Classrooms releases their own survey results.  And we do know that from the survey of students at the Mountain View school, which used TtO, showed a 413% increase in the number of students who said they hated math as a result.

 Nor does John Pane’s response relate to the larger question which I discussed in my post, about how problematic it is to use MAP scores to evaluate these programs, especially scores from students  that aren’t disaggregated by race or economic status.  One might expect that with all the data that NWEA has by now they would have done that by now.

Finally, it is extremely unfortunate that Gates, Zuckerberg etc. haven’t bothered to commission any truly randomized  small-scale evaluation of Summit Learning, TtO or any of the other personalized learning programs that they so heavily fund and promote before expanding their reach and subjecting hundreds of thousands of students to them.   Summit has rejected  any independent evaluation of its results.  One can only speculate why.