About two weeks
ago, I posted a history of the program
Teach to One (TtO): how it had first been developed in NYC Department of
Education as a blended learning math program called School of One, how after it
had spun off from DOE as a separate company called New Classrooms, the
developer Joel Rose had promised never to charge NYC schools a fee to use it, instead
granting them with a “royalty free, perpetual, non-exclusive
license”, but then how the company has continued to charge a license
fee to NYC schools anyway. The main focus of the piece was to describe how the huge hype surrounding the Teach to One program and
the suppression of the findings of negative or null evaluations of its results has allowed it to expand to more
schools, despite disappointing results
and a 60 percent school attrition rate.
In a single paragraph towards
the end of this rather lengthy post, I summarized the findings of a RAND report on the Next Generation
Learning Challenge (NGLC) schools, assuming that schools using Teach to One were part of the evaluation, since TtO is a grantee of the NGLC program. Diane Ravitch subsequently ran excerpts
of my blog on hers.
John Pane, the lead researcher on the Rand report, wrote
to Diane that New Classrooms / Teach to One was not one of the programs included in this evaluation. I have posted a correction on that matter on my original blog post.
He also critiqued the way I reported
his remarks to Education Week about “personalized
learning” schools in general, that “the
evidence base is very weak at this point,” and said that the paragraph in which I described the results
of the Rand report had “numerous false and misleading statements,” including my summary of survey
results that suggest that the students at NGLC schools “were more likely to feel
alienated and unsafe compared to matched students at similar schools.”
He has granted his permission to quote his
letter in full below, which I have done, along with my response to the points
in his letter.
On Thu, Mar 7,
2019 at 12:17 PM Pane, John <jpane@rand.org>
wrote:
Dear Diane,
On March 4, 2018 you
published this blog entry, “Leonie Haimson: Reality Vs. Hype in
“Teach to One” Program,” excerpting from Leonie Haimson’s blog. Your excerpt included this
paragraph about my own research (with colleagues) and my public statements:
“The most recent RAND analysis of schools that used
personalized learning programs that received funding through the Next
Generation Learning initiative, which have included both Summit and Teach to One, concluded there were
small and mostly insignificant gains in achievement at these schools, and their
students were more likely to feel alienated and unsafe compared to matched
students at similar schools. The overall results caused John Pane, the
lead RAND researcher, to say to Ed Week that ‘the
evidence base [for these schools] is very weak at this point.’“
This paragraph by
Haimson has numerous false and misleading statements. Here I summarize my
critique, excerpting the original paragraph:
“The most recent RAND analysis of schools that used
personalized learning programs that received funding through the Next
Generation Learning initiative, which have included both Summit and Teach to One, …”
None of the schools in our
sample reported using Teach to One (TtO) among the 194 education technology
products they mentioned. Our sample includes schools in the Next Generation
Learning Challenges (NGLC) wave IIIa and wave IV programs, a subset of all the
NGLC initiatives. Haimson points to blog posts by NGLC about Summit and TtO,
but that does not mean our study included them.
“…included both Summit and Teach to One, concluded there were
small and mostly insignificant gains in achievement at these schools, …”
Our conclusions were about
the whole sample of schools, and did not single out any particular schools as
is implied by juxtaposing “Summit and Teach to One” with “these schools.” Our
concluding remarks related to achievement did not say “small and mostly
insignificant.” What we actually said was, “Students in NGLC schools
experienced positive achievement effects in mathematics and reading, although
the effects were only statistically significant in mathematics. On average,
students overcame gaps relative to national norms after two years in NGLC
schools. Students at all levels of achievement relative to grade-level norms
appeared to benefit. Results varied widely across schools and appeared
strongest in the middle grades.”
“… and their
students were more likely to feel alienated and unsafe compared to matched
students at similar schools”
This was not a conclusion
of our report. In a supplemental appendix we did compare results from our
sample (again, the whole sample of schools in the study, none of which reported
using TtO) to a national sample. Our method did not use “matched students at
similar schools.” Given data limitations, we were able to make the student samples
similar (through weighting) only on grade level, gender, and broad
classifications of geographic locale (e.g., urban vs. suburban). Even after
weighting, we suspect the high-minority, high-poverty schools in the NGLC
sample may be located in more distressed communities than the national survey
counterparts, and that this could be related to feelings of safety. Indeed,
fewer NGLC students (78 vs. 82 percent) agreed that “I feel safe in this
school,” but this small difference cannot be attributed to personalized
learning and has no direct relevance to TtO. None of our survey items or
reports used the word “alienated.” Possibly related, 77 percent of NGLC
students agreed that “at least one adult in this school knows me well” and “I
feel good about being in this school,” 76 percent agreed that “I care about
this school” and 72 percent agreed “I am an important part of my school
community.”
The overall results caused
John Pane, the lead RAND researcher, to say to Ed Week that ‘the
evidence base [for these schools] is very weak at this point.’“
This EdWeek article clearly
states that it is about “what K-12 educators and policymakers need to know
about the research on personalized learning” broadly. Quoting accurately, “RAND
has found some positive results, including modest achievement gains in some of
the Gates-funded personalized-learning schools. But
overall, ‘the evidence base is very weak at this point, Pane said.” There is no
justification for Haimson to insert “[for these schools]” into my quoted
remark. It appears as though Haimson is attempting to give a misleading
impression that I was specifically talking about Summit and TtO rather than the
entire body of personalized learning research.
I find it very unfortunate
that you accepted Haimson’s claims without fact checking, and increased their
visibility and attention through your own platform.
I am requesting that you
please issue a correction in a way that previous readers of your March 4 post will
likely notice. You may include this letter if you wish.
With regards,
John Pane, RAND
Corporation
____
My response:
I have now posted a correction on my blog post
about the NGLC report – which was only a
small part of my post on TTO which
is here .
It is
unfortunate that the names of the specific online program that were evaluated were
left out of the RAND evaluation. I had wrongly assumed that TTO was
included, since it is one of the most heavily funded and promoted by Gates and
others of the Next Generation Learning Challenge “personalized learning”
programs.
I would also like
to point out that the following survey statistics that John Pane includes in
his letter about the NGLC schools omit the results from the comparison schools,
as cited in the appendix
of the Rand report:
Possibly
related, 77 percent of NGLC students agreed that “at least one adult in this
school knows me well” [compared to 86% of the national sample] and “I feel good
about being in this school,” [vs. 89% of the national sample] 76 percent
agreed that “I care about this school” [vs. 87% of the national sample]
and 72 percent agreed “I am an important part of my school community.”
[compared to 79% of the national sample.]
In addition, the
students at the NGLC personalized learning schools were more likely to say
that that “their classes do not keep their attention, and they get bored”
compared to the national sample (30% to 23%). Only 35% of students at the NGLC
schools said that “learning is enjoyable” compared to 45% of the national
sample. With results like this it is difficult to see what was wrong with my statement
that students at these schools were more likely to feel alienated and unsafe.
Now we know that
TtO students aren’t included in these surveys, but there is no reason to assume
that their responses would be significantly different until and unless New
Classrooms releases their own survey results. And we do know that from
the survey of students at the Mountain
View school, which used TtO, showed a 413% increase in the number of students
who said they hated math as a result.
Nor does John
Pane’s response relate to the larger question which I discussed in my post,
about how problematic it is to use MAP scores to evaluate these programs,
especially scores from students that aren’t disaggregated by race or
economic status. One might expect that
with all the data that NWEA has by now they would have done that by now.
Finally, it is
extremely unfortunate that Gates, Zuckerberg etc. haven’t bothered to
commission any truly randomized small-scale evaluation of Summit Learning,
TtO or any of the other personalized learning programs that they so heavily fund and
promote before expanding their reach and subjecting hundreds of thousands of
students to them. Summit has rejected
any independent evaluation of its results. One can only
speculate why.
1 comment:
Good for you. Hold your ground. Your thoughts and conclusions are clearly expressed and written in an open an honest way. Theirs are not. It's really that simple, anyone with an open mind can see how they attempt to mislead. You called them out and they don't like that. Ha, Ha. Keep up the good reporting and thorough analysis. Your work and efforts are appreciated.
Post a Comment