Thursday, January 26, 2012

Concerns with the MDRC study on small schools released today

MDRC released a  study today, which the NY Times writes “appeared to validate the Bloomberg administration’s decade-long push to create small schools to replace larger, failing high schools.” The report mentions the current controversy over the massive number of school closings, here in NYC and across the country, and thus there may be a political element in the timing of its release:

MDRC’s findings about SSCs are relevant to current federal policy on high school reform, particularly the U. S. Department of Education’s School Improvement Grants (SIGs) for failing schools. Reforms funded by SIGs include school transformation, school restart, school closing, and school turnaround. SSCs straddle several of these categories since they are typically replacements for schools that have closed and they operate as regular public schools.

This is the second MDRC study to conclude that students who attended the new small schools had significantly improved outcomes.  The first MDRC study, released in 2010, looked at students who entered these schools in 2005; this one adds students who entered in 2006 to that group.

I have a lot of reservations about using this study or the previous study to justify the small schools initiative and especially to justify the current massive round of school closings.  I am no expert in statistics, but my concerns revolve around these issues: 

1-      Though the study points out that the small schools were supposedly “unscreened” and evaluates their results by comparing the outcomes of students who applied to the school through lotteries,  compared to those who lost the lottery, it  ignores that the students who attended these small schools were far less high-needs on average, as evidence by their lower rates of English language learners and special needs, as shown by this Annenberg study by Pallas and Jennings.  In fact, they were allowed to openly exclude special needs students during the first two years. Thus even with a “lottery” for admissions, there are substantial peer effects for students who are grouped with higher-achieving students which this study does not mention.  (This is also a problem with many of the charter school studies, like this one, which tend to ignore peer effects.)

2-      The results in terms of higher graduation rates and college readiness (based on Regents scores and credit accumulation) ignore how in NYC, teachers and principals are able manipulate these in ways that do not reflect real learning (especially as teachers grade the Regents of students in their own schools, and in these schools, their own students!)  It has also been alleged that the small schools pioneered the now widespread and largely discredited practice of “credit recovery.” With newer teachers at many of the small schools, who did not have a memory or tradition as to earlier practices, it may have been easier to pressure these teachers into employing such methods.

3-      The study ignores that the small schools on average were allowed to have smaller classes and were far less overcrowded than the large high schools, which legitimately could have led to better results.  The class size at the small schools during these years were from 13 to 20 students per class, according to the PSA first year report, compared to 30 or more at the larger schools.  If the higher needs students in the larger schools had been provided with smaller classes, very likely their chance of success would have been improved substantially as well.

4-      Yet the study doesn’t examine how the opening of the small schools had a negative impact on the system as a whole, by flooding nearly large high schools with the most disadvantaged and academically challenged students, leading to even more overcrowding, larger class sizes, and damaging their opportunity to learn, as reported by many observers and confirmed by the New School’s report, The New Marketplace.

5-      The MDRC study deals with only a subsection of the small schools that were oversubscribed and required a lottery for admissions, so like the charter school studies which use a similar methodology, their success rate may not be representative of small schools overall.
6-      It study ignores the reality that these so-called random “lotteries” may be far from actually random. The MDRC study compares the baseline characteristics of students who “won” the lotteries to those who lost, in a  Supplemental Table 1, which purports to show that both groups were “virtually identical”; but the comparison does not include students who required collaborative team-teaching or self-contained classes; does not include  prior attendance rates, a key factor that principals often examing when selecting students; and does not differentiate between free and reduced price lunch students.  The table also does not include data on previous rates of suspension.  According to an earlier evaluation done by Policy Studies Associates. the ninth graders who entered the small schools had far better attendance records (91% compared to 81%), and were less likely to have been suspended as 8th graders compared to students at the schools they replaced.

Even in the categories the MRDC study does compare, there are greater numbers of higher achieving students, though the differences are not statistically significant, according to the model used.   Most strangely, the table compares the baseline data for four cohorts – entering ninth-graders from the 2004-2005 to 2007-2008 years; while the report compares outcomes for only the first two of these cohorts, students who entered these schools from 2005-6 and 2006-7. This is despite the fact that that the need level of students increased significantly after that point – and likely, the challenges faced by these schools as well.  See the above chart, for example, from the Annenberg study by Jennings and Pallas.

I have no idea why the MRDC study lumped together all these cohorts to examine their baseline characteristics, even as they only compared the outcomes for the first two, but it may considerably bias their conclusions.  Indeed, more than half of the middle and high schools being closed by the DOE this year for poor results are small schools that were founded after 2003.

7- There are several ways in which, especially in the early years of the small schools initiative, principals were able to manipulate the admissions process to get the students who were more likely to succeed, even though by definition these schools were supposedly “unscreened” and used “random” lotteries for admissions.  See this excerpt from a published study by Jennifer Jennings, who embedded herself with three small schools between March 2004 and September 2005: 

My observations revealed that many schools used applications, mandatory information sessions, and much stronger language to deter unwanted applicants. For example, 12 unscreened schools shared a similar application requiring that students provide the most recent report card and two letters of recommendation, one from an eighth-grade teacher and one from a guidance counselor, assistant principal, or principal. The application also asked for the student’s test scores, retention history, and involvement in advanced courses during the eighth grade. Finally, the application included additional questions requiring a narrative response….
The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. ….
The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. This data file included each student’s English-language-learner and special education classification, reading and math test scores, absences, grades, address, and junior high school. Schools were told to identify students who made an ‘‘informed choice’’ by assigning them a 1, while students who did not make an informed choice but the school was willing to accept were assigned a 2. If the school did not fill all of its seats with students making an informed choice, additional seats would be filled by students in the second category.  The Department of Education prohibits unscreened schools from using student performance data to select students. Nonetheless, both Marlena and Anna [pseudonyms for two principals of small schools] learned through their relationships with other principals that such regulations were loosely enforced….
 In addition to the English language learners and full-time special education students whom new schools had a waiver to eliminate, Renaissance [pseudonym for one of these small schools] eliminated part-time special education students and chose only those with 90 percent or higher attendance. Excel eliminated full- and part-time special education students and chose students with attendance rates of 93 percent or higher. 
There are many more revealing insights in the Jennings study about how the small schools were able to deflect over-the-counter students and counsel out low-performers to achieve better results, despite the fact that they claimed to be non-selective.  (Update: I have removed the link to the paper at the author's request; the abstract is here.)
All of these concerns  should provide one with reservations about the validity of the MDRC study and especially its apparent endorsement of the mayor's policies.


Anonymous said...

MDRC does not even mean anything...They are a shell corporation set up to undermine public education and they don't even have a name, the letters standvfor nothing,
Very appropriate..

Anonymous said...

Leoine, I love your blog and your posts but I am afraid that your lengthy, unrelenting rebuttal to this study makes you appear less credible and opposed to any and all evidence that is contrary to your views no matter the cost.

By studying students that were accepted and rejected in these schools, MDRC can prove the statistical impact each of these small schools had on each student's performance. My honest take is that as inconvenient as it may be, it absolutely proves that these small schools serves these students better than other alterrnatives which is a very plausible finding that quite frankly, makes a lot of sense given that these are exceptionally small schools.

Even if the applicants to small schools are less needy than students that apply to large high schools, this does not negate the fact that the same-profile student that was admitted, out-performed the same-profile student that did not get in and went to another school instead.

I'm afraid that your points come across as hyptothetical, unproven conspiracy theories supported by facts that do raise suspicion but don't prove anything.

My question to you is: if this study is proved to be conducted in a legitimate fashion, is there a scenario where you would accept its results and final conclusions? Or is this not a possibility for you?

Leonie Haimson said...

Anonymous 2, simply saying so does not make it so. I think there are many problems with this study and I pointed them out. If you have an argument with any of these points, make it.

Anonymous said...

Leonie, thanks for writing this. Public schools should not screen kids - period, the end. I can only offer the example of my neighborhood - Hell's Kitchen. Midtown West - P.S. 212 is a choice school. Parents fill out an application to get in. Why does a public school require an application - I don't know. In order to get the application you must take a school tour. So immediately you are screening out parents who can't take time off to go on a tour which is given during working hours. I can tell you the results of this screening process. Midtown West is overwhelmingly white and Asian. It's overwhelmingly middle class. Most of the Latino and Arabic kids in our neighborhood go to the two zoned schools - P.S. 51 and P.S. 111. This puts a tremendous burden on the resources of these two schools. What is the effect of having Midtown West in our neighborhood? It siphons off the best students and the most engaged parents. Look at the statistics for the schools in the Hell's Kitchen neighborhood and you will see Midtown West has 1/3 of the Ells the other schools have and 1/3 fewer kids who qualify for free lunch. Midtown West asks parents to pay a whopping $750 each year for extra programs for the kids and the parents can pay. It runs an on-line auction each year which raises over 200K. Why is this allowed in a public school system? It's only one example but this type of chicanery is going on all over the city. It's disgraceful.

Ready said...

Anonymous 2 said "[the study] absolutely proves that these small schools serves these students better than other alterrnatives"

How does Anonymous and the report account for large high schools like Midwood, Stuyvesant, Brooklyn Tech, Bronx Science, Townsend Harris, Francis Lewis, etc?

It's not the size (I'll avoid quoting Yoda here); it's the quality of the programming and the students.

Anonymous said...

I teach in a 3yr old small school in the Brandeis campus.. Since our existence class sizes have hovered around 30 or more and we have the highest percentage of special ed and ells in our peer group. Eva moved her elementary school into the 2nd floor of our building. This year we were supposed to get 4 classrooms as part of our 3rd yr expansion. We were also supposed to take in 108-115 freshman.. The DOE sent us a whopping 162 of which 23% were Special ed and ELLS. On top of that, some these students required services we could not provide such as self contained. This is largley due to space. We simply do not have available classrooms because the simpletons at the DOE think you make classes by dividing 4 rooms into 110 students. Clearly they have never programmed a student body.. Anyway I'm ranting but when I read the times article I knew they had not looked at my school because we have never been allowed to exclude special ed kids. In fact our CFN has advocated we change IEP's to fit what our school can offer instead of what students need. Maybe my principal didn't believe it was right to turn students away.. Oh well I'm sure no good deed will go unpunished and we"ll be targeted for destruction so Eva can expand.

Campus Management Software said...
This comment has been removed by a blog administrator.