NYC DOE still putting out false discharge data and inflating the graduation rate

Throughout the Bloomberg years, when the administration would trumpet rising graduation rates, I noticed how the discharge numbers were very high and seemed to be increasing. Every student listed as a “discharge” rather than a “dropout” can inflate a school’s figures,  as he or she are no longer counted in the cohort  -- in either the denominator or the numerator for the purposes of calculating the graduation rate.

So in 2009, I co-authored a report with Jennifer Jennings, entitled High School Discharges Revisited: Trends in NYC’s Discharge Rates, 2000-2007.   Our analysis showed that discharge rates had increased over this period, especially among Black and Hispanic students, students with disabilities, and English language learners.  

 Between the classes of 2000 and 2007, the discharge rate for students with disabilities increased from 17 to 23 percent, including in the class of 2005 where it spiked at 39 percent.  The report provided evidence that a thousand students had been “moved” into the special education cohort that year, possibly in order so DOE could claim an increase in the overall graduation rate.  Finally, we pointed out how some of the students categorized as discharges according to the DOE codes, such as students who left school to attend GED programs or because of pregnancy, should have been listed as drop-outs instead, according to state and federal standards. 

Our report led the DOE to change its coding for some of the categories and the City Council to pass a law called Local Law 42, to require detailed and disaggregated discharge reporting each year.  The results of that reporting are here.

A related law, Local Law 43, was also passed required the reporting of discharge rates at closing high schools, shown here; in these schools the discharge and drop out rates increase sharply.  Here is my testimony in support of both these bills.

Also as a direct result of our report, Betsy Gotbaum, then the Public Advocate, asked the NY State Comptroller to audit DOE’s discharge rates.  When the results of that audit were finally released in 2011, they revealed that 14.8% of students who were labelled as discharged should have been identified as dropouts instead, and fully 20% of the special education students. Moreover, the auditors found that DOE had no evidence to show that more than half of their sample of discharged students weren't actually dropouts; taking DOE months to come up with documentation. 

Just a few weeks ago, on Sept. 9, 2014, the NY State Comptroller’s office released a little noted, follow-up report showing almost no improvement in this area. According to a DOE internal audit from 2012, as many as 14% of reported discharges still should have been reported as dropouts.  A subsequent audit from December 2013 continued to find unspecified errors in DOE’s discharge classifications. 

I have now FOILed DOE and the State Comptroller for these two audits; we will see how long it takes them to respond.  Yet it is disappointing that there has been so little progress in the accurate reporting of this data, whether out of sloppiness or to inflate NYC's graduation rate, especially given DOE’s claims of being a “data-driven” agency.

Concerns with the MDRC study on small schools released today

MDRC released a  study today, which the NY Times writes “appeared to validate the Bloomberg administration’s decade-long push to create small schools to replace larger, failing high schools.” The report mentions the current controversy over the massive number of school closings, here in NYC and across the country, and thus there may be a political element in the timing of its release:

MDRC’s findings about SSCs are relevant to current federal policy on high school reform, particularly the U. S. Department of Education’s School Improvement Grants (SIGs) for failing schools. Reforms funded by SIGs include school transformation, school restart, school closing, and school turnaround. SSCs straddle several of these categories since they are typically replacements for schools that have closed and they operate as regular public schools.

This is the second MDRC study to conclude that students who attended the new small schools had significantly improved outcomes.  The first MDRC study, released in 2010, looked at students who entered these schools in 2005; this one adds students who entered in 2006 to that group.

I have a lot of reservations about using this study or the previous study to justify the small schools initiative and especially to justify the current massive round of school closings.  I am no expert in statistics, but my concerns revolve around these issues: 

1-      Though the study points out that the small schools were supposedly “unscreened” and evaluates their results by comparing the outcomes of students who applied to the school through lotteries,  compared to those who lost the lottery, it  ignores that the students who attended these small schools were far less high-needs on average, as evidence by their lower rates of English language learners and special needs, as shown by this Annenberg study by Pallas and Jennings.  In fact, they were allowed to openly exclude special needs students during the first two years. Thus even with a “lottery” for admissions, there are substantial peer effects for students who are grouped with higher-achieving students which this study does not mention.  (This is also a problem with many of the charter school studies, like this one, which tend to ignore peer effects.)

2-      The results in terms of higher graduation rates and college readiness (based on Regents scores and credit accumulation) ignore how in NYC, teachers and principals are able manipulate these in ways that do not reflect real learning (especially as teachers grade the Regents of students in their own schools, and in these schools, their own students!)  It has also been alleged that the small schools pioneered the now widespread and largely discredited practice of “credit recovery.” With newer teachers at many of the small schools, who did not have a memory or tradition as to earlier practices, it may have been easier to pressure these teachers into employing such methods.

3-      The study ignores that the small schools on average were allowed to have smaller classes and were far less overcrowded than the large high schools, which legitimately could have led to better results.  The class size at the small schools during these years were from 13 to 20 students per class, according to the PSA first year report, compared to 30 or more at the larger schools.  If the higher needs students in the larger schools had been provided with smaller classes, very likely their chance of success would have been improved substantially as well.

4-      Yet the study doesn’t examine how the opening of the small schools had a negative impact on the system as a whole, by flooding nearly large high schools with the most disadvantaged and academically challenged students, leading to even more overcrowding, larger class sizes, and damaging their opportunity to learn, as reported by many observers and confirmed by the New School’s report, The New Marketplace.

5-      The MDRC study deals with only a subsection of the small schools that were oversubscribed and required a lottery for admissions, so like the charter school studies which use a similar methodology, their success rate may not be representative of small schools overall.

6-      It study ignores the reality that these so-called random “lotteries” may be far from actually random. The MDRC study compares the baseline characteristics of students who “won” the lotteries to those who lost, in a  Supplemental Table 1, which purports to show that both groups were “virtually identical”; but the comparison does not include students who required collaborative team-teaching or self-contained classes; does not include  prior attendance rates, a key factor that principals often examing when selecting students; and does not differentiate between free and reduced price lunch students.  The table also does not include data on previous rates of suspension.  According to an earlier evaluation done by Policy Studies Associates. the ninth graders who entered the small schools had far better attendance records (91% compared to 81%), and were less likely to have been suspended as 8^th graders compared to students at the schools they replaced.

Even in the categories the MRDC study does compare, there are greater numbers of higher achieving students, though the differences are not statistically significant, according to the model used.   Most strangely, the table compares the baseline data for four cohorts – entering ninth-graders from the 2004-2005 to 2007-2008 years; while the report compares outcomes for only the first two of these cohorts, students who entered these schools from 2005-6 and 2006-7. This is despite the fact that that the need level of students increased significantly after that point – and likely, the challenges faced by these schools as well.  See the above chart, for example, from the Annenberg study by Jennings and Pallas.

I have no idea why the MRDC study lumped together all these cohorts to examine their baseline characteristics, even as they only compared the outcomes for the first two, but it may considerably bias their conclusions.  Indeed, more than half of the middle and high schools being closed by the DOE this year for poor results are small schools that were founded after 2003.

7- There are several ways in which, especially in the early years of the small schools initiative, principals were able to manipulate the admissions process to get the students who were more likely to succeed, even though by definition these schools were supposedly “unscreened” and used “random” lotteries for admissions.  See this excerpt from a published study by Jennifer Jennings, who embedded herself with three small schools between March 2004 and September 2005: 

My observations revealed that many schools used applications, mandatory information sessions, and much stronger language to deter unwanted applicants. For example, 12 unscreened schools shared a similar application requiring that students provide the most recent report card and two letters of recommendation, one from an eighth-grade teacher and one from a guidance counselor, assistant principal, or principal. The application also asked for the student’s test scores, retention history, and involvement in advanced courses during the eighth grade. Finally, the application included additional questions requiring a narrative response….

The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. ….

The district’s application system provided opportunities for unscreened schools to choose higher achieving students. Through this computer system, each school received a list of students applying to the school, although the school did not know whether the student ranked it, for example, 1st or 12th. This data file included each student’s English-language-learner and special education classification, reading and math test scores, absences, grades, address, and junior high school. Schools were told to identify students who made an ‘‘informed choice’’ by assigning them a 1, while students who did not make an informed choice but the school was willing to accept were assigned a 2. If the school did not fill all of its seats with students making an informed choice, additional seats would be filled by students in the second category.  The Department of Education prohibits unscreened schools from using student performance data to select students. Nonetheless, both Marlena and Anna [pseudonyms for two principals of small schools] learned through their relationships with other principals that such regulations were loosely enforced….

 In addition to the English language learners and full-time special education students whom new schools had a waiver to eliminate, Renaissance [pseudonym for one of these small schools] eliminated part-time special education students and chose only those with 90 percent or higher attendance. Excel eliminated full- and part-time special education students and chose students with attendance rates of 93 percent or higher. 

There are many more revealing insights in the Jennings study about how the small schools were able to deflect over-the-counter students and counsel out low-performers to achieve better results, despite the fact that they claimed to be non-selective.  (Update: I have removed the link to the paper at the author's request; the abstract is here.)

All of these concerns  should provide one with reservations about the validity of the MDRC study and especially its apparent endorsement of the mayor's policies.

Two important education laws requiring more transparency from DOE passed today!

Today, along with the budget, the City Council approved two new important bills requiring more transparent reporting from the DOE.

According to the first bill, 354-A, the DOE will now be required to report each year by June 30 on the number of students discharged by individual schools and systemwide, as well as by discharge code, so we will know better what is happening to the thousands of students that continue to leave NYC high schools each year before graduation but are not counted as dropouts. 

Some students move out of state or transfer to parochial and private schools, but many transfer to GED programs or alternative schools where they do not have a chance to graduate with a real diploma, and currently we have no idea how many are in each category.  Moreover, the percent of 9^th graders who are discharged from high school has doubled under this administration, without any explanation. 

I am particularly proud of this bill because the work of Class Size Matters helped bring more prominence to this issue.  See the discharge rate report  that we released in 2009 by lead author Jennifer Jennings, this story in the NY Times which covered our findings, and our testimony in support of this bill in January.  An audit we asked for from the State Comptroller’s office, along with then-Public Advocate Betsy Gotbaum released in March found that 15-20% of the students reported as discharges by DOE should really have been reported as dropouts.

The other bill, 364-A,  just passed by the Council, will report on the fate of students at the closing schools; whether they are discharged, drop out, or are given credit recovery to graduate before the school closes its doors.  For more on this issue, see our blog posting about the hearings.  If students are behind credits at the closing schools – which many are – they have not been allowed by DOE to transfer to other degree-bearing regular high schools, and there have been sky-high discharge and dropout rates at many of these schools. Hopefully this bill will provide DOE more incentive to ensure that students are not left behind when their school phases out.

At the hearings in January, DOE officials made the absurd claim that these two bills violated FERPA, or federal privacy protections, and  threatened that they would not comply with the laws as written  even though they contain language that specifically gives them a pass if the number of students in any category is so small that individuals might be able to be identified. I hope they have changed their mind and will report this critical information responsibly and accurately.

Good news; the DOE says they will comply with these two laws, according to this report in GothamSchools:  Bills will hold DOE’s feet to fire on discharge, graduation rates

April 30 press conference on rising discharge rates

Here is video from our press conference at the Public Advocate's office, including an introduction by Betsy Gotbaum and a lucid power point given by Jennifer Jennings, the co-author of our report on the discharge rate.

Also check out Part 2, with the rest of Jennifer's presentation and some relevant comments from Kim Sweet, head of Advocates for Children, about how worrisome these figures are, particularly the extremely high discharge figures for full-time special education students at 24 percent, nowhere reported in the city's official graduation rate. Part 3 features Dianne Morales, Executive Director of the Door, who has seen increased numbers of students coming to her program after having been encouraged to leave their high schools over the past five years, and some recommendations that I offer about what should be done about this troubling phenomenon. Part 4 and Part 5 has us answering questions from reporters.

Here are some news stories about our findings: Number of Students Leaving School Early Continues to Increase (NY Times); Study looks at city discharge rates (Channel 7 news); Saying discharges are up, report demands grad rate audit (Gotham Schools.)

(Video thanks to David Bellel)

Discharge rates still rising; especially for students in their first year of high school!

There are literally thousands of NYC students who disappear off the rolls every year -- leaving high school without a diploma, but who never counted as dropouts in the DOE's official graduation rate or anywhere in its porous accountability system. There were over 20,000 students who entered our high schools in 2003, and should have graduated in 2007, but never did. These are the "desparacidos" -- the ones who disappear.

Check out the report released on Thursday -- written primarily by the brilliant Jennifer Jennings (a/k/a Eduwonkette) with minor contributions from myself -- showing that in NYC, the discharge rate has significantly increased between 2000-2007.

While the city reported an "official" graduation rate of 62% for the class of 2007, if all special education students were included this would drop to 58 percent. If GEDs were excluded, this would fall to 55 percent, and if discharged students were counted as dropouts, this would further fall even further -- to 44 percent. (click on the graph to the left to enlarge it.)

Most shocking is the fact that the rate and numbers of students discharged in their first year of high school literally doubled. This may be because these students moved out of the city or to parochial or private schools in larger numbers than ever before, yet analysis of census and enrollment data provide no evidence for a rising rate of migration or transfer to parochial schools.

More likely, these students are entering HS even more overage than before (due to multiple grade retentions), since no student can be legally discharged before the age of 17. Or perhaps many of these discharges are illegal.

What is especially tragic is little or nothing has been done to address the problem of discharged students since the problem of "pushouts" was first exposed by Advocates for Children in 2002. Perhaps this is because the higher the discharge rate, the higher the graduation rate by definition, since all these students are excluded from the denominator for the purposes of calculating the graduation rate.

In fact, in the report, we point out several features of the DOE's high-stakes accountability system which encourage schools to rid themselves of low-performing students as fast as possible through discharging them, instead of giving them the support, resources, and smaller classes they need to graduate.

This is a "black hole" of accounting which must be addressed. As a result of our report, and at the request of Public Advocate Betsy Gotbaum, the State Comptroller has agreed to audit NYC's graduation and discharge data. Hurray!

There's loads more interesting information in our report, including tantalizing evidence about possible data manipulation for the Class of 2005. This class had originally caused headlines when its graduation rate was first released in the Mayor's management report in February 2006, revealing a drop from the year before -- 53.4 percent compared to 54.3 percent for the previous class. According to the NY Times at the time:

The political touchiness over yesterday's numbers was evident in how the mayor's office chose to report them this year. Administration officials created a new category in the preliminary management report that had the effect of masking the decline in the four-year graduation rate. Although that rate continued to be reported separately, the new category factored in students who stayed on for an extra year of school, allowing the mayor's office to state, "More students graduated from high school in four years or are still enrolled in school for a fifth year."

Speaking to reporters yesterday, Mr. Bloomberg said it would take years before many of his changes, like grade retention policies that hold back elementary and middle school children largely on the basis of test scores, were reflected in improved graduation rates.

Then, in June, the DOE announced that the figure of 53.4 percent had been a mistake and that this class had really graduated at 58.2 percent. As the NY Times uncritically reported at the time, "because of a computer glitch, last year's citywide graduation rate was five points higher than previously reported — the highest on-time graduation rate in more than two decades.....Officials said the mayor was angry after learning of the mistake and intent on getting an accurate tally."

How did this more "accurate tally" happen? By looking at the data, and comparing DOE's original and "corrected" graduation reports , it appears that over 1000 students may have been recategorized as full-time special education and then discharged at the astonishing rate of 39%.

At the same time, the number of general education students who had entered four years before fell by over 2,000. All of these changes, unremarked at the time by any reporter, had the convenient effect of allowing the Chancellor to claim a sharp rise rather than a fall in the city's "official" graduation rate, (which includes only general education and part-time special education students), as well as proof of the efficacy of his reforms.
We also found that several of the categories that NYC calls "discharges" are more properly considered dropouts by the federal government -- including students who leave because they've turned 21, withdrawn because of pregnancy, were expelled, or had transferred to GED programs outside the DOE system. Check out our press release here.

Join us for a very special celebration!

Announcing the first annual

Skinny Awards

When: Thursday May 7, at 6 PM

Where: Jerry's Café, 90 Chambers St (between Church and Broadway)

Please join us for a very special evening

Presenting awards to the three best education bloggers, who provide us with the real "skinny" on NYC schools:

Diane Ravitch, Lifetime Achievement Award

Jennifer Jennings (AKA Eduwonkette), the Shooting Star Award

Gary Babad, Humorist Supreme

A rare opportunity to meet these three celebrated bloggers

and enjoy a three course dinner with wine.

A fundraiser sponsored by the NYC Public School Parent Blog and Class Size Matters.

Tickets: $100 --Patron, $75 -- Supporter

To attend, please send a tax-deductible contribution to Class Size Matters, 124 Waverly Pl., New York, NY 10011

Or click on this link: In the section at the bottom entitled "Designate your donation to a specific program or fund," please write May 7 dinner, along with the number of tickets you are purchasing.

Be there or be square!

Eduwonkette revealed!

Eduwonkette revealed herself on her blog last night as Jennifer Jennings, a grad student in Sociology at Columbia University. See also this article in NY Magazine.

Jennifer is beautiful and brilliant and an expert in deconstructing the fraudulent statistics of the NYC Department of Education.

As one of the few individuals who has known her identity for many months, I must say it’s a relief not to have to keep it secret any more.

Jennifer also did the seminal study of the “bubble kids” in Texas, revealing the "educational triage" that high-stakes testing had given rise to; see her study here and her Washington Post oped summarizing the results.

There will undoubtedly be many more path-breaking studies to come – that is, if Bloomberg and Klein do not put out a hit against her.

Here’s hoping that this emboldens some of the other academics who in private, are extremely critical of this administration’s policies, to be courageous enough to speak out publicly themselves.