A mother's lament after learning the grades of her sons' schools

 Lots more commentary and criticism of the latest school grades.  Gary Rubinstein explained how the DOE's claims of stability are an example of "how to lie with statistics" here. The NY Post pointed out that the Ella Baker school  which got a "C" had just been singled out as one of the best schools in the city by the State Education Department. An editorial in the same paper criticized the grades as being entirely untrustworthy.  Eagle Academy, which last spring was praised by Bloomberg at a ceremony at Yankee Stadium as "a model for improving student achievement." fell from the 81% percentile to to 14%,   receiving a "C"  overall and a "D" for progress.   The Daily News reported that of the seven elementary and middle schools that DOE tried to close last year -- blocked by a lawsuit -- two got "B"s and the others "C"s.  Not a single one received a failing grade.  More about why these grades are so unreliable in our previous post, Why no one in his right mind should believe the school grades. 

Here is what one mom wrote, Tory Frye, after learning the grades of her sons' schools:



My sons' schools have a C and a D!  Last year was great for my older son at the "C" school and my little guy come homes from Kindergarten at the "D" school and describes the most loving, stimulating and creative environment.

My older son wrote a letter to  the [US Attorney General] Eric Holder last year about the effectiveness of incarceration of non-violent offenders as a crime prevention strategy (and got a response) and wrote a 9-page paper on Thomas Jefferson.  My 5-year old son came home the other day and said that a month is a long time because it is 4 weeks long and a week is 7 days long so that is 28 days altogether.  Oh, and he has art, music, movement and drama all year long!

Yeah, these two schools suck.  -Tory Frye

Why no one in his right mind should believe the school grades OR the teacher growth scores

UPDATE: Here is Gary Rubinstein's new graph of school ranks. this year compared to last, according to the Progress reports.  The "x" axis is 2011; the "y" axis is 2012. 
The new DOE school grades for elementary and middle schools, euphemistically called “Progress reports,” came out with much fanfare, with 217 schools potentially put on the closing list because they either received failing grades or three “Cs” in a row, more than ever before.  As parent leader Shino Tanikawa pointed out in the New York Post, DOE is using unreliable grades to implement terribly misguided policies. (If you'd like to see your school's grade anyway, you can find it here.)

Last year’s headlines ran like this: School report cards stabilize after years of unpredictability. This year again, reporters cited claims of DOE officials, who “highlighted the stability of this year’s reports.”

It wasn’t true last year and it isn’t this year either.  Here is a figure from Gary Rubinstein’s blog from last year, showing no correlation between the rank order of schools in 2010 and 2011. I suspect the Gary’s illustration will be similar for this year, once he gets around to making it.

As InsideSchools reported, 24 out of the 102 schools that received “D”s or “F”s this year had received top grades of “A” or “B” the year before. Other high-performing schools, such as PS 234 in Tribeca that received an “A” last year, fell precipitously to a “C” with the same principal, same staff and most of the same students.  The school plunged from the 81^st to the 4^th percentile, which would have meant a “D,” if not for the DOE rule that no school that performs in the top third citywide can receive a grade lower than C, as Michael Markowitz pointed out in a comment on GothamSchools.

According to the DOE formula, 80-85% of school’s grade depends on last year’s test scores on the state exams.  Most of that figure is based on supposed “progress,” i.e. the change in test scores from the year before.  As I have pointed out many times, including in this 2007 Daily News oped, “Why parents and teachers should reject the new grades”, experts in statistics have found that the change in test scores at the school level from one year to the next is highly erratic and up to 32-80% random. 

In recognition of this fact, Jim Liebman, who developed the school grading system, originally told skeptical parents that the system would eventually incorporate three years of test scores, which would lessen the huge amount of volatility, a promise that DOE has failed to abide by.  For proof of this promise, see Beth Fertig’s book, Why Can’t U teach me 2 read?:

“…[Liebman] then proceeded to explain how the system would eventually include three years’ worth of data on every school, so the risk of big fluctuations from one year to the next wouldn’t be such a problem. (p.121)”

To make things worse, the state exams last year and the scoring guides were riddled with errors, as many parents, teachers and students noted . Finally, test scores are not a good way to assess school quality, for myriad reasons, even if the tests were perfect and the formula based on multiple years worth of data, as I pointed out in this NYT Room for Debate column. 

LESSON: Anyone who believes in the accuracy of these school grades is sadly misinformed, and DOE’s attempts to dissuade parents from sending their kids to schools with low grades or to close schools based upon such unreliable system is intellectually and morally bankrupt.

In another untenable move, the teacher “growth scores,” also based on the one year’s change in test scores, but this time at the classroom level, have been released to principals outside NYC.  These growth scores will be incorporated in the new teacher evaluation system to be imposed statewide.  See this excellent column by Carol Burris about how the heedless use of growth scores is likely to damage the education of our kids. 

Here are the detailed results of the statewide survey by principals, including their comments, showing that the vast majority believe that these scores are NOT an accurate reflection of the effectiveness of individual teachers.  More than 70% of principals also said they were “doubtful” or strongly opposed to any use of growth scores in teacher evaluations.  Aside from the annual volatility in these scores, there are many other problems with relying upon such flawed measures of teacher quality:

The growth scores, developed for the state by the consulting company AIR, attempted to adjust only for the following demographic factors: 

• Economic disadvantage (ED), but without differentiating free lunch or reduced lunch students – very different, with very different expected outcomes;
• Students with disabilities (SWDs), but not types or severity of disability;
• English language learners (ELLs).
  

And even though AIR did attempt to control for the above factors, they still admitted that teachers who work at schools with large numbers of students who were poor or had disabilities tended to have lower growth scores. 

AIR made NO attempt to control for classroom characteristics such as class size, or the racial/ethnic background of students, or any other variable that is known to affect achievement. This is different from the NYC value-added teacher data reports, released last year by DOE, widely derided as unfair and unreliable, which at least attempted to control for many of these factors.

Even so, Bruce Baker, professor at Rutgers, found that while the teacher data reports claimed to control for class size, teachers at schools with larger class sizes were significantly more likely to be found ineffective than those who taught at schools with small classes, and as “class size increases by one student, the likelihood that a teacher in that school gets back to back bad ratings goes up by nearly 8%.”  The situation with these growth scores is yet worse, with no attempt made to control for class size at all.  Is it fair to deny NYC teachers tenure and/or risk losing their jobs because they are saddled with larger classes than teachers in the rest of the state?

AIR also found that teacher growth scores tended to be higher in “schools with students with higher mean levels of ability in both ELA and in Mathematics.” This shows that teachers who work in schools with large numbers of low-achieving students are more likely to be found ineffective. 

Apparently NYC principals have not yet been offered the opportunity to examine the growth scores of their teachers, unlike principals in the rest of the state. As far as I know, DOE has not explained why.  With far larger numbers of struggling students who are economically disadvantaged and crammed into larger classes, it is quite likely that a higher proportion of NYC teachers will be found ineffective than elsewhere in the state– and thus unfairly penalized for teaching disadvantaged students in worse conditions.

Here is the link where you can find the AIR technical manuals on growth scores.  

The worst schools in NYC?

The only information out of the school progress reports that I would trust is the list of schools that got an “F” for environment, based predominantly on extremely low ratings from parents and teachers on the learning environment surveys, and in the case of middle and high schools, from students as well.

Here is a chart of the schools that received the lowest scores and an “F” for environment.

All of these schools should get intensive intervention from DOE, and probably a new principal, though it’s likely they won’t, because all DOE really cares about is test scores.

And since only 15% of the overall school grade comes from the environment score (10% from survey results, and 5% from attendance) most of these schools got progress report grades of B or C. (Here is more info on these surveys, including citywide results.)

There are schools on the list in Districts 3, 6-13, 16-19, 23-24, 28-29, 31, 75 and 85 (charters) The districts that had the most of these schools were D7, D29 and D31 (three in each), D12 (four), and D75 (five) .

Though I am aware of how survey results can be skewed because some principals put pressure on parents and teachers to give positive marks to the school or otherwise DOE will close the school down, these are schools in which these pressures didn’t work, and parents and teachers are extremely unhappy.

Some of the schools on this list are I know from other sources have had extremely problematic principals. For example, Muscota in Washington Heights, where parents and teachers were very active protesting the actions of their principal last year. (See here and here.)

And Ross Global Charter had the lowest environment score of all; see this account from Mariama Sanoh, a parent at the school.

The Muscota parents were very media savvy and managed to get rid of their principal, Tomasz Grabski, after lots of protests, but then DOE then put him in charge of M.S. 45/S.T.A.R.S. Prep Academy in D4 this year.

If anyone has anything good or bad to say about these schools or their principals, please leave a comment!

Why the school grading system, and Joel Klein, still deserve a big "F"

Amidst all the hype and furor of the release of today’s NYC school "progress reports", everyone should remember how the grades are not to be trusted. By their inherent design, the grades are statistically invalid, and the DOE must be fully aware of this fact. Why?

See this Daily News oped I wrote in 2007, in which all the criticisms still hold true, “Why parents and teachers should reject the new grades”.

In part, this is because 85% of each school’s grade depends on one year’s test scores alone – which according to experts, is highly unreliable. Researchers have found that 32 to 80% of the annual fluctuations in a typical school’s scores are random or due to one time factors alone, unrelated to the amount of learning taking place. Thus, given the formula used by the Department of Education, a school’s grade may be based more on chance than anything else.

(source: Thomas Kane, Douglas O. Staiger, “The Promise and Pitfalls of Using Imprecise School Accountability Measures, The Journal of Economic Perspectives, Autumn, 2002.)

Now Jim Liebman admitted this fact, that one year’s test score data was inherently unreliable, in testimony to the City Council, and to numerous parent groups, including to CEC D2, as recounted on p. 121 of Beth Fertig’s book, Why can’t U teach me 2 read.” In responding to Michael Markowitz’s observations that the grading system was designed to provide essentially random results, he admitted:

“There’s a lot I actually agree with, he said in a concession to his opponent…He then proceeded to explain how the system would eventually include three years’ worth of data on every school so the risk of big fluctuations from one year to the next wouldn’t be such a problem.”

Nevertheless, the DOE and Liebman have refused to comply with this promise, which reveals a basic intellectual dishonesty. This is what Suransky emailed me about the issue, a couple of weeks ago, when I asked him about it before our NY Law school “debate.”

“We use one year of data because it is critical to focus schools’ attention on making progress with their students every year. While we have made gains as a system over the last 9 years, we still have a long way to reach our goal of ensuring that all students who come out of a New York City school are prepared for post-secondary opportunities. Measuring multiple years’ results on the Progress Report could allow some schools to “ride the coattails” of prior years’ success or unduly punish schools that rebound quickly from a difficult year.”

Of course, this is nonsense. No educators would “coast” on a prior year’s “success”, but they would be far more confident in a system that didn’t give them an inherently inaccurate rating.

Given the fact that that school grades bounce up and down each year, most teachers, administrators and even parents have long figured out how they should be discounted, and justifiably believe that any administration that would punish or reward a school based on such invalid measures is not to be trusted.

That DOE has changed the school grading formula in other ways every year for the last three years also doesn’t give one any confidence….though they refuse to change the most fundamental flaw. Yet another major problem is while the teacher data reports take class size into account as a significant limiting factor in how much schools can get student test scores to improve, the progress reports do not.

There are lots more problems with the school grading system, including the fact that they are primarily based upon state exams that we know are themselves completely unreliable. As MIT professor Doug Ariely recently wrote about the damaging nature of value-added teacher pay, because of the way they are based on highly unreliable measurements:

…What if, after you finished kicking [a ball] somebody comes and moves the ball either 20 feet right or 20 feet left? How good would you be under those conditions? It turns out you would be terrible. Because human beings can learn very well in deterministic systems, but in a probabilistic system—what we call a stochastic system, with some random error—people very quickly become very bad at it.

So now imagine a schoolteacher. A schoolteacher is doing what [he or she] thinks is best for the class, who then gets feedback. Feedback, for example, from a standardized test. How much random error is in the feedback of the teacher? How much is somebody moving the ball right and left? A ton. Teachers actually control a very small part of the variance. Parents control some of it. Neighborhoods control some of it. What people decide to put on the test controls some of it. And the weather, and whether a kid is sick, and lots of other things determine the final score.

So when we create these score-based systems, we not only tend to focus teachers on a very small subset of [what we want schools to accomplish], but we also reward them largely on things that are outside of their control. And that's a very, very bad system.”

Indeed. The invalid nature of the school grades are just one more indication of the fundamentally dishonest nature of the Bloomberg/Klein administration, and yet another reason for the cynicism, frustration and justifiable anger of teachers and parents.

Also be sure to check out this Aaron Pallas classic: Could a Monkey Do a Better Job of Predicting Which Schools Show Student Progress in English Skills than the New York City Department of Education?