FOILed documents show how the DOE dissembled regarding their Teacher Data reports

Several years ago, Chancellor Klein determined to use value-added methods to measure the achievement gains of teachers of English and math teachers in grades 4-8, by comparing the standardized test scores of their students to these students' test scores previous year.

Since then, numerous studies have shown how inherently unreliable this value-added approach is in estimating teacher effectiveness, including an analysis by Mathematica for the US Department of Education, showing that there is a 25-35% chance of misidentifying the worst teachers as the best; as well as a recent study by Sean Corcoran of NYU demonstrating that the NYC teacher data reports have an average margin of error of 34-61 percentage points out of 100.

Critiques from the National Academy of Sciences in their comments on "Race to the Top" program, and noted academics assembled by the Economic Policy Institute have also warned of the potentially damaging consequences of implementing these unfair and inherently unreliable evaluation systems.

Yet in 2007, Klein hired a consultant from Battelle to develop a mathematical model that took a few school and classroom factors into account, including aspects of student background, as well as the class size and the experience level of the teacher. (Smaller classes and greater teaching experience are the only two observable factors that consistently lead to more learning, and yet DOE officials consistently devalues both of them. They are included in the model nevertheless, apparently because the research is so clear on this. )

Battelle then devised a formula that the DOE then used to produce "teacher data reports" which would ostensibly measure the effectiveness of these teachers (See here, for a sample version of these reports.)

In October 2008, Chancellor Klein made an agreement with the UFT that the teacher data reports would not be used to evaluate teachers, but only to help them improve their instruction:

“…as a tool for schools and teachers to use for instructional improvement. They are not be used to evaluate teachers…. Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers…”

Klein later went back on this promise, and in February 2010, he instructed principals to consider these reports when deciding whether to give teachers tenure :

“Principals and superintendents will consider the performance of each teacher who is up for tenure more carefully than ever, weighing multiple factors including Teacher Data Reports, where available and appropriate.”

As Gotham Schools reported at the time: “Those teachers who fall into the bottom or top 25 percent of the rankings will be red-flagged, alerting principals that the DOE recommends giving them tenure or cutting them lose [sic] . In total, about 160 teachers will fall into that bottom percentile. “

Though Klein also originally agreed with the UFT to keep the individual reports confidential, as are most performance ratings , and to resist releasing them to the public even if FOILed, he has gone back on this promise as well.

Yet even back in the fall of 2008, when the reports were first provided to principals, it was clear to me and many others that DOE would eventually use them to evaluate teachers, as by nearly all accounts, they have little or no value to helping teachers improve.

I also thought (and believe to this day) it is critical that any model used to determine a teacher’s effectiveness and professional future should be made publicly available, and independently vetted by experts in statistics and testing.

After spending months of unproductive requests to former chief press officer David Cantor and Amy McIntosh, the head of the DOE “talent office”, asking for more information about the model used and evidence of its reliability, I decided to FOIL this information.

Here is an excerpt from my original FOIL request, dated Feb. 23, 2009, along with the partial DOE response that finally came in May 24, 2010, more than fifteen months later:

Dear FOIL Officer:

This is a request for records pursuant to the Freedom of Information Law ("FOIL"), Article 6 of the Public Officers Law. We hereby request disclosure of the following information concerning the Teacher Performance Data Reports:

1) The model specification used in the report to produce estimates of teacher effectiveness;

2) The sources of the data for class size at the classroom and school level over the last ten years;

After many months of delay, here is an excerpt from the DOE response, dated May 24, 2010:

"With respect to items one and two of your request, while certain records being released today may be responsive to these items…The model was not designed to ascertain the impact of class size or other classroom or school level variables…"

Nevertheless, the “draft” technical report from the Battelle consultant indicated a very substantial impact of class size on achievement in math:

“In math...the characteristics that had a negative impact were percent free or reduced price lunch and class size.” (see also Table 5.2 in the analysis.)

Teacher experience level also had a significant effect, in both math and ELA.

In December 2008, more than a year before I filed my FOIL, in a document supposed to allay teachers’ fears, entitled: “Teacher Data Initiative: Support for Schools; Frequently Asked Questions, DOE had claimed that an independent panel of experts had attested to the model's validity and reliability, writing:

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

.So in my FOIL I asked for more information about this panel, including:

3) The identity of the members of the "panel of technical experts" who approved the model and/or methodology of these Reports, as well as the times and locations in which these experts met to discuss these issues with DOE staff;

Yet in 2010, in response to my FOIL, the DOE contradicted their earlier claim:

" With respect to item three of your request, I have been informed that approval of the model and methodology rested with the DOE, and not with any “panel of technical experts.” As a consequence, I have been informed that there are no responsive records that will answer this aspect of item three."

Instead, DOE sent a list of names on a document entitled “Technical Expert Panel: Value-added Data for Teachers Initiative”, dated Sept. 25, 2007 with representatives from the various groups, including the UFT, academia etc.

They also sent a report from a subset of these individuals, entitled “Statement on the New York City Teacher Value-Added Model” dated August 29, 2008, written many months after the teacher data reports were first released, and months after the DOE had claimed that an independent panel had validated their reports.

This statement was written by Tom Kane, then at Harvard and now at the Gates Foundation; Jon Fullerton of Harvard; Jonah Rockoff of the Columbia Business School, and Douglas Staiger of Dartmouth College. Far from validating the reliability of DOE’s methodology, these men expressed numerous reservations and caveats about value-added approaches in general, and made the following points, among others:

"1) Test scores capture only one dimension of teacher effectiveness, and they are not intended to serve as a summary measure of teacher performance…

2) If high stakes are attached, there will be potential to game these measures by teaching to the test, selecting students, altering difficult-to-audit student characteristics, or outright cheating. …

3) To calculate expected test scores…there are likely to be additional factors not yet considered that influence student achievement. …etc. "

In the FOIL, I also asked:"Whether the members of this panel were paid for their services and if so, the source of these funds..."

DOE responded: "With respect to item four of your request, the DOE did not pay any members of the panel for their services. It is my understanding that some members of the panel were paid by the Fund for Public Schools (the Fund) for related research work. Consequently there are no responsive record to provide."

Even though Klein runs the Fund for Public Schools out of Tweed, the DOE claims that it is not a public agency and does not have to release its financial records to the public.

In its 2007 FAQ, DOE had also claimed that there was another document that potentially confirmed the accuracy of their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

So I also asked for a copy of this "research study", as well as a few other items, none of which have been provided to this day.

I last heard from the DOE on August 12 and again on September 10, 2010, saying that the above "research study" is still not complete, nearly three years after the DOE had claimed it existed.

In any event, the DOE has now commissioned researchers at the University of Wisconsin to "update" their teacher data reports, apparently not satisfied with the earlier versions produced by Battelle.

For copies of all these FOILed documents, see the Class Size Matters website ; for more on the problems with the teacher data reports, see today's column by Juan Gonzalez, as well as articles in the Daily News, the NY Times, GothamSchools and the Christian Science Monitor.