Who to believe among "experts" on teacher evaluation at Albany Summit on teacher evaluation?

Today, from Albany NYSED is livestreaming what they call a "Learning Summit on Annual Professional Performance Review (APPR), the teacher and principal evaluation system.   The ostensible purpose of the meeting is to get input from experts, educators and parents about how to go about crafting their new teacher evaluation system that they are supposed to come up with by June 30, but that is severely restricted by the damaging rubric imposed by the Governor.

On the agenda from 1-2 PM, is a panel of  "National experts in the field on education, economics and psychometrics."  The invitees include:

• Thomas Kane, an economist from Harvard University who strongly supports test-based teacher evaluation and led the Gates Foundation’s Measures of Effective Teaching study;
• Catherine Brown, VP of the Center for American Progress, which has published papers endorsing the use of value-added and has received more than $5 million from the Gates Foundation for its education work.  Brown is also married to Robert Gordon, formerly of NYC DOE, OMB and the US Dept of Education, who pushed test-based teacher evaluation in NYC and throughout the country.
• Sandi Jacobs, a vice president at the National Council on Teacher Quality which also strongly supports test-based teacher evaluation and has gotten more than $12 million from the Gates Foundation;
• Leslie Guggenheim of TNTP, an advocacy organization whose 2009 paper “The Widget Effect” promoted test-based teacher evaluation and has gotten more than $33 million from the Gates Foundation.

On the other side with a more skeptical view include academics who are not on the Gates payroll: Aaron Pallas of Teachers College, Jesse Rothstein of UC Berkeley, and  Stephen Caldas of Manhattanville College.

So here we have three representatives from inside-the-Beltway advocacy groups that collectively received more than $50 million to make the case for test-based teacher evaluation and one professor who led the $45 million MET project for Gates, vs three independent academic scholars.

Also  speaking at 4 PM is a parent panel selected by the NYS PTA, including a representative from NY State Allies for Public Education, a coalition of more than 50 parent and advocacy groups statewide (full disclosure: including Class Size Matters.)  NYSAPE has helped lead the anti-testing movement that garnered at least 200,000 students opting out this spring.  Also on that panel, strangely enough, is Matt Barnum, the policy director of Educators for Excellence, which has received  $4 million from the Gates Foundation.

Who to believe?  You be the judge.

Teaching experience matters!

Even as Mayor Bloomberg, Chancellor Cathie Black and Michelle Rhee are arguing for experienced teachers to be laid off regardless of seniority, every single study shows teaching experience matters.

In fact, the only two observable factors that have been found consistently to lead to higher student achievement are class size and teacher experience, so that it's ironic that these same individuals are trying to undermine both.

Generally speaking the corporate reformers argue that only the first few years of experience matter,these charts from a study by Thomas J. Kane , Jonah E. Rockoff , and Douglas O. Staiger reveal that at year five, effectiveness is still increasing  for all categories of teachers --other than uncertified teachers. (see charts to the right.)

Of course, most Teach for America recruits are gone by then, which is why their dotted line vanishes at year three.

Actually, there are many studies that show that teaching experience matters, for 15-20 years – with each year in the profession leading to more student gains, especially in reading.








The importance of experience may be clearer in the teaching profession than any other, as shown in a recent re-analysis of the STAR Tennessee experiment, showing that Kindergarten students had higher achievement and earnings as adults, depending on how long their teachers had been in the profession, with gains for every year up to twenty. 

Another study of low-performing, high poverty schools, shows that years of teaching experience at the 2^nd grade level was associated with higher reading achievement, for up to 21 years.

“A teacher who had been teaching at a particular grade level for more than 5 years was positively and significantly associated with increased student achievement (effect size=.27)…grade level experience was sizable compared to race (minority status effect size = -.33) and SES (economically disadvantaged effect size= -.08)….Teachers constantly improved teaching effectiveness until the 21^st year and declined beyond that.”

See also this study , showing that for 4^th and 5^th grade teachers in Florida, "in elementary school reading there may be student achievement returns to as many as 15 years of additional teacher experience."

Here's another study that shows student gains for teacher experience-- again, for up to 20 years. (See charts to the right).

So when people who claim to care about kids advocate for laying off experienced teachers, don't believe them.

Private schools and well-funded public schools don't have to choose between having experienced teachers and smaller classes, and neither should our public schools.

FOILed documents show how the DOE dissembled regarding their Teacher Data reports

Several years ago, Chancellor Klein determined to use value-added methods to measure the achievement gains of teachers of English and math teachers in grades 4-8, by comparing the standardized test scores of their students to these students' test scores previous year.

Since then, numerous studies have shown how inherently unreliable this value-added approach is in estimating teacher effectiveness, including an analysis by Mathematica for the US Department of Education, showing that there is a 25-35% chance of misidentifying the worst teachers as the best; as well as a recent study by Sean Corcoran of NYU demonstrating that the NYC teacher data reports have an average margin of error of 34-61 percentage points out of 100.

Critiques from the National Academy of Sciences in their comments on "Race to the Top" program, and noted academics assembled by the Economic Policy Institute have also warned of the potentially damaging consequences of implementing these unfair and inherently unreliable evaluation systems.

Yet in 2007, Klein hired a consultant from Battelle to develop a mathematical model that took a few school and classroom factors into account, including aspects of student background, as well as the class size and the experience level of the teacher. (Smaller classes and greater teaching experience are the only two observable factors that consistently lead to more learning, and yet DOE officials consistently devalues both of them. They are included in the model nevertheless, apparently because the research is so clear on this. )

Battelle then devised a formula that the DOE then used to produce "teacher data reports" which would ostensibly measure the effectiveness of these teachers (See here, for a sample version of these reports.)

In October 2008, Chancellor Klein made an agreement with the UFT that the teacher data reports would not be used to evaluate teachers, but only to help them improve their instruction:

“…as a tool for schools and teachers to use for instructional improvement. They are not be used to evaluate teachers…. Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers…”

Klein later went back on this promise, and in February 2010, he instructed principals to consider these reports when deciding whether to give teachers tenure :

“Principals and superintendents will consider the performance of each teacher who is up for tenure more carefully than ever, weighing multiple factors including Teacher Data Reports, where available and appropriate.”

As Gotham Schools reported at the time: “Those teachers who fall into the bottom or top 25 percent of the rankings will be red-flagged, alerting principals that the DOE recommends giving them tenure or cutting them lose [sic] . In total, about 160 teachers will fall into that bottom percentile. “

Though Klein also originally agreed with the UFT to keep the individual reports confidential, as are most performance ratings , and to resist releasing them to the public even if FOILed, he has gone back on this promise as well.

Yet even back in the fall of 2008, when the reports were first provided to principals, it was clear to me and many others that DOE would eventually use them to evaluate teachers, as by nearly all accounts, they have little or no value to helping teachers improve.

I also thought (and believe to this day) it is critical that any model used to determine a teacher’s effectiveness and professional future should be made publicly available, and independently vetted by experts in statistics and testing.

After spending months of unproductive requests to former chief press officer David Cantor and Amy McIntosh, the head of the DOE “talent office”, asking for more information about the model used and evidence of its reliability, I decided to FOIL this information.

Here is an excerpt from my original FOIL request, dated Feb. 23, 2009, along with the partial DOE response that finally came in May 24, 2010, more than fifteen months later:

Dear FOIL Officer:

This is a request for records pursuant to the Freedom of Information Law ("FOIL"), Article 6 of the Public Officers Law. We hereby request disclosure of the following information concerning the Teacher Performance Data Reports:

1) The model specification used in the report to produce estimates of teacher effectiveness;

2) The sources of the data for class size at the classroom and school level over the last ten years;

After many months of delay, here is an excerpt from the DOE response, dated May 24, 2010:

"With respect to items one and two of your request, while certain records being released today may be responsive to these items…The model was not designed to ascertain the impact of class size or other classroom or school level variables…"

Nevertheless, the “draft” technical report from the Battelle consultant indicated a very substantial impact of class size on achievement in math:

“In math...the characteristics that had a negative impact were percent free or reduced price lunch and class size.” (see also Table 5.2 in the analysis.)

Teacher experience level also had a significant effect, in both math and ELA.

In December 2008, more than a year before I filed my FOIL, in a document supposed to allay teachers’ fears, entitled: “Teacher Data Initiative: Support for Schools; Frequently Asked Questions, DOE had claimed that an independent panel of experts had attested to the model's validity and reliability, writing:

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

.So in my FOIL I asked for more information about this panel, including:

3) The identity of the members of the "panel of technical experts" who approved the model and/or methodology of these Reports, as well as the times and locations in which these experts met to discuss these issues with DOE staff;

Yet in 2010, in response to my FOIL, the DOE contradicted their earlier claim:

" With respect to item three of your request, I have been informed that approval of the model and methodology rested with the DOE, and not with any “panel of technical experts.” As a consequence, I have been informed that there are no responsive records that will answer this aspect of item three."

Instead, DOE sent a list of names on a document entitled “Technical Expert Panel: Value-added Data for Teachers Initiative”, dated Sept. 25, 2007 with representatives from the various groups, including the UFT, academia etc.

They also sent a report from a subset of these individuals, entitled “Statement on the New York City Teacher Value-Added Model” dated August 29, 2008, written many months after the teacher data reports were first released, and months after the DOE had claimed that an independent panel had validated their reports.

This statement was written by Tom Kane, then at Harvard and now at the Gates Foundation; Jon Fullerton of Harvard; Jonah Rockoff of the Columbia Business School, and Douglas Staiger of Dartmouth College. Far from validating the reliability of DOE’s methodology, these men expressed numerous reservations and caveats about value-added approaches in general, and made the following points, among others:

"1) Test scores capture only one dimension of teacher effectiveness, and they are not intended to serve as a summary measure of teacher performance…

2) If high stakes are attached, there will be potential to game these measures by teaching to the test, selecting students, altering difficult-to-audit student characteristics, or outright cheating. …

3) To calculate expected test scores…there are likely to be additional factors not yet considered that influence student achievement. …etc. "

In the FOIL, I also asked:"Whether the members of this panel were paid for their services and if so, the source of these funds..."

DOE responded: "With respect to item four of your request, the DOE did not pay any members of the panel for their services. It is my understanding that some members of the panel were paid by the Fund for Public Schools (the Fund) for related research work. Consequently there are no responsive record to provide."

Even though Klein runs the Fund for Public Schools out of Tweed, the DOE claims that it is not a public agency and does not have to release its financial records to the public.

In its 2007 FAQ, DOE had also claimed that there was another document that potentially confirmed the accuracy of their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

So I also asked for a copy of this "research study", as well as a few other items, none of which have been provided to this day.

I last heard from the DOE on August 12 and again on September 10, 2010, saying that the above "research study" is still not complete, nearly three years after the DOE had claimed it existed.

In any event, the DOE has now commissioned researchers at the University of Wisconsin to "update" their teacher data reports, apparently not satisfied with the earlier versions produced by Battelle.

For copies of all these FOILed documents, see the Class Size Matters website ; for more on the problems with the teacher data reports, see today's column by Juan Gonzalez, as well as articles in the Daily News, the NY Times, GothamSchools and the Christian Science Monitor.

The myth of the great teacher, hopefully euthanized once and for all

I hereby nominate the best blog posts of the month so far to be two by Diane Ravitch in EdWeek and one by Skoolboy (Aaron Pallas) in Gotham Schools. These two brilliant critics dissect and hopefully put to sleep for once and for all the great teacher myth, as propounded in a recent Nicholas Kristof column in the New York Times.

Kristof writes, “It turns out that having a great teacher is far more important than being in a small class, or going to a good school with a mediocre teacher.” Okay, so what does that mean? Especially as we don't know how to identify a great teacher, or to produce one.

The corollary of the great teacher myth is that the main thing wrong with our educational system – particularly in urban, high needs schools -- are lousy teachers, and the true evil resides in the teacher unions that protect them.

It amazes me that anyone could actually believe this -- but this is the standard argument in DC think tanks and mainstream foundations. No one would seriously argue that the main problem with our inequitable health care system is that there are too many lousy and incompetent doctors serving the poor– or that the best way to address this problem would be to fire more and more of these doctors, and in their place hire new college-graduates from Ivy League schools– but this is the accepted ideology in education policy circles.

As Diane Ravitch writes in her column, Why Are People So Gullible About Miracle Cures in Education?:

Teacher-bashing has become the motif of the day. It is usually cloaked in some high-minded rhetoric that pretends to praise teachers. Say the bashers: We need great teachers; great teachers can solve all our problems; great teachers can close the achievement gap; if you don't have great teachers, you are doomed; blah, blah, blah. What they really mean—read between the lines—is that they think most of the teachers we now have are no good. We have to start firing the stragglers, the ones whose kids don't get high test scores. The theory is that—emulating Jack Welch at GE—we should fire the bottom 10 percent every year, and over time we will have a staff of "great" teachers because all the bums will be gone.

Recently, I attended a conference where a well-known scholar actually proposed this as the way school systems should function. Just keep firing the "weak" and replacing them with newbies. That way, the teaching force will get continually better. …

The great mystery is why so many people are so gullible about miracle cures when it comes to education. They certainly don't expect miracles in any other part of their life. But the schools just can't seem to shake this belief that all children will learn to the highest standards when: 1) all teachers are great teachers; 2) every school has a brilliant leader as principal; 3) every superintendent has an M.B.A.; 4) every school is run by entrepreneurs; 5) every school is organized around a theme; 6) every school is small; 7) all schools are charters.

In a subsequent column, The Miracle Teacher, Revisited, she directly addresses the weaknesses in Kristof’s argument:

If I read Kristof correctly, a "great" teacher is one who can produce higher test scores. We know that this can happen through relentless test-prepping. Is that what a great teacher does?

But if that is the definition of a great teacher, then we can't possibly identify them until they have had at least three, or better yet, five years in the classroom, so there is sufficient data showing that they produced dramatic gains in their classroom. So, that means that no new teacher—certainly no Teach for America teacher—could possibly be a great teacher, because we don't know whether they are great teachers until they have created a consistent record of big test score gains over three-five years.

Let's suppose that a district uses its data to identify the teachers who consistently produce big gains. What happens next? Do these teachers get assigned to the lowest-performing schools? Which children in those schools are assigned to these teachers? What happens to these teachers if they don't get the big gains in the next years? No one has tried to explain how this would be implemented, whether successful teachers would be willing to go wherever they are assigned, and how their services would be parceled out among many needy students.

Isn't it wonderful that we have economists with tons of data (but no practical experience) to tell us how to find and reward great teachers?

In support of his claim in the pre-eminent value of a great teacher, Kristof cites findings from “A Los Angeles study [that] suggested that four consecutive years of having a teacher from the top 25 percent of the pool would erase the black-white testing gap.” Yet in Gotham Schools, Skoolboy points out that this oft-repeated statistic is an urban legend, originally propounded by Gordon, Kane, and Staiger in a study of LA schools that has since proven to be wrong.

“As eduwonkette pointed out last summer, Brian Jacob and his colleagues have shown that these effects do not cumulate. Only about 20% of the effect remains after a single year, and only about 12% after two years. After two years, then, the 10 percentage point swing is down to about 1 percentage point.”

The LA study was written by three men, the first of whom, Robert Gordon, was hired by Klein to perform the “fair student funding” hit job on NYC schools and write nasty opeds in the Daily news, attacking public school parents for their “obsession” with class size. Gordon now works in the White House (alas!). The second author, Tom Kane, is now employed by the Gates foundation, undoubtedly propounding more myths, such as the key to improving our schools is better data collection and teacher performance pay.

Skoolboy further explains that “the vaunted value-added methods show that a teacher who is great’ one year may not be so hot the following year.” A recent report from the National Center on Performance Incentives reveals the substantial volatility of teacher performance.

In fact, one study from San Diego cited by the report shows that “35 percent of teachers initially ranked in the top quintile remain there in the second year while 30 percent fall into the first or second quintiles of the quality distribution in year two.” Apparently, even “using different tests can affect the stability of estimated teacher effects.”

Oh well. I don’t suppose anyone at the New York Times, the DOE, the White House, or the Gates Foundation is listening – that would be too lucky.