Wednesday, February 6, 2008

More Editorial Nonsense in the Major NYC Newspapers

Can it be that a couple of major NYC newspapers have quietly been acquired by billionaire Michael Bloomberg and turned over to the DOE’s relentless PR machine?

Hard on the heels of recent editorials attacking the integrity of Diane Ravitch (NY Post) and proclaiming the wonders of charter schools (NY Daily News) now comes the latest barrage in these two newspapers' relentlessly right wing editorial assaults on public education. This time, the perpetrator is one Kevin Carey, identified as a research and policy manager at an education think tank called Education Sector.

In an outlandish and mathematically inept Daily News editorial, Mr. Carey regales Chancellor Klein as the education field’s revolutionary counterpart to professional baseball’s number-crunching hero, Billy Beane. To champion the Chancellor’s pilot program of evaluating individual classroom teachers through “value-added analysis,” Carey compares this data driven approach to the one successfully used by the Oakland Athletics’ general manager for assessing and acquiring baseball players. This comparison is fallacious on so many levels, one hardly knows where to begin.

As all baseball fans know, every pitch, swing of the bat, fielding play, and stolen base is a measurable event, now routinely cataloged and amassed in gigantic statistical databases that allow for ready comparison with other players (or with that same player himself) in similar game situations. What aspects of education can be so measured? A child’s asked question, an idea or answer ventured, a reading assignment completed and understood, a new fact integrated into a broader context, an “Aha!” moment of new understanding, a quiz or test passed, a lesson in tolerance, empathy, or social development achieved, an original homework submitted? Of course not. Learning – that is, integrating and imprinting intellectual information in the human mind – is not a visible activity in even the best circumstances. As Einstein reportedly observed, “Not everything that counts can be counted, and not everything that can be counted counts.”

True baseball aficionados -- those familiar with the work of Bill James, for example -- also understand that these now-famous analytical models are almost exclusively multivariate regression models. In other words, baseball general managers like Billy Beane use mathematical models that predict a player’s value or performance from many different variables simultaneously, each variable clearly measurable and each contributing a portion of the total “value added.” These models are mathematically complex, fraught with issues of relevance, cross-interference among variables, and time series interdependencies (respectively called statistical significance, multicollinearity, and autoregressive conditional heteroskedasticity) that must be carefully considered in their formation and use.

Contrast this approach with the DOE’s under Chancellor Klein, where a teacher’s ostensible “value added” is derived entirely from a single variable, standardized test scores, that is itself an arguably spurious measure. Imagine baseball owners paying their players on the basis of just one variable, such as number of home runs. Within a few years, it would hard to tell the New York Yankees from the New York Giants – every Yankee would be 6’6”, weigh 275 pounds, bench press 500 pounds, and hit 40+ home runs per year. With players judged and rewarded on any single variable, the game of baseball would be rendered unrecognizable, grossly perverted from the multiple-skill game it is today.

No less is true in NYC public schools, where absurdly disingenuous arguments by faceless “experts” are routinely being foisted upon an unsuspecting public by dressing them in pseudo-intellectual clothing. In reality, such notions merely support the gradual perversion of public education into a form few of us would recognize from our school days. Instead of oversized, home run swatting baseball players, our teachers and schools are increasingly being pressured by this Mayor and Chancellor to produce one-dimensional, steroid-charged test takers otherwise devoid of the broad skills, experiences, and enthusiasms we used to recognize as desired products of public education. Worse, what teacher (in the classic, Socratic sense of the term) is willing to commit to a 20- or 30-year career in such a stultifying environment?

Note also that on the very same day (February 6) of Mr. Carey’s editorial, the Daily News ran a separate piece lionizing the accomplishments of P.S. 62 in Richmond Hill. The secret of Principal Angela O’Dowd’s success? The Daily News knows the answer, and they feature it right up front, in just the story’s third paragraph where it will most likely be read. “We teach them the tricks and the tips of test-taking,” says Ms. O’Dowd.

As if that wasn't already enough, another Daily News piece, also on the same day, described how students at 23 Brooklyn schools are now being "paid to perform" on Citywide reading and math exams -- "up to $500 for seventh-graders and up to $250 for fourth-graders."

Would somebody please pass me the steroids?


David M. Quintana said...

Great piece Steve...How damn ridiculous comparing our children with millionaire baseball players...

davidb said...

GREAT column, Steve. Seriously, is there a Pulitzer for blog posts? Apart from the particular position, the column is a joy to read: informative and inventive. This deserves the widest readership. Thanks, Steve!

OTAC said...

Interestingly, he comments on your post over on his blog, but you can't leave a comment there. Again, he's dead wrong. He claims that NYC's measures are multivariate, but then goes on to show that you are right -- the ONLY *dependent* variable is the test score:

Moreover, Koss doesn't know what he's talking about. The NYC value-added measures are not "derived from a single variable," they're exactly the kind of complicated multi-variate measure he describes. As the NY Times reported.

The city’s pilot program uses a statistical analysis to measure students’ previous-year test scores, their numbers of absences and whether they receive special education services or free lunch, as well as class size, among other factors. Based on all those factors, that analysis then sets a “predicted gain” for a teacher’s class, which is measured against students’ actual gains to determine how much a teacher has contributed to students’ growth.

Steve Koss said...

I beg to differ with the comment from Mr. OTAC. All those variables he mentioned, while commendable in themselves, serve only to establish a rationalized basis for comparison, the sort of analysis one usually sees in the media as "adjusted for other factors." The truth is, after these adjustments are made, we are STILL just looking at one variable, the student's measured gain in a standardized test score. This is no more sensible in Mr. Carey's baseball context than evaluating every player's yearly home run production versus what could be expected, adjusted for height, weight, age, years of experience, etc. No matter how you adjust it, evaluating a major leaguer (or his manager) on a single dependent variable -- home run -- only serves, as I wrote, to pervert the multi-faceted nature of what you're trying to measure. I resolutely stand by my critique of Mr. Carey's analogy as being totally inappropriate to education and horribly misleading to parents who are not so fortunate as to understand the mathematical intricacies of multivariate regression modeling.

Patrick Sullivan said...
This comment has been removed by a blog administrator.
Steve Koss said...

My apologies to OTAC, I think. I read his comment as a response to my posting, while he/she was apparently quoting from Carey's blog. I didn't see any quote marks, so I thought the comments were OTAC's own response.

My disagreement with Carey's blogged response comes from the fact that I never argued about the use or non-use of multivariate modeling per se. His suggesting THAT as the point of contention is only to misdirect the discussion into irrelevant technical considerations and away from the real fallacy of the Billy Beane baseball comparison. My point was to argue the notion that measuring teachers solely by a single variable -- whether absolute achievement or relative progress versus mathematically adjusted expectations -- is only weakly comparable, at best, to the sort of multifaceted analysis baseball statisticians like Billy Beane employ to evaluate players' "value added" performance. Efforts like Mr. Carey's to legitimize in the public's mind the DOE's teacher evaluation plans by using a baseball analogy are facile and applealing for their recognizable content. And maliciously inappropriate and misleading for the very same reasons.

Leonie Haimson said...

The author of the oped that Steve so adeptly skewers is Kevin Carey -- formerly of Ed Trust and now Education Sector.

Kevin Carey's response is here:

It's amazing how statistically illiterate these Beltway education pundits are.

Leonie Haimson said...

Another major flaw about the school grading system: not only is it based primarily upon one variable as Steve points out -- but one year's worth of a variable -- the difference between last year's test scores and the year before.

Thus, if you keep the analogy on baseball, a hitter could hit 43 homeruns and have an average of .306, but if he hit 45 the year before and had an average of .325, he would receive a lower grade than a player who improved their scores over the year before, even if hehad no homeruns and had an average of .206. I don't think many people would consider this a good way to measure athletic prowess.

This is why certain NYC schools that had 80-90% of kids at grade level got Fs, while others who had up to 70% below grade level and were on the state failing list received As.

Angela O'Dowd said...
This comment has been removed by the author.
Angela O'Dowd said...
This comment has been removed by the author.
Angela O'Dowd said...
This comment has been removed by the author.