Tuesday, December 28, 2010

Fact-checking "Waiting for Superman": False data and fraudulent claims

In response to critical comments, I have added clarifications and corrections in bold italics to my original post. I apologize for some sloppy math in not annualizing what was a six year rate in the apparent source article for the film. After further analysis, the movie’s claims remain clearly inaccurate as well as misleading; though the source article was only partially erroneous, at least as far as I can tell. Sorry for the mistake -- and thanks to all my readers, and especially those of you who checked my figures so assiduously.

In the movie Waiting for Superman, nominated for an Oscar as the best Documentary of 2010, the following statement is made:

" ...in Illinois, 1 in 57 doctors loses his or her medical license, and 1 in 97 attorneys loses his or her law license, but only 1 teacher in 2500 has ever lost his or her credentials."

Since the movie was released, these figures have been repeated frequently. They take up five pages in the Google search engine, were cited in the NY Times review of the film, the British newspaper the Independent, as well as Brian Williams of NBC in the television program Education Nation.

But apparently not a single one of these news outlets, or the makers of Waiting for Superman, ever bothered to check them.

While looking for the source of this claim, which is repeated without citation in the movie and its companion book, I came upon a 2007 newspaper article by Scott Reeder of the Small Newspaper Group:

During the past six years, 1 in 2,500 Illinois educators have lost their teaching credentials through suspension, revocation or surrender. By comparison, during the same period 1 in 57 doctors practicing in Illinois lost their medical licenses and 1 in 97 Illinois attorneys lost their law licenses.

"Either Illinois teachers are 43 times better behaved than doctors or they are being held to a considerably lower professional standard than other professions,'' said Jeff Mays, executive director of the Illinois Business Roundtable and an advocate for educator accountability standards. ``Just like doctors and lawyers, teachers are members of an important and demanding profession. It's time that they be held to the same professional standards."

One should note that the data cited in the source article is substantially different from the claim made in the film. In the movie, the period of six years is omitted for the disbarment of physicians and/or attorneys– making indefinite the time span over which the data was collected. The film also says that only 1 in 2500 Illinois teachers have “ever” lost his or her credentials, rather than over six years.

In an effort to verify these claims, I first consulted the annual summary put out by the Federation of State Medical Boards. In reality, 121 doctors lost their licenses in Illinois in 2009, out of 43,670 physicians. That means an average of 0.3% of doctors per year lost their licenses; or 3 out 1,000 per year. Over six years, this would equal 1.8% -- substantially the same as the 1 in 57 figure cited in the source material.

I also checked the claim that 1 in 97 attorneys in Illinois lose their licenses over six years. According to data reported by the American Bar Association, 26 lawyers in Illinois were disbarred in 2009, out of a total of 58,457 - in some cases, by mutual consent.

Since 2001, the average rate of Illinois attorneys disbarred is 32 per year – with more than half of them leaving their professions “voluntarily.” This is an annual rate of about 0.05%, for a six year rate of 0 .3% -- 3 out of 1,000 – not one out of 97, as the source material claimed. As mentioned above, the movie did not specify the time frame over which this disbarment is supposed to have occurred.

The total number of lawyers disbarred in the entire country, either involuntarily or by mutual consent, is 800 per year out of 1,180,386; which is about 0.07% per year, or 7 out of 10,000. The number of those involuntarily disbarred is 441- about 0 .04% or 4 out of 10,000 per year. The six year rate for disbarment nationally would be 0.42% -- about ten times the figure cited in the film of one in 2500 Illinois teachers who “ever” lost their credentials.

I could not find any independent data verifying the number of Illinois of teachers who lose their credentials each year. According to the NY Daily News, over the past three years, 88 out of about 80,000 New York City schoolteachers have lost their jobs for "poor performance." This represents an annual rate of about 30 out of 80,000, or 0.03%, which is about the same rate as attorneys who are involuntarily disbarred each year nationally.

According to the Houston Chronicle, over the last five years, 364 Houston teachers have been fired, out of about 12,000: "Of those, 140 were ousted for performance reasons, a broad category that generally covers teachers not fulfilling their job duties."

So the rate of Houston teachers who lose their jobs due to poor performance is about 0.2% per year - higher than the rate of either doctors or attorneys in the state of Texas removed from their profession annually. For example, only 32 Texan attorneys were disbarred in 2009 out of 75,087; for an annual rate of 0.04% -- at one fifth the rate. 64 doctors per year on average lost their licenses in Texas between 2005 and 2009; out of about 60,000 physicians, at an annual rate of about 0.1 % -- about half the percentage.

Moreover, many more teachers who are untenured and/or uncertified are removed from their jobs for poor performance. Roughly 3.7% of New York City teachers were denied tenure this year, according to the NY Times.

The overall attrition rate of teachers is much higher - many of whom would probably otherwise be cited for poor performance, but who leave the profession either willingly, or "counseled" out. In New York City, the four year attrition rate is more than 40% -- a mind-boggling figure.

In reality, one of the most serious problems plaguing our urban schools, along with excessive class sizes, overcrowding, and poor support for teachers and students, is the fact that we have far too many inexperienced educators revolving through our high-needs schools each year.

Can you imagine if 40% of physicians or attorneys left their jobs after four years? A national emergency would be declared, with a commission appointed to find out how their working conditions could be improved.

Yet instead of examining this critical issue objectively, the movie Waiting for Superman cites false statistics in their effort to scapegoat teachers, unfairly blaming them for all the failures of our urban schools. The film features the views of Eric Hanushek of the Hoover Institute, a well-known conservative critic of equitable educational funding, claiming that the best way to improve our schools would be to fire 5-10% of teachers each year.

To the contrary, eliminating teacher tenure and seniority protections would likely produce an even less experienced and less effective teaching force - especially in our urban public schools, which already suffer from excessively high rates of turnover.

As a parent, I support a higher standard for teacher tenure and more rigorous teacher evaluation systems. I have seen my own children benefit from excellent teachers over the years, but also occasionally suffer as a result of poor teaching, though the latter has occurred as often in schools without union protections as those that were unionized. An improved evaluation system would take into account not only test score data, but also feedback from other teachers, administrators, students and parents.

But at this point, we simply cannot trust the corporate oligarchy currently making policies for our schools to create a fair evaluation system, including those who backed Waiting for Superman, given their proclivity to misuse and distort data, as shown by the egregiously inaccurate figures cited in the film.

Rather than a documentary, perhaps the movie should be re-categorized, with an appropriate disclaimer, as an urban myth.


DeVo said...

Indignant! Tired of falsehoods, lies and other myths that the "media" and current naysayers are spreading. Yet those people who are getting all of the press on "Education Reform" are where they are because of their teachers. Ironic, isn't it.

reino said...

Be careful not to create your own falsehood. "So the rate of Houston teachers who lost their license to teach is about 3% per year" is, like the statement in Waiting For Superman, false. There is a big difference between losing your job and losing your license. Most of what you write above is correct--I know lots of doctors and lawyers in Illinois, and it's not easy to lose your license here.

Anonymous said...

If we're saying that teachers should be held to the same "professional" standards as doctors and lawyers, doesn't it follow that teachers should be paid on a par with these professions, as well?

Anonymous said...

they should go to school as long as doctors and lawyers and work the entire year.

The Perimeter Primate said...

Then there's this trend:


In the past 10 years, the California Medical Board has reinstated the licenses of doctors who were convicted of sexually assaulting patients, defrauding insurance companies of millions and hiring hit men to kill their wives...

The Orange County Register examined records of 123 doctors who sought reinstatement in the past decade after they lost their licenses for misconduct or negligence. More than half were able to satisfy a judge and the medical board that they were fit to practice. Among the 66 who were reinstated, 16 got into trouble again.

Among those was Dr. Andrew Rutland, an Anaheim obstetrician/gynecologist who could lose his license for a second time after the death of another patient...

Rutland was one of a dozen doctors who were reinstated after losing their licenses for negligent patient care. But the records show that even doctors convicted of felonies and sentenced to prison were able to win reinstatement...


Anonymous said...

"they should go to school as long as doctors and lawyers and work the entire year."

I am a teacher, my brother is a doctor. All told, I have spent more years in school than he. My contract calls for 37 weeks of work. My brother's contract has him work every other week year-round, or 26 weeks. For less schooling and 70% the amount of work, he earns about ten times my salary.

Ray Beckerman said...

Thank you so much, Leonie; an excellent bit of research.

I had a hunch that this movie was pure propaganda for the corporate privatizers.

Anonymous said...

I'm amazed that in all the articles and "talk" about education, teachers are the first to be blamed. I'm not saying that they're aren't bad teachers out there, but there's also bad parents. How come parents are never held accountable for their child's education? A teacher can't make a student do their homework, study or arrive to school on time. At some point, parents and students themselves need to be accountable also. Unfortunately, you NEVER hear anyone mention parents' responsibility, it's always the teacher's fault. I think teachers should have to sign off in order for a parent to receive tax benefits for their child. I wonder then, how many of them would show up to parent/teacher conferences and take a more active role in their child's education??

Anonymous said...

I believe the focus of our energies would be better spent on the children. We have all been exposed to teachers who WE thought were less than what we desired. They too have taught me many things. Life isn't easy. I have meet teachers in private schools who I would not want my children to be taught by. My beliefs, my values. What is trash to one person is a treasure to another. If you want something badly enough you will find a way to get it.

danasdream said...

You ignore the newspaper saying "over six years..." When you take the figure of 121 doctors and multiply it by 6 years, and divide by the total number of doctors, you get about 1.66 percent, equal to the 1 in 57 number. Same with the attorneys. For teachers, being fired for performance issues does not result in loss of license, so your numbers are way off. If your math and other skills are from NY Public Schools, you're reinforcing the thing you're trying to argue against.

kaleervt said...

In Vermont in 2009 6 teachers lost their license to teach, the number of FTE positions for teachers was 8,558.30 in 2009. That probably translates to more than 9,000 practicing teachers. But I think the point of the article is that hyperbole does not serve the public or our public schools well. The agenda to discredit public education and disparage teachers is not useful, accurate or necessary.

Leonie Haimson said...

To reino: I have corrected the original statement that this figure reflected how many Houston teachers lost their licenses, as we don't know that info from the article. To Dana's dream; see the new intro and the sections in bold italics -- clarifications and corrections to the column. the movie is incorrect still -- as it omitted any reference to a six year period; and the figures for attorneys were off in the source material as well. thanks to all

Jon said...

No, Danasdream. You're operating under the assumption that the total pool of doctors/lawyers/teachers is static. (In all fairness, the original post makes the same mistake.)

Without more data to extrapolate otherwise, For the doctors, it is sounder to use the results for a single year, 0.3%, as the value for every year. For the lawyers, we are given much better data - a 9 year average. (Although it suggests that the percentage will rise slightly, since 32 is 0.05% of 64000, so the 58457 is a significant population drop.)

Reino - calling the Houston data into question seems legitimate.

Bottom line: The original post has still unearthed enough to make a good case that "Waiting for Superman" has misled the public, even if it wasn't presented in the best way.

Anonymous said...

I think everyone is missing the point. It really doesn't matter how teacher firings compare to lawyers or doctors. The fact that every year only 1 in every 2,727 teachers on average in NYC are fired for poor job performance is staggering. The question should be, "How many should be fired for poor performance?" The difference between the two then represents how many poor teachers are teaching our children. One bad teacher, especially early on in a child's education, can have a profound impact on their chances for success.

Anonymous said...

It does look like the film did play a little fast and loose with the statistics, and it'd be nice if they cited them so they could be verified. But this article doesn't clarify the situation, it just muddies the water further. You can't compare the number of people that lost their licenses in one profession to the number that were fired in the other. That's comparing apples and oranges. And talking about attrition rate in teaching, without talking about attrition rate in other professions is meaningless.

And even if the stats might be a little off, the point the film makes is still the same: tenure doesn't make sense for schoolteachers. What purpose does it serve? It can't be academic freedom; these aren't academics. It's more difficult to fire bad teachers than it should be.

And talking about how teachers shouldn't be blamed, but rather the parents is a complete red herring. Both of them matter. But the difference is that we have the ability to choose which teachers are teaching, whereas we don't have a choice on which parents are raising kids. And besides, it's completely besides the point. Good teachers are beneficial whether the parents are good parents or not.

Anonymous said...

Questioning the validity of the statistics used is a laughable defense. The fact is, as stated, teachers are less likely to lose their credentials than doctors or lawyers (in Illinois, in that time period). The fact remains that "tenure" for primary and secondary school teachers is a poor practice that hurts students. It leads to teachers failing to be held accountable. Furthermore, comparing teaching to being a doctor or lawyer is comparing apples and oranges. There's precious little screener for teachers, namely, a college degree in one of the least demanding disciplines. Doctors have med school, internship and residency, which weeds out those who already have done well on the MCATs and have a college degree in biology (or something similar). Lawyers have gone through college (albeit in whatever they want), but then have gone to Law school and passed the bar. Neither of these professions has tenure, and neither should. Frankly, other than judges (for completely different reasons), no profession in America should have tenure. I'm a military officer, we unfortunately do have a system comparable to tenure, and it dramatically hurts the product.

Anonymous said...

Another thing...why should teachers be paid more? Public school teachers are often paid more than teachers in private school...which is subject to market forces. Considering hours worked and level of education, most teachers are paid too much. If teachers feel some great heartache over their compensation, they have every right to go do something else. Of course, very, very few have the ability to be paid more on the open market.