Saturday, February 25, 2012

A principal at a high performing school explains why she is "absolutely sick" about the public release of the TDRs"


Here is a message from Elizabeth Phillips, Principal of P.S. 321 in Brooklyn, who writes: “ I am absolutely sick about the public release of the TDRs.  See below for some details in terms of what it actually means at PS 321.”
Principal Liz Phillips; credit NY Press
              Having seen the TDRs when they first came out, I can say that they are extremely inaccurate, both in terms of actual mistakes and in how data is interpreted, particularly for teachers of high performing children.   Here is some more detail on that:  
                1.       The amount of data that is simply wrong is staggering.  In my school alone, the first year of the TDRS, for just two grades (since of course that is all we have getting TDRs) 4-6 teachers have inaccurate data as follows:
                ·         One teacher who taught in 08-09 but was on child care leave for years before that time has data for a previous year-impossible...it must be data from someone who was in that same room the previous year.
                ·         For both of my upper grade CTT (inclusion) classes, the special education teacher has a data report that is for all 29 kids; the general education teachers in those classes have no TDRs. (This does appear to be corrected for 2010.)
                ·         In one case, a teacher who has taught 4th grade for 5 years has no data for previous years.      
                2.       Even in cases where the data is correct, I believe the conclusions are arbitrary and often flawed.  I do not believe that because the average scale score went from 3.97 (taken from students' third grade test scores) to 3.92 (their fourth grade scores) a teacher is necessarily a poor teacher, but in this case, she ended up in the 6th percentile for this particular year due to a statistically insignificant change.  In fact, the particular teacher in question is an exceptionally strong teacher by any other measure (parent feedback, colleague's opinions, my observations over many years).  As in the School Progress Reports, particularly when you get to high levels of achievement these small differences are not meaningful.
I realize that some would argue that since teachers are compared to "like" teachers, the data is accurate-however, when you actually know the teachers it just isn't so.  We have great teachers with very high scores, very low scores, and middle scores.  In this example, whether we are saying 3.97 or 3.92 (with a perfect score equating to 4.5) the class average is extremely high.  Once you get to this high level, small changes are meaningless in terms of tracking children's progress.  
The assistant principals and I have often debated one or two ambiguous questions, since the answers are not always clear!   It is wrong to call a great teacher a failing teacher because a few kids got 3-4 questions wrong one year rather than 2-3 questions wrong the year before.  It is particularly problematic given that the 3rd grade test in the past was very different from the 4th grade test.  It could be that the children in a particular class were always weaker in writing, but the 3rd grade test for the years the TDRs are being released had very little writing compared to the 4th grade test, so the children may not actually do worse; it may be that they are just tested on different material.
                3.       Related to the above, there are many reasons why the data may not truly be comparable from class to class.  Some of it has to do with the differing tests from grade to grade (which will be improved at least in terms of the format of the test as of the 10-11 year), but there are other factors too.  Even though we work hard to make all of our classes equal in terms of academic level, behavior, etc, there is no question that in certain situations where children have recently gone through a very traumatic experience, I will hand pick a teacher who is strong academically and also nurturing.  So, one teacher may have some "harder to teach" children, even if they are children with the same test score.     
There is also the issue of who the AIS (Academic Intervention Services) teacher is for each class.  She may have a big impact on the test scores in some classes, yet that isn't taken into account.  Or, it could be that in some schools we decide that with kids scoring high 3s and 4s we devote more time to non-tested subjects--art, music, dance, drama--while in some schools more test prep is done.
4. We know from the School Progress Reports how inaccurate grading based on minute differences in test scores can be.  One example:  PS 321 was in the 83rd percentile for 2010-11, the 95th percentile in 2009-10, the 59th percentile in 08-09; the 36th percentile in 07-08 and the 57th percentile in 06-07.   (Click on chart above.)
Basically, there is no way that our school has changed that dramatically year to year.  In fact, the difference in grades wasn't great (B, B, A, A, A), but the percentile is ...and that is with 550 tested kids in the sample.  When we're looking at a class of 29 kids, a couple of kids having a bad day can make a huge difference.  I know that the "average" range on the TDRs is huge because of the DOE's awareness of the inaccuracy of looking at small differences, but there are two problems with this.
First of all, whether the DOE says it is average or not (which it is according to the DOE), parents seeing a teacher rated in the 30th percentile are going to be upset!  And, as I note above, it doesn't even really seem to work for the very top or bottom.  As many statisticians have written, there is no data that supports using "value added" in the way that these TDRs do.   Just FYI, here is a chart I prepared that shows how the percentiles fluctuate wildly from year to year, even with the % of children getting 3s and 4s in ELA and Math stay fairly constant and with over 550 test children!  The fluctuations for a sample size of 29 children (a class) will be even greater and the percentiles meaningless.
5.       The idea of the TDRs being publicly released with names attached is incredibly demoralizing to teachers--and this includes ones who scored above average.  Because they understand that some of their well respected colleagues scored low, there is the feeling that this is arbitrary, and that "this could me next year. "  I worry that some of the best teachers, who are the ones who have options for jobs elsewhere, will leave the system.  The timing of this is particularly problematic given that four years of budget cuts have made teachers jobs that much harder, with higher class size and fewer support services.  I think it is clear that when teachers are demoralized they cannot do as good a job teaching, so it is the children who will suffer.       
                6.       To improve the quality of education in a significant way, we need to get thoughtful, high performing, dedicated young people to enter the profession.  Treating teachers disrespectfully, which is what I believe the public release of TDRs with names attached does, will make teaching-at least in public schools--a less and less appealing option for high performing college graduates with other choices.    This is partly Bill Gates' argument in his New York Times op ed piece.  
                7.       I honestly cannot understand how public ranking of teachers by percentile will have anything but a negative effect on teaching and learning.  Particularly in middle school, I can imagine teachers losing control as children and parents take the position, "why should I listen to you, you're a below average teacher." 
                There are many other very serious problems with the TDRs.  There is no question that as testing becomes more and more high stakes, with teachers' jobs dependent on student test scores,  in many many schools, the curriculum will be narrowed.  I believe it will lead to a widening of the "achievement gap" since it will be much easier for high performing middle class or upper middle class schools with very involved parents to resist the impulse to narrow the curriculum.  With low performing schools, the temptation will be greater as they face state and city sanctions that can result in school closure.   In all elementary schools, it will be harder to get senior teachers to teach grades 4-5...until of course everyone is tested in every grade, which will just make it hard to get knowledgeable people to teach in public schools at all!            

29 comments:

cynthia said...

I've been a nyc doe teacher for 17 years (drama specialist at PS58 @ before that at PS29) and feel SO disgusted by this that I wasn't even going to comment. I just wanted to thank Liz do much for taking the time to respond in such a thorough and thoughtful way.
Cynthia Blackwell
mom to Jean-Michel & Olivier 2.5
and DOE d15 arts educator

Joel said...

It is nice to see a principal who realizes how inaccurate and ridiculous these evaluations are.
It is the duty of every principal in the city to ignore the data,inform the teachers that they will not be evaluated either positively or negatively based on this nonsense.
They must also meet with the parents and explain how inaccurate and flawed these results are and in no way will any decisions be based on these results.

brooklynbreeder said...

As a parent of NYC public schoolers, I am very dismayed and embarrassed to think that educators are being placed in such an untenable position supposedly on my behalf.

aph said...

I also want to echo Cynthia and just say thank you to Liz for putting this out there. Although Schoolbook says they are "putting the data in context," there is so much information that the average reader does not have when they look at the numbers. I appreciate that you are helping to clarify that.

Cheers929 said...

I'm 33 years old. I've been teaching 3rd and 5th grade for 11 years. My TDR says I'm average...in the 30s. My dad said to me today, "Back to work tomorrow! Now let's work on improving your score." And he began to laugh- a bit if a nervous laugh because apparently he saw the look of anger instantly appear on my face. I told him it wasn't funny and not to get me started. The truth is, at that moment, i realized MY DAD was judging me on my TDR!!! And to him, to anyone, it looks like I'm a failure. How sad is that!?!?! I work hard, I've putt school and my students ahead of so many other things over the years. And yet, all you see, all my dad sees, is that I'm ranked in the 30th percentile. That makes me sad.
Mary Willson

Former Teacher said...

No matter what, these scores will live on the internet for a LONG time and when people are looking for other jobs or promotions, I am sure that hiring committees will google their names and these numbers will come up and cast a person in a different light. Honestly, I believe the numbers have far reaching effects on people and may be a person's scarlet letter.

Catalog Jewelry said...

I coached a nationally ranked 16u travel baseball team in 2010. The team was comprised of 18 spots, which 300 boys tried out or inquired about playing on my team. Our goal was simple, win. We played over 130 games, ending with a record of 97-36. What's impressive about our record is that 70% of our games were against elite 18u teams. 8 players on this team are considered pro prospect. 4 players received full scholarships to play division 1 baseball. My coaching style is solely merit based and no promises are made to any players regarding playing time. The team is solely merit based and players performance and preperation are continually evaluated. Players who underperform are put on probation and replaced on roster if they do not show marked improvement. This is done to create a competitive culture among teammates. This system works and my players buy into it.
I asked one my D1 scholarship players would I have the same success coaching his school team, he replied" all due respect coach you wouldn't last one day in my school". He explained to me that in his school only a handful of kids were interested in playing baseball. I would have to build my roster from a smaller sample of players who may not be as commited as the kids on my travel team. It's simple supply and demand. My competitive coaching style while sucessful in travel baseball would be perceived by many on his school team as alienating. This example points out the paradox teachers face today and the many variables not covered by TDRs. Teachers can't cut kids with low scores from the classroom as I do in tryouts. Schools have fixed demographics and a fixed number of students who all must make the team. Kids on academic probation require more services and support vs getting cut from the team.
Many parents and coaches find my coaching approach controversial and disagree with many of my tactics. This I leave up to you to decide. As a coach my performance is measured by wins and loses. What constitute wins and loses for a teacher and how do we keep score is the million dollar question?

Tom said...

The DOE reports that there is no relationship between a school's demographics and the ratings of its teachers. It follows, then, that "teacher effectiveness" is not a factor in the achievement gap. Or else the ratings are completely invalid.

Anonymous said...

The only site that seems to have the data is http://teachrate.com

jim said...

I think that we can all agree that the testing and system are flawed. I also think that most reasonable people feel education is extremely important, and advocate strongly for a meritocracy. Anyone who disagrees with this is, in my opinion, most likely concerned with self-preservation, at the expense of our children. I suspect that teachers would be the ones who are best-equipped to decide how to measure merit. I propose to handle it the way the government handles regulating industries. They'll allow the industry to offer proposals, and if they don't do it, then they are left with what the government decides. I am upset as a parent that the teachers union doesn't propose a better system than seniority. So this is the imperfect system that we are left with, and it's the one that I intend to use to judge teachers, until they propose something real themselves. We're all waiting. Seniority is not a solution.

Anonymous said...

I can't help but wonder if corporate reformers jump for joy every time they hear a good teacher say he or she will leave if this keeps up.

If there is a teacher shortage, then it provides an opening for distance learning and other technical substitutes for the teacher-student relationship.

Ttony Da Fighter said...

Liz is right! However, to me, the problem is mayoral control with an erratic, illogical and mentally unstable mayor and a judge in the same state of mind as the mayor. The TDRs should not have been released even if they were accurate because this is not the way you treat human beings, professional or not. Embarrassing people in such a way is a form of violence against their dignity and humanity and does not solve the problem. It does not help children to do this. It's a self-serving act of a blood-thirsty despot meant to demonize, and hurt people. That's all it is. I would advise all my colleagues to refrain from trying to defend themselves on public forums because this gives too much importance to this nonsense. That's what it is, 'NONSENSE!'

Anonymous said...

How do teachers feel they should be graded? And please do not tell me its complicated bc it is not that difficult when there are schools in the poorest, most dangerous & uneducated neighborhoods yet these schools still manage to get high schores & HIGH SCORING STUDENTS year after year. So if teachers are not responsible for their students grades tests or otherwise who the hell is??? Teaching means they should be learning - enough to inprove throughout the year- maybe not get 100's but if they start at a 50 or below they should not only be at a 60 by June. Maybe if kids were not off so much they would do better with shorter breaks in between their studies bc clearly there are some kids that do not retain the information & being off to 'play' does not help matters.

I fee like all school staff wants to say the City is the reason why public schools are doing so poorly but at the same time school staff are the 1st people to take credit for the progress??? Things need to change & ppl need to take responsibilty FOR THEIR JOBS.

Anonymous said...

You asked who else is responsible If teachers are not? How about the parents! Where is the public release of information regarding parent involvement! Let's list the parents who never answer a phone call, never attend parent teacher conferences, don't make sure homework is complete. Teachers can't control if students are getting enough sleep and healthy meals. There is only so much teachers can control. It's a joint effort. Parents need to take responsibility for THEIR KIDS.

Anonymous said...

Also please cite your sources, which schools are you referring to when you say there are schools in the poorest most dangerous and uneducated neighborhoods receiving high scores year after year....

Cheers929 said...

Wow, Anonymous, why no name attached? Perhaps because you don't want to be embarrassed later when people respond to your ignorant comment. Clearly you do not work for the DOE. Teachers are not saying that WE shouldn't be evaluated, use common sense. What WE are saying is that rating a teacher based SOLELY on STANDARDIZED TEST SCORES, with a 53% margin of error is ABSURD! Is that how you think students should be evaluated as well, based on their score on ONE reading and ONE math test per year. There are so many JOBS that a teacher is responsible for which have nothing to do with TEACHING specific content. And for the record, I did not become a teacher "to get kids to pass a standardized test" which is what this PROFESSION, not job, has become.

jim said...

@Cheers,

I am a parent. Let's have a mature discussion. If you agree that you should be measured, and are capable of saying that this does NOT work, please, as an expert in the field, propose to me how it WOULD work. Offer something constructive. How should we measure teacher merit? We're all ears....

DrPhrogg said...

@Jim- how do you evaluate doctors? One doctor has a 75% survival rate while another has a 100%. The Dr with 75% is a CCU emergency physician and the other is a dermatologist. How do you evaluate teachers? I saw an increase of only 2% while a colleague had 12%. Who is better? I had honors students who went from 95% to 97% and he had basic troubled learners who went from 45% to 57%- still not passing. My honors students participated in a state-wide competition. some years we did very well (top 10) and some years not so well. Same teacher, same style, different kids. All got good grades and passed the standardized test with flying colors. On another point, I taught science. Should I be graded on student's writing ability and math scores? Or should we give up 2 weeks of instruction to run annual tests in all subjects? The article points out flaws in the program, but political reformers reject the input from teachers as "flawed by self interest". Teaching is an art, not a book of knowledge. How should you evaluate artists?

Anonymous said...

What about measuring the IQ level of each student? Everybody knows that that the low IQ level of the learner affects a lot the educational outcome, the performance on the standardized tests. Should we be more aware of what kind of students particular teacher teaches?
Publishing the students' IQ levels along with the teacher's score would give us more precise picture of the TDRs.

Anonymous said...

What about measuring the IQ level of each student? Everybody knows that that the low IQ level of the learner affects a lot the educational outcome, the performance on the standardized tests. Should we be more aware of what kind of students particular teacher teaches?
Publishing the students' IQ levels along with the teacher's score would give us more precise picture of the TDRs.

Anonymous said...

What about measuring the IQ level of each student? Everybody knows that that the low IQ level of the learner affects a lot the educational outcome, the performance on the standardized tests. Should we be more aware of what kind of students particular teacher teaches?
Publishing the students' IQ levels along with the teacher's score would give us more precise picture of the TDRs.

cheers929 said...

Jim...

I appreciate your response. That shows me parents ARE looking into TDR further, thank the Lord! Yes, let's have a mature discussion. I re-read my post several times and can't seem to find where I stated that I was an expert, but thank you for the pat on the back.

As in ANY and ALL jobs/professions, workers should certainly be evaluated based on what is expected of them. Currently, the DOE has an evaluation form that is completed by each school principal at the end of the school year, which states, "satisfactory," or, "unsatisfactory." The list contains criteria such as attendance, punctuality, observations of lessons taught throughout the year, etc. (After an observation, a principal puts into writing a narrative of your lesson, including positive feedback and suggestions for improvement.) These observations and yearly evaluations must be signed by both the principal and pedagogue. It is then kept in the teacher's "file" for eternity. Problems can arise with this as well such as a teacher stating that, "The principal doesn't like me...that's why I didn't get a good write up for my lesson." Anyway, being that this IS how we currently ARE evaluated each year, why is that not taken into account IN ADDITION to rating a teacher on improving a student's standardized test score??

cheers929 said...

Let's think about it with me as an example. In all of my school years, I was a good student. I believe my high school average was 88 when I graduated. I was not a great math student but I tried. I asked for help from the teacher, I had peer tutoring... I was an athlete. I played basketball, softball, and swam on the swimming team all four years. I was part of the choir, volunteered, became a Eucharistic minister... and, I managed to have a fun social life, too. BUT, when it came to taking standardized tests...I admit, I was not very good. In fact, knowing that, made me care even less about the P-SAT and SAT tests. The combination of having a difficult time sitting through a long, timed test, long reading passages and essays, difficult math concepts and equations...I did not do well! Actually, I did SO poorly on the P-SAT as a sophomore in high school, I was called into a meeting with the guidance counselor, my parents, and teachers to figure out what was going on with me. I was embarrassed! But, I tried my best, my parents encouraged me, enrolled me in a boring Saturday morning test-prep course, and although I improved quite a bit on the SAT exam, I did not, "Break a thousand." I took the test AGAIN, because hey, what did I have to lose if they took the best scores, and I got the EXACT same score, 990. With the exception of the University of Delaware where I was put on the waiting list, I was accepted to every school I applied to including the College of Mt. St. Vincent, Brooklyn College, Sacred Heart University, St. Francis College...

cheers929 said...

The POINT of my honestly is not to post an autobiography, but to put things in perspective. Obviously, I did not get accepted into those schools based on my SAT score. It was a combination of my SAT test, academic success during high school, and other extracurricular activities that gave a thorough picture of WHO I WAS AS A STUDENT. I was not being judged based on my standardized test score and NO STUDENTS SHOULD! In that respect, my teachers shouldn't be held accountable for my SAT score and then THEY get rated on how well I can do on a standardized test.

I am not trying to convince you or anyone else of anything other than the fact that in order for a person, any person to be evaluated, everything must be taken into consideration.

Preparing students to do well on a standardized test is NOT what teaching is about and unfortunately, that's what it has become for the New York City teachers. I feel terrible for these kids. There's no room for creativity, character building, critical thinking, etc. It kills me to think that these kids are going to look back at these years and have TESTING right in the forefront! Let kids be kids. Let kids learn in fun ways...in a variety of ways that suit their needs. THAT'S what teachers are trained to do. THAT'S what matters. Should my career be scrutinized in the public? NO! What was the purpose of releasing this unreliable data to the public??

cheers929 said...

I am upset and frustrated. I love my job. I love being with kids and I learn from them and laugh with them every single day.

I am a hard worker. I am dedicated. I won't let this current situation change who I am as a teacher. If you want to judge me by my TDR (which is 32 and considered average, which makes no sense) then at least judge me on everything I have to offer and everything I do for "my kids."

Now...it's 10pm. I made it to my doctor appointment, came home and made dinner and ate with my husband and baby girl, bathed my daughter and put her to bed, set up my clothes and lunch for work tomorrow, responded to this post, and...as much as I'd like to sit and watch the Godfather on AMC, report cards are due soon so I've got to continue where I left off last night.

Have a good night.

Anonymous said...

It is the Mayor who has been totally ineffective regarding improving schools. He said he would be held accountable but instead is scapegoating the teachers and schools. Since the testing debacle, all the papers write about is teacher evaluations, closing schools, etc.etc. This Mayor was a disaster for the public school children of New York City.

jim said...

@Dr. Phrogg,

did you answer a question with a question? Drs. do not belong to a union and try to protect jobs through seniority. Patients have a right to choose their Dr., amongst a lot of choices, based on reviews and referrals. Public schools are districted. I am not saying it's EASY to judge, but I am saying that you MUST be judged. Regrettably, too many people just talk about how it's impossible. That's simply unacceptable, so you are left with what we have now. Tell Mulgrew to get with the program and come up with a reasonable idea. The more he digs his heels in, the less sympathy you'll have from parents.

Anonymous said...

Jim,

Doctors can decline patients. My father's cardiologist said if he does not stop smoking he will not see him as a patient.

Teachers and schools can't refuse students.... well,at least not in public schools.

jake said...
This comment has been removed by a blog administrator.