The uselessness of the interim assessments that the DOE purchased for $36 million

The NYC Department of Education is mandating that all schools administer standardized interim tests three times a year, also called benchmark or formative assessments.  Here is the excerpt from the DOE handbook, “Instructional Principles”, describing these assessments, which are euphemistically call “screeners”. 

Alternative standardized assessments can be selected by schools but must be approved by DOE central.

According to the UFT, the first round of testing for grades K-10 will occur starting Sept. 27 and will continue through Oct. 22.  Students in grades 3-10 will be administered computer-based tests during this period, and K-2 kids will be assessed via “individual 15-minute question-and-answer sessions.” Given that many kids in grades 1 and 2 are in classes of 28 or more, these sessions could take a teacher seven to eight hours of class size over this three -week period.

This new regimen of commercially-prepared tests according to the DOE will cost $36 million – more than twice the amount they are spending for their “class size pilot”. 

Some of us remember how in the first phase of Children's First under Chancellor Joel Klein, similar interim assessments were required at every school, along with literacy and math coaches and a unified curriculum, similar to the regime of reforms that DOE is now attempting once again.  Here is an excerpt from a DOE 2005 press release, heralding these measures as leading to gains in state test scores:

Our new Citywide core curriculum, coaches, professional development, interim assessments, and aggressive intervention programs for struggling students, coupled with the enormously hard work of our teachers and other school staff, all contributed to these tremendous gains.

These same test score gains, however, were not seen on  the NAEPs,  the more accurate and reliable national assessments, and when the state exams were re-calibrated in 2010 to eliminate the rampant test score inflation that had occurred over this period, the gains disappeared.

This is not surprising.  As I wrote recently  in the Gotham Gazette, there is little or no research showing standardized interim assessments help students learn, or reliably diagnose their learning problems.

Few if any independent peer-reviewed evaluations have offered evidence for their validity or demonstrated that they have any positive impact on learning. One of the few randomized studies showed that administering the MAP exams had “no significant effect” on achievement. Many teachers have critiqued iReady exams, and many students despise them. 

As the late Robert Slavin wrote, director of the Center for Research and Reform in Education at Johns Hopkins University:

Research finds that benchmark assessments do not make any difference in achievement. …In a rational world, these findings would put an end to benchmark assessments, at least as they are used now. The average outcomes are not just small, they are zero. They use up a lot of student time and district money… Interim assessments fall into the enormous category of educational interventions that are simple, compelling and wrong. Yes, teachers need to know what students are learning and what is needed to improve it, but they have available many more tools that are far more sensitive, useful, timely, and tied to actions teachers can take.

Why are these assessments so relatively useless?  Because by and large, they are disconnected from the actual curriculum and work being done in classrooms. UFT President Michael Mulgrew pointed out  in 2019,when then-Chancellor Carranza first proposed imposing these tests, “How do you use a standardized formative assessment when you don’t have any sort of standardized curriculum….You don’t even know what you’ll be measuring.”

Indeed, after the first round of Children’s First in NYC, interim assessments were quietly eliminated after a few years, along with all those literacy and math coaches.   

Scott Marion, the Executive Director of the Center for Assessment  explains,

Interim assessments do not have a de facto place in balanced assessment systems. In fact, my colleagues and I argued in our Tricky Balance paper and policy brief that interim assessments more likely create unbalanced systems….Widely-used commercial interim assessments, in particular, generally are not tied to any specific curriculum and are not necessarily coherent with instruction and other assessments in the system…commercial interim assessments have a limited role, at best, in balanced systems of assessment, and any role must be supported by positive evidence that outweighs negative consequences.

In Seattle, the MAP tests were dropped in 2013 in all high schools after a successful boycott of teachers and students, and made optional in other grades.  Teachers pointed out how they didn’t test students on the material they were learning in the classroom.   

And last spring, after many Chicago students were found to be taking  hours and even days  to complete the open-ended MAP exams, the Chicago Board of Education dropped its contract with North West Evaluation Association, the company that produces them.

There are also serious privacy concerns with these exams.  The MAP exam attempts to measures student engagement and self-regulation by a student’s pattern of clicks and responses.   NWEA claims that “Teachers can use this information to identify students who may benefit from SEL (social and emotional learning) interventions.”  Parents have expressed concerns about the company’s use of this data, and the fact that NWEA hasn’t signed the Student Privacy pledge. 

Recently, a Brooklyn parent wrote me: “One of my sons already took a round of MAP when his middle school began administering them ….  The data still follows him, the test was so "ridiculous and meaningless, unrelated to what he was learning" (his words) and the predictive analysis so deeply flawed (forecasting that this honor roll student would not graduate HS or go to college on time).

She added: “I will be opting both of my HS students out of MAP and will continue to advocate for what our schools and all students really need to flourish and be safe, especially this year.”

Other NYC parents, students and teachers should seriously consider doing the same.

Watch Democracy Now's coverage of the successful Seattle boycott of these exams nearly a decade ago. 

Revelations at today's hearings on high-stakes testing and testimony by Emily Carrazana

Teacher Sara Steinweiss and actor Anthony Ramos at today's hearings

Update:  the Daily News ran a good story about these hearings, including the surprise announcement by DOE in the midst of a hearing about excessive testing in our schools that they plan to introduce a new set of standardized tests administered four times a year in our schools. Here is the WSJ  article , focusing on Anthony Ramos and his experience in NYC public schools.

Today's NYC Council hearings on "Breaking the testing culture" were fascinating.  CM Treyger did a great job in asking piercing questions to DOE, especially about the NY State high school Regents exams and why they don't encourage the creation of more Performance- based assessments like those used by the New York Performance Standards Consortium.

There was also compelling testimony from Anthony Ramos, now a Broadway actor who had a lead role in Hamilton and is now starring in the film "In the Heights".  He spoke about how he failed multiple Regents exams at New Utrecht high school, and struggled in school.  His high school had class sizes of 35 which severely hampered his learning: "There are like 35 kids in a class with 35 different issues in their head and one person to deal with that , who probably doesn't even want to teach that because it's just in the curriculum."   

His testimony and that of the Sara Steinweiss, the NYC theater teacher who recognized his talent, put him in a play, found a college scholarship for him and changed his life, is posted here, along with an animated short about his story. Video of the second part of the hearings with testimony from Linda Chen of DOE as well as students, teachers, and parents is here.  

Several Consortium students and teachers also talked about how engaging and challenging the education provided by their schools has been in comparison to the rote learning that the Regents exams require.  When Linda Chen, Chief Academic Officer of DOE was asked why the DOE doesn't celebrate the Consortium school practices, she said it's a complex process to implement performance-based assessments. CM Treyger responded "So is dismantling inequity in our schools, but it's worth doing."

CM Treyger asked about the cost of testing. DOE officials spends $3 million on the administration of Regents/3-8 state exams and prep materials, $1.9 million to Pearson and for "administrative support" for the SHSAT test, and $4.4 million for the gifted exams -- totaling nearly $10 million in all.

The most shocking thing was to hear Linda Chen, Chief Academic Officer of DOE confirm that they intend to implement a uniform system of standardized formative assessments four times a year in place of current school-selected assessments, in order to "streamline" them, because "we can't compare results with so many different tests."

This will undoubtedly lead to more meaningless and wasteful testing and test-prep. They can't completely eliminate the locally-chosen school-based assessments because by the UFT contract, those are selected by individual schools for the purpose of teacher evaluation.

I recall when DOE purchased Acuity exams for $22 million from McGraw Hill, and imposed them as "formative assessments" across all schools and nearly all grades.  After a certain amount of time these exams were dropped because they weren't aligned to what was happening in the classroom and found to be neither predictive nor diagnostic.  They flopped and were eventually dropped.

More on this in the WSJ, which reports on what Chancellor Carranza said on Monday:

 "...he plans to 'dipstick' schools several times a year in every grade to see whether students are on track. He refers to these as “checkups” to measure progress rather than waiting for the “autopsies” of state tests in the spring. While he said many schools are skilled in conducting such periodic assessments that give teachers real-time information, his team is looking for ways to make the practice more uniform."

CM Treyger asked Chen about the Edustat system that the DOE plans to build, the new data system modeled on Compstat.  Chen didn't have any estimate of how much the system would cost or what data it would include, but it is likely that these interim test scores will be fed into this system -- rather than more predictive and important data like class size. 
 
He also asked Chen whether the DOE has done any analysis to look at the relationship between test scores and class size, and of course she said not.  In short, it doesn't seem like DOE is interested in breaking the testing culture at all.
 
Below is the testimony of Emily Carrazana of Class Size Matters, about her experience with the specialized high school exam, the SHSAT, and why that exam as well as the exams for gifted programs should be eliminated. 

Testimony on High Stakes Testing before the City Council Education Committee

 September 24, 2019

Thank you, Chair Treyger for holding these important hearings and for the opportunity to testify before you today. My name is Emily Carrazana, I work at Class Size Matters and I attended public elementary and middle schools in the Bronx. Beginning in the 6^th grade, I would trek up to Bronx Science two or three days a week, and for most of my summers to participate in the Dream/Specialized High School Institute (SHSI), a program designed to help prepare low income and high achieving students for the SHSAT.

After many hours of sacrificed time out of my childhood , I did not get into any of the specialized high schools, despite my good grades and high scores on the state exams. . My parents, first generation immigrants, did not know the first thing about the bureaucratic process that is the complex high school admissions process in this city. So, when I was rejected from the specialized high schools, they did the only thing they thought they could do.

They moved our entire family out of the state to neighboring New Jersey. I ended up graduating from my town’s public high school, successfully completing AP and International Baccalaureate courses and went on to earn my bachelor’s degrees from Rutgers in three years. My results on the SHSAT were no indication of where my abilities stood back in the 8^th grade, just as they are not a valid marker for success for any student today.

While many people argue that eliminating this exam and/or gifted programs will cause the families of high-achieving students to move out of the city, the example of my family shows how the opposite happens currently because of the use of an unfair high-stakes exam – which has been shown not only to discriminate against students of color but also high-achieving girls.

The SHSAT is an invalid and biased exam. While nearly all of the discussion and debate has so far revolved around the way in which it leads to racial disparities, this exam has also been shown conclusively to be highly gender-biased. Though NYC girls receive higher test scores on the state exams and better grades, they are accepted into the specialized high schools at much lower rates. Here are last year’s results by gender, revealing a gender gap of eight percentage points. ^[1]

Gender	#stud tested	% students tested	#got offer	% of total offers
F	14,116	51%	2,206	46%
M	13,405	49%	2,592	54%
Total	27,521	100%	4,798	100%

The disparity at the most selective schools such as Stuyvesant is even greater. Last year, 56% of those admitted to Stuyvesant were boys and only 44% were girls.^[2]

The fact that the SHSAT is biased against girls has been conclusively proven by Jon Taylor, a research analyst at Hunter College, who has published his research findings in a peer-reviewed journal.^[3] He discovered that girls who are admitted to the specialized high schools with the same test scores as boys do better on their course work and receive higher grades, including in the most advanced courses.^[4] His research also shows that a student’s 7th grade point average is the most valid predictive factor for success at New York City high schools in general, including the Specialized High Schools.^[5] This merely underscores the need to eliminate the SHSAT as the sole determinant for admissions to a NYC school.

We oppose the use of high-stakes testing in general and the SHSAT in particular. No other school district in the country bases admissions to any one of their schools on the basis of one test alone.^[6] Moreover, this practice has long been opposed by the American Psychological Association, the National Council on Measurement in Education, and the American Education Research Association, whose standards contain the following statement: “Any decision about a student's continued education… should not be based on the results of a single test, but should include other relevant and valid information.” ^[7]

As the National Academy of Sciences has explained, “current psychometric standards… recommend that a decision that will have a major impact on a test taker should not be made solely or automatically on the basis of a single test score, and that other relevant information about the student’s knowledge and skills should also be taken into account.”^[8]

If we really want more diverse, integrated schools throughout our system, we must rely on multiple measures, including grades and more holistic factors. In addition, we should discourage tracking as much as possible – another form of segregation that occurs within schools that merely widens the achievement gap between racial and ethnic groups. ^[9] Separating out kids by “ability” has been shown to disadvantage those who are concentrated in the lowest–performing classrooms.^[10] This is why we also oppose segregated gifted programs, especially those based on a high-stakes exam given to children as young as four.

Yet only about four percent of all elementary school children are in gifted classes.^[11] As NYC schools begin to enroll students of diverse backgrounds, both racially and economically, this will further highlight the reality that children from disadvantaged backgrounds need more support from their teachers to reach the same goals, and that equality isn’t the same as real equity.^[12] Integration alone without small classes cannot erase those differences.

Moreover, teachers often understandably complain that it is very difficult to individualize instruction with students of different achievement levels, and indeed it is especially challenging given the large class sizes we have in NYC. But if class sizes were lowered, this would make teachers’ jobs much easier.

In Finland, when the government decided to stop tracking in the middle grades, the national teachers union successfully demanded systematic reductions in class size, to ensure that they could meet the needs of students of different academic levels. Both the elimination of tracking and the concurrent lowering of class sizes contributed to the rapid improvement of Finnish schools in the 1970's, along with the elimination of most standardized tests.^[13]

The Department of Education must also reduce class size to allow teachers to deepen their interactions with students and meaningfully individualize instruction. All students benefit from smaller classes in terms of heightened engagement, fewer disciplinary problems and increased learning, but, as studies show, students of color benefit the most. This is why class size reduction is one of very few reforms proven to work to narrow the opportunity and achievement gap. ^[14]

As Shino Tanikawa, co-chair of the Education Council Consortium (ECC) and a member of the School Diversity Advisory Group, and Leonie Haimson of Class Size Matters concluded in an op-ed published in the Daily News last May, the integration of classrooms must be accompanied by class size reduction if we want to provide true equity to all children. ^[15] This op-ed is included as an appendix to my testimony.

As NYC schools begin to enroll students of diverse backgrounds, both racially and economically, this will further highlight the reality that children from disadvantaged backgrounds need more support from their teachers to reach the same goals, and that equality isn’t the same as real equity. Integration alone without small classes cannot erase those differences.

In our schools, class sizes have increased substantially since 2007 and are 15% to 30% percent larger on average than class sizes in the rest of the state. More than 336,165 students were crammed into classes of 30 or more this fall. In the early grades, the number of first-through-third-graders in classes of 30 or more has ballooned by nearly 3000 percent since 2007. Our schools can never provide students with an equitable chance to learn with classes this large.

As we eliminate the use of high-stakes testing and move to create classrooms and schools that are diverse in race, ethnicity, gender, language, ability and more, we need to simultaneously push for small classes because this will make truly differentiated instruction possible.

Thank you for the opportunity to testify to you today.

________

^[1] https://nycpublicschoolparents.blogspot.com/2019/03/dismal-results-and-gender-bias-remains.html

^[2] http://www.gothamgazette.com/opinion/7871-new-research-shows-shsat-less-valuable-predictor-than-middle-school-grades

^[3] Jonathan Taylor, Fairness To Gifted Girls: Admissions To New York City’s Elite Public High Schools, Journal Of Women And Minorities In Science And Engineering 25(1): 75–91 (2019) at: http://www.dl.begellhouse.com/journals/00551c876cc2f027,294b56436594090b,2e036b8a364ae7df.html See also: https://hechingerreport.org/the-problem-with-high-stakes-testing-and-women-in-stem/

^[4] Jonathan Taylor, Fairness To Gifted Girls: Admissions To New York City’s Elite Public High Schools, Journal of Women and Minorities in Science and Engineering, 25(1): 75–91 (2019).

^[5] http://www.gothamgazette.com/opinion/7871-new-research-shows-shsat-less-valuable-predictor-than-middle-school-grades

^[6] https://www.nytimes.com/2012/09/28/nyregion/specialized-high-school-admissions-test-is-racially-discriminatory-complaint-says.html See also Chester E. Finn, Jr. & Jessica A. Hockett, Exam Schools Inside America's Most Selective Public High Schools, 2012.

^[7] https://www.apa.org/pubs/info/brochures/testing see also: http://www.aera.net/About-AERA/AERA-Rules-Policies/Association-Policies/Position-Statement-on-High-Stakes-Testing

^[8] National Research Council. 1999. High Stakes: Testing for Tracking, Promotion, and Graduation. Washington, DC: The National Academies Press, p169 at: https://doi.org/10.17226/6336.

^[9] https://www.theatlantic.com/education/archive/2019/04/gifted-and-talented-programs-separate-students-race/587614/

^[10] https://nepc.colorado.edu/sites/default/files/pb-options-10-tracking.pdf

^[11] https://nycpublicschoolparents.blogspot.com/2019/09/what-both-supporters-and-critics-of.html

^[12] https://www.nydailynews.com/opinion/ny-oped-lower-class-size-and-school-integration-go-hand-in-hand-20190517-jhwyfyxzhvcrrmp5p2vqoeieau-story.html

^[13] Samuel Abrams, Education and the Commercial Mindset, 2016 , p. 281 and footnote 3 on p. 382. Also https://newrepublic.com/article/82329/education-reform-finland-us

^[14] See the numerous research studies at: https://www.classsizematters.org/research-and-links/#opportunity

^[15] https://www.nydailynews.com/opinion/ny-oped-lower-class-size-and-school-integration-go-hand-in-hand-20190517-jhwyfyxzhvcrrmp5p2vqoeieau-story.html