Fred Smith on questions raised by the release of some test items from the 2016 3-8th grade state exams

I've looked at the information SED (Commissioner Elia) set forth last week on EngageNY.  It consists of a grade by grade release of 75% of the reading passages and multiple-choice items that appeared on the April ELA exams--by way of Pearson to Questar.  All of the material and questions that required constructed responses have been provided, as well. I did not look at the math test.

It is true that the amount of operational test material and the number of items disclosed is more than was given out in each of the prior three years of Pearson's core-aligned testing.  And since 2012, this is the earliest this has happened.  [Note: When CTB/McGraw-Hill was the test publisher during the NCLB years, the complete test was accessible to the public on SED's web site within weeks of its administration, along with answer keys. Item analysis data followed shortly thereafter.]

Upon review of the just-released spring 2016 testing output, however, certain useful data have not been made available. SED has been moved to offer us a translucent view of the exams, but it still is not being entirely transparent.

In order to make the SED information more accessible to reviewers, I re-cast it in the attached Excel workbook. It shows the name of the reading passages that were released, the type of item involved (M-C or CR), the sequence number of each item and the word count of each passage.

In addition, there are four separate measures of readability (also referred to by Pearson/SED as the "complexity metrics"). And beside these numerical indexes, there is a column called Qualitative Review, where the appropriateness of the material is judged.  The outcome of SED's review process was that all 55 ELA reading passages were deemed to be appropriate. The keyed correct answer to each M-C item is presented at the end.

 I have four misgivings about SED's presentation:

1- Twenty-five percent (25%) of the passages and multiple-choice items that counted on the tests remain unreleased.  This amounts to six reading selections and a combined 40 items that appeared in Book 1 on the operational exams given on April 5^th.  Why should that be the case in light of the Commissioner's goal of providing more information?  Non-disclosure raises questions about the quality of the material being withheld. If you remember the contract, the 2016 operational material came from Pearson's item bank.  Since this is supposedly Pearson's last year and since NYS owns the material, why hold back?

As a consumer issue, how can SED justify that we, taxpayers, purchased a product we cannot see?  [And now that Pearson is on the way out, what about releasing the mounds of 2013-2015 test content that no one has been allowed to talk about?  Or did Pearson’s 5-year contract concede ownership of the still hidden material to the vendor?]

2- Item statistics have not been made available.  This kind of overarching data based on how the test population performed on each item is useful to researchers, analysts and anyone interested in seeing how the items functioned.  The items are the bricks that go into constructing the exams and on whose strength and quality decisions about children, teachers and schools have come to depend. 

Prior to Pearson, CTB/McGraw-Hill posted the item-level analytic data within months of test administration.  This included item difficulties (p-values = percentage of children choosing the correct answer) on multiple-choice items or the average score on constructed response questions.  CTB also showed how students responded to each distractor—i.e, the proportion of students choosing the wrong answers. Such data provide insights into possible weaknesses in the items (ambiguous choices, more than one best answer, distractors that are non-functional).  And a correlation was provided (known as the item discrimination index) showing the relation between performance by students on an item and their performance on the entire test. The expectation is that students who do well on an item also do well on the test.

This full set of statistics—referred to as classic item analysis data—ceased being presented after Pearson won the testing contract.  Since then, only some statistics have been provided—and more than a year after the operational tests have been given—ensuring that the exams could not undergo scrutiny until after Pearson’s next round of testing had taken place.

In the absence of empirical data, a vacuum that SED created, the department has been able to blunt criticisms of the exams—at first, dismissing them as anecdotal, and then, when the complaints became widespread, providing partial data (p-values and discrimination indexes) but well past the time SED had complete information readily on hand, yet didn’t make any available to those who might otherwise have had facts with which to challenge the exams. [Aside: I remember when a member of Governor Cuomo’s Task Force attempted to stifle Lisa Rudley’s critique that the Common Core Standards and core-aligned exams were being advanced without sound research data to prove their efficacy.  He pointedly asked where her evidence was to support complaints about flaws in the exams.  Given SED’s reluctance to dispense information, this was a preposterous question.]

The demand for complete timely data is not academic or trivial.  Let’s look at just-released Item# 37 from the Grade 6 ELA passage Weed Wars.  You may recall that Leonie Haimson came upon and brought to light information that one passage contained the confusing concept of “impossibly improbable”.  

Here now from SED’s rush to divulge 2016 information is the statement in question and the multiple-choice options 11-year olds had to choose from in answering.

Once in a while, changes to a weed’s DNA would allow that weed to survive the glyphosate.  The chances of changes like this were very, very small.  But when farmers used glyphosate years after year on millions of hectares¹ of crops, “what seems almost impossibly improbable becomes more probable,” Duke says.

37.   What is the meaning of the phrase “impossibly improbable” as it is used in lines 21 through 23?

         A   usually certain

         B   highly unlikely

         C   extremely slow

         D   rarely noteworthy

This item won’t reach the game-changing level of ridicule that Leonie’s exposure of the Pineapple and the Hare did in 2012.  But it underscores the value of having statistics available to evaluate items.  What percentage of kids chose the correct answer (B)?  How did the distractors work?  That is, what proportion of children chose each of the wrong answers?  Having analytic data would enable us to see how this dubious item played out. 

According to the Learning Standards, #37 is coded RI. 6.4, which means it was classified as a Reading for Information item to see whether sixth graders can “determine the meaning of words and phrases as they are used in a text, including figurative, connotative and technical meanings.”

My guess is that most kids got the item right—making this an “easy” item—because the distractors seem implausible. I would mark it as a poor item for two reasons: It can be answered correctly without reading the passage from which it is drawn; and the distractors likely didn’t carry much weight. Of course, my hunch may be off.  Perhaps many children chose A in response to this convoluted question. We shouldn’t be left to speculate, however, in the absence of data that SED has in its possession.  Note: SED already has the statistics sought virtually as soon as the tests are scored or else it couldn’t issue the instructional reports it just distributed as referenced in Elia’s June 2016 letter to colleagues,

3- SED took away information it provided from 2013 – 2015 when it released questions with annotations.  The information was posted in August of those years in EngageNY. In this year’s zeal to reveal more items sooner, SED has not presented the statewide p-values for the items as it had over the last three years. Significantly, the annotations, which SED described as teaching tools, are also gone.  So, thus far this year SED has offered more items but has not included a rationale “to demonstrate why any of the released questions measures the intended standards; why the correct answer is correct; and why each wrong answer is plausible but incorrect.”  It is helpful to gain SED’s perspective about the material and its defense of the correct answer choice.  Unfortunately these explanations have not come out.  I think we should campaign to have SED’s annotations for the 2016 material released immediately,

4- SED failed to follow its own decision-making rules regarding which reading passages were appropriate to include on the operational exams.  Four ways to estimate the readability of potential passages were used in constructing the ELA tests: the Lexile Framework, Flesch-Kincaid, the Degrees of Reading Power and the Reading Maturity Metric (a Pearson measure). Each involves a scale that can be applied to reading material and sets forth a range that is appropriate for each grade.  For example, the Lexile Framework indicates that reading material ranging from 740L – 1010L is appropriate for 4^th and 5^th graders.  Ergo, material outside that band may not be right for children in these grades. 

SED and Pearson applied three of the four methods to each selection and said in releasing this year’s material that “to make the final determination as to whether a text is at grade-level and thus appropriate to be included on a grade 3-8 assessment, all prospective passages undergo quantitative text complexity analysis using three text complexity measures…. Only passages that are determined appropriate by at least two of three quantitative measures of complexity and are determined appropriate by the qualitative measure of complexity are deemed appropriate for use on the exam.

In reviewing SED’s latest data on the 2016 exams, I counted 11 of the 55 operational passages as failing to meet the criterion that they had to be found to be appropriate by at least two of the methods.  I don’t know how SED will resolve this contradiction.

~~~~~~~~~~~~~~~~~~

Two final observations: The material just released by SED makes no mention of the Common Core Learning Standards—as had been the case in the Released Questions with Annotations of 2013-2015. Instead, SED has kept the boilerplate found in these releases and reverts to New York state p-12 Learning Standards as the framework it follows.  Nor could I find any reference to “college and career readiness. I guess they have been discarded due to the botched implementation of the Common Core.

Finally, I think we should press SED for the missing information outlined above and keep demonstrating how the department and commissioner continue to pose as being responsive, while taking business as usual actions.  Once we let up, they will fall back to disdaining that messy part of democracy—the will of the people.

- Fred Smith

Warning! Field Tests ahead!

For all of you parents who thought the testing season was over, here is an alert from Fred Smith, testing expert:

Memo To Parents and Guardians of 88,000 NYC Children Regarding Upcoming Field Tests:

Good morning. It’s late in this school year. I hope that you and your sons and daughters are well.  I know you have been busy with everyday matters.  Therefore, the field tests your children are expected to take are not high on the list of things you want to hear about.

Evidently, NYC Chancellor Farina respects that too and has chosen not to disturb you with information about them.  But, we know that you are capable of multi-tasking.  Here are a few points you should be aware of:

Over the next three weeks, field tests will be administered to students in grades 3-8.  The tests may be given any day through June 10.   

Field tests let vendors like Pearson try out questions on children in order to develop future exams. 

Since 2012, Pearson has given separate field tests in June (aka stand-alone field tests) to a large sample of students throughout New York State.  Results don’t count for the students.

The way students respond to the try-out material is used to select the reading passages and items that will go on the English Language Arts and Math tests to be given next April to New York State’s 1.2 million test population—which encompasses 440,000 children in New York City.

The cost of the field tests is borne entirely by taxpayers, free of charge to their producer.  They take less than an hour to administer but disrupt the school day.  

New York City’s Department of Education does not call attention to the field tests.  It fails to mention that 774 of our schools have been assigned to give them or that 87,330 children have been targeted to try out items in the service of a commercial test publisher.

The bulk of the stand-alone field testing won’t begin until May 31.  That’s when 85% of the ELA and Math tests are due to start.  These are the “Paper-Based” Tests.  The PBTs are aimed at 517 of our schools and 61,000 students.

We have also been asked to do “Computer-Based” field testing for the first time.  This experiment will kick off on Monday, May 23 and involve 11,000 city children.  Enlisting their participation is intended to help transition us from PBTs to computerized exams.

Introducing CBTs will accelerate the profit-making encroachment of technology in the classroom. This objective must be important. Just this week the State Education Department posted a cheerful online letter anticipating “technology issues” but assuring us that the benefits outweigh the bugs.

Over the last four years, the stand-alone approach has proven to be ineffective.  Teachers have roundly criticized the poor quality of the resulting operational exams given each April. They complain that the test material is inappropriate and the items are flawed. Yet, the bad exams were built via the same kind of field testing process that will be repeated here over the next few weeks

A large majority of parents (and many teachers, too) have not been informed about the upcoming tests or their purpose.  The DOE has made little effort to notify parents in a forthright way.

Chancellor Farina, who speaks about the importance of “parent engagement,” apparently wants the field tests to go forward without parents knowing much about them or learning that taking them is voluntary.  Some parents might even say they don’t want their children to participate.

In fact, since 2012, more and more moms and dads have become aware of these extra tests and have refused to let their children sit for them.  Here is a letter parents can submit when they choose not to not let their children take the field tests.

Please find out if your children are in a school that has been assigned to give the field tests by checking the list of schools posted here.

Even if your child is not in one of the grades that is scheduled to be tested, please alert other parents in targeted schools about the field tests and share information with them. 

At least you will have a chance to engage in what’s happening and decide whether you want to exercise your right to say Yes or No.  Knowledge is power.  Thank you. -- Fred Smith

NYC KidsPAC Report Card for Mayor de Blasio

See the NYC Kids PAC press release below the report card.  Please leave your comments if you agree or disagree with our assessment!   Reposted from NYC Kids PAC website here.

DeBlasio's Ed Report Card 2016 Final by Leonie Haimson

For immediate release: May 18, 2016

Contact: Shino Tanikawa, 917-770-8438, estuaryqueen@gmail.com

Leonie Haimson, 917-435-9329, leoniehaimson@gmail.com

NYC Kids PAC Issue Education Report Card for Mayor de Blasio

Today, NYC Kids PAC released their second annual report card on Mayor de Blasio, grading him in several education categories based primarily on whether he has followed up on his campaign promises.   The report card is posted here: http://tinyurl.com/huu57xa

It has been more than two years since Bill de Blasio became Mayor, and Carmen Fariña was appointed Chancellor.  The good news is that the Mayor has restored the district structure, increased funding for the arts, rescinded the ban on cellphones, and imposed a moratorium on closing schools, though the moratorium has now ended.  He has also begun to reform school discipline, without providing sufficient resources or staff to ensure a positive school climate.  

The members of NYC Kids PAC include four sitting Presidents of Citywide and Community Education Councils, three past presidents of CECs, and one member of the Panel for Educational Policy.  Though these individuals would like to make it clear that they speak for themselves and not on behalf of their organizations, they have deep experience of how the current governance system and policies work, or do not work, for parents, students and schools.  

The report card exhibits particular disappointment with the lack of parent input at the school, district and citywide levels.  Citywide and Community Education Councils remain largely disempowered, with little or no say as to co-locations and space planning, and the DOE has argued in court that School Leadership Teams have only advisory powers, in an effort to keep their meetings closed to the public. School overcrowding and class size also continue to be major concerns.

Naila Rosario, President of the CEC in District 15 in Brooklyn, explained: “As a parent in one of the most overcrowded districts in the city, I am disappointed by the Mayor’s response. Our district, which is economically and culturally diverse, has experienced enormous growth as gentrification and immigration have expanded. Families are anxious as they wonder where their children will attend school, because enrollment is capped at their neighborhood schools.  Students are crammed into overcrowded classrooms year after year, and we have experienced the stress of losing space for specialists, interventions and even lunch. We urge this administration to show leadership and to press for expedited, responsible approval and construction of new schools.”

Gloria Corsino, President of the Citywide Council for District 75, says: "As a public school parent and committed advocate for all children and children with special needs, I remain hopeful that the campaign promises that Mayor De Blasio's made continue to move in the direction that they were stated.  I look forward to the growing relationships of CEC's and Citywide Councils with the DOE and becoming more than simply advisory boards, as parents are huge stakeholders in the education of their children.”
   

Eduardo Hernandez, President of the CEC in District 8 in the Bronx adds, “Although this new administration has taken some strides toward engaging parents and addressing their concerns, most of the time this engagement is simply done to fulfill regulations and mandates.  Parents are still dealing with some of the same issues that have been plaguing our school system for years,  like overcrowded classes, student enrollment,  and lack of diversity,  just to name a few."

“Parents are disappointed that specific promises de Blasio made during his campaign to reduce class size have not been fulfilled, especially considering that this is their top priority for their schools, according to DOE’s own surveys.  In addition, transparency and accountability in spending has not improved, and large contracts with vendors who have engaged in fraud and other questionable activities continue to be pushed through and approved by the Panel for Educational Policy,” points out Leonie Haimson, Executive Director of Class Size Matters.

Shino Tanikawa, President of NYC Kids PAC as well as the CEC 2 in Manhattan and co-chair of the Blue Book Working Group concludes, “While we recognize Mayor de Blasio and his administration have made some improvements, we want to remind him of the unmet promises he made during his campaign.  We hope this report card will motivate him to review his campaign promises and implement more of them in the coming year.”

Skinny award dinner June 9 honoring Juan Gonzalez and Robert Powell; please buy your tickets now!

Class Size Matters is holding our annual "Skinny" award dinner on Thursday June 9, presenting a Lifetime Achievement award to Juan Gonzalez, investigative reporter, who just retired from the NY Daily News.

Juan has done more than any other human being to uncover corruption in city spending and the charter school sector, as well as having cast light on many other education issues over the course of his illustrious career.

We are also honoring Robert Powell, who recently resigned from the Panel for Education Policy. Robert, as the Bronx representative on the Panel, cast the lone vote against a huge DOE contract to be given Computer Consultant Specialists, which had engaged in a massive kickback scheme just a few years ago.  This was a story Juan covered so well that City Hall eventually rejected the contract, saving taxpayers hundreds of millions of dollars in the process.

To buy your ticket for this special evening honoring Juan and Robert, along with a four course dinner and wine with friends and allies, please click here .  Last year we sold out, so please reserve your tickets now!

Please post your comments & observations about the ELA exams here!

Today is the first day of the NY State ELA exams in grades 3-8.  As in past years, I am reaching out to students, parents, and teachers to let us know if there are confusing or ambiguous questions, overly long reading passages or passages with commercial product placements or vocabulary that is anachronistic and/or not age-appropriate.  Please note if there are any other issues that concern you or make the exams flawed or problematic.

This year the exams are also untimed.  How long did it take your child or students to complete the questions?

It was nearly four years ago, on April 18 when I first learned from a comment on our blog about the absurd Pineapple passage on the 2012 ELA exam.

 After I did a little research and blogged about this passage the next day, the story of the race between a Hare and a Pineapple with no sleeves quickly blew up into a national scandal.  To this day, the Pineapple has endured as emblematic of the lack of accountability on the part of the test-makers and policymakers, who insist on using these exams to rate schools, students and teachers.


Please also note below if you have info about the opt-out rates at your schools.  Already we have received reports of high rates in some Rochester-area districts and at PS 261 in Brooklyn.

thanks Leonie