All is not well with the admissions practices of the specialized high schools.

Josh Feinman, a NYC public school parent, recently published a study showing that the admissions practices at the specialized science high schools -- Stuyvesant, Bronx Science, and Brooklyn Tech --violate generally-accepted educational standards and practices, and may be invalid, unreliable, and unfair. While state law mandates that an exam be the sole criterion for admissions for these three high schools, Chancellor Klein extended its use to other schools across the city, including Staten Island Tech, which previously had a more holistic admissions process, as well as other new selective schools, such as the HS for Mathematics, Science and Engineering at City College, the HS of American Studies at Lehman College, the Queens HS for the Sciences at York College and Brooklyn Latin. Feinman's summary follows:

My wife and I are graduates of the specialized high schools, and our daughter currently attends one. We are supporters of public schools, and would not want to see these schools weakened. But every student – regardless of race, gender, or ethnicity -- deserves a properly vetted system for determining who is admitted to these schools. And that’s not what the NYC Department of Education currently provides.

I recently published a study -- High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools, that raises serious concerns about the admissions practices at these schools. These concerns transcend the issues of race and gender that were the focus of the recent New York Times article which revealed declining rates of admissions among minority students. In fact, the DOE’s failure to investigate whether the admissions test suffers from prediction bias along racial or gender lines is only one of many problems with current practices.

Using test results from the 2005 and 2006 specialized high schools test (SHSAT), which is the sole determinant of admissions, I found a number of glaring violations of widely-accepted educational testing standards and practices.

For example, thousands of students of all backgrounds are rejected with scores that are statistically indistinguishable from those who are admitted. And the NYC Department of Education fails to provide estimates of how well the SHSAT is able to differentiate among students who score close to the admission/rejection line, or whether other criteria could be used to reduce these uncertainties. I made several requests for this information to senior officials at the NYCDOE, to no avail.

Different test versions are used, but no details are provided about how these versions are statistically equated and how accurate that equating is (again, despite requests, and in violation of testing standards and practices). The scaled scores vary across different versions more than the chance distribution would suggest is plausible, suggesting that the equating system may not be leveling the playing field across test versions of varying difficulty – so that students who received certain versions may be more likely to gain admission than those who received other versions.

The SHSAT exhibits an unusual scoring feature that is not widely known, and may give an edge to those who have access to expensive tutors. Someone with a very high score in math and a relatively poor score in English, or vice versa, has a better chance of admission than someone with relatively strong performances in both. Alternative scoring systems would yield far different results, and no evidence is offered to support the current system.

No predictive validity studies have ever been done– not only to see if the test suffers from prediction bias across genders and ethnic groups, but to see if the test is linked to any desired outcomes. In fact, the NYCDOE has never established what specific, measurable objectives the SHSAT is supposed to achieve. Without well-specified objectives and carefully constructed validity studies, there’s no way to know if these admissions criteria are serving their purpose, or if an alternative system would be more reliable.

The SHSAT is widely assumed to produce clear-cut, valid, and equitable results. But for many students who are rejected, they might have been admitted if they’d been assigned a different test version, if the winds of random variation had blown a bit differently, if a slightly different scoring system had been used, or if they’d been made aware in advance of how the scoring was done.

Of course, no admissions criteria is “perfect.” Uncertainty and imprecision are inherent in all decisions, whether they be based on test scores, grades, portfolios, or a combination of the above. Standard psychometric practice is to choose criteria that minimize uncertainties and enable students to demonstrate the skills needed to succeed in ways other than captured on a single standardized test.

The only systematic, objective way to do this is by conducting predictive validity studies, as are regularly carried out for tests like the SAT to help refine the test, and help colleges decide how much weight to put on SAT scores, grades, and other factors in their admission decisions. Overwhelmingly, studies have found that multiple criteria, used in tandem, provide a better guide to future student performance than a single one. Indeed, it’s partly because of such validity studies that psychometric standards caution strenuously against using any single metric as the sole criterion for admission, and virtually all educational institutions use multiple criteria to determine admissions decisions.

The DOE violates accepted psychometric standards, by refusing to provide detailed information about these exams, refusing to carry out any validity studies for them, or even reveal what the tests are designed to accomplish.

We should press the DOE for answers, and more importantly, to reform the system. Formal predictive validity studies need to be carried out. Based on the results of these studies, a determination should be made as to what admissions process is most likely to achieve a specific, quantifiable admissions goal in a transparent, equitable way.

If these studies conclude that it is best to use additional criteria along with a standardized test, the New York State law—which says that admissions to these schools must be based solely on a test—would need to be changed. Whatever admissions procedures are established, all applicants should know their implications.

As parents, we should bring these issues to the attention of our elected representatives, on the City Council and in the State Legislature. I sent a copy of my study to my City Council member Jessica Lappin, my State Senator Liz Krueger, and State Senator Kenneth LaValle -- but have yet to hear back from any of them. This is unacceptable. --Josh Feinman

You can contact Josh for more information at josh.feinman@db.com

---

---