The City Council likes to create task forces to investigate important policy issues and propose changes, but then leaves them unattended and under the entire control of the Mayor, whose office screws them up.
One recent example is Automated Decision System Task force, created by Council Local Law 49 , that was supposed to investigate city agencies’ use of automated decision-making and deliver a report with recommendations.
The law was widely celebrated and promoted by the Mayor’s office as the first attempt by any municipality to embark on a systematic analysis of the controversial trend of using non-transparent algorithms to make crucial governmental decisions in criminal justice, education and elsewhere.
In NYC, algorithms have been used by many cities agencies, and are used by the DOE for school enrollment and admission decisions, as well as well as teacher evaluation and even flagging certain students as potential risk for dropping out through "early warning systems". The use of such algorithms have been criticized as both "black boxes" that are difficult to understand and may further reify various kinds of bias and injustice in a way that is difficult to discern and counteract. According to a hearing transcript, Jimmy Vacca, the bill's prime sponsor, said:
"I strongly believe the public has a right to know when decisions are made using algorithms, and they have a right to know how these decisions are made.When the Department of Education uses an algorithm to assign children to different high schools, and a child is assigned to their sixth choice, they and their family have a right to know how that algorithm determined that their child would get their sixth choice. They should not merely be told that they were assigned to a school because an algorithm made the most efficient allocation of school seats. What is considered to be most efficient? Who decided this? A mathematician, a computer programmer? "
Later, in written response to questions from Education Week, Vacca said he hoped his legislation would also bring transparency and improvements to the district's "inaccurate or erratic teacher evaluations," which he said "can occasionally spit out pretty different scores for the same teachers from year to year, or low scores for good teachers."
The AI institute provided a list of algorithms used in NYC agencies and elsewhere, some of which had egregiously failed. See Gary Rubinstein and Cathy O'Neil, for example, of the travesty that resulted in NYC's absurd teacher evaluation system – and yet the resulting teacher ratings were printed in all the daily papers at Joel Klein’s encouragement.
Although many of the members appointed to this Task force were independent experts in the field, (though none with a special interest in education) , the Mayor's office kept a tight hold on the proceedings, which by and large were not open to the public, and no minutes were kept. The co-chair, Jeff Thamkittikasem, director of the Mayor’s Office of Operations, explained that this was because they wanted to create a “safe space” for discussion. After much media attention and criticism, including a letter sent in March by a number of advocacy groups, they did finally schedule two public sessions.
Yet for months, the Mayor's office refused to provide the members with any examples of the proprietary software the city uses that they could analyze. Thus the problem they were supposed to deal with -- the secrecy and unaccountable nature of these algorithms --plagued the operations of the Task Force itself.
Unsurprisingly, the final report from the task force released on Nov. 19, and was widely criticized, including by many of its members:
Unsurprisingly, the final report from the task force released on Nov. 19, and was widely criticized, including by many of its members:
“It’s a waste, really,” says Meredith Whittaker, co-founder of the AI Now Institute and a member of the task force. “This is a sad precedent.”
Ultimately, she says, the report, penned by city officials, “reflects the city’s view and disappointingly fails to leave out a lot of the dissenting views of task force members.” Members of the task force were given presentations on automated systems that Whittaker says “felt more like pitches or endorsements.” Efforts to make specific policy changes, like developing informational cards on algorithms, were scrapped, she says.
Instead, the city is going to hire an "algorithms officer" who may do little but protect the secrecy of the algorithms and further obscure the methodology and discriminatory results of their use. The AI Institute tweeted, "During their recommendation process, the NYC ADS Task Force: X Didn't provide info about current ADS use in NYC X Ignored recommendations from experts & advocates X Failed to engage our community."
In a scathing article in Fast Company, entitled The first effort to regulate AI was a spectacular failure, Albert Fox Cahn, a member who quit midstream, described how the city had stalled and stalled giving the Taskforce the data and algorithms they needed to make any informed judgements:
In a scathing article in Fast Company, entitled The first effort to regulate AI was a spectacular failure, Albert Fox Cahn, a member who quit midstream, described how the city had stalled and stalled giving the Taskforce the data and algorithms they needed to make any informed judgements:
There were delays, and obfuscations, and then, by the spring of 2019, outright denials. It wasn’t because the data didn’t exist: The city had information on existing systems, including data that can help third parties understand if a model is being used in a context where it is likely to discriminate or have other adverse impacts. They even had model cards, which provide performance data, an explanation of intended use, and details on training and evaluation data, for dozens of different systems, all of which they kept to themselves. This is the sad reality of city politics. The City Council passed the task force into law to hold the administration accountable for its secretive use of algorithms, but that was the last thing the administration wanted.
He describes the final report this way: “The document holds the air of a college paper hastily prepared by a student the day before the deadline. Of the document’s 36 pages, more than half is allocated to simply explaining the task force’s history, presenting the CVs of the members, and providing thanks to those groups that testified at task force hearings. The group’s recommendations, the entire point of its existence, are just eight pages long.”
Yet if the ADS Taskforce was a failure, it was positively brilliant compared to the results of the School Siting Taskforce, created by Local Law No. 168 in Sept. 2018. This law, sponsored by Council Members Gibson, Menchaca, Cumbo, Kallos, Holden, Lander, Levin and Rivera, was supposed to create an “interagency task force” that would facilitate the identification of potential sites for schools by sharing and analyzing data on city-owned and privately-owned empty lots.
This is an important goal, as the current capital plan funds 57,000 seats but has sites for fewer than 17,000 and the process of locating sites is often so slow it can take the School Construction Authority up to twenty years to site a single school in overcrowded communities. Partly as a result of this dysfunctional process, more than 500,000 students attend overcrowded schools, according to the DOE’s own metrics.
This is an important goal, as the current capital plan funds 57,000 seats but has sites for fewer than 17,000 and the process of locating sites is often so slow it can take the School Construction Authority up to twenty years to site a single school in overcrowded communities. Partly as a result of this dysfunctional process, more than 500,000 students attend overcrowded schools, according to the DOE’s own metrics.
Yet the School Siting task force was run out of the Mayor’s office and met only twice, once in February and then held a public final meeting in July, just days before their final report was due.
The only reason the final meeting was held in public was that the Committee on Open Government of the State of NY wrote an advisory opinion that the Taskforce meetings were subject to the Open Meetings law, as it was created by city law and had a specific task to fulfill. The City Comptroller Scott Stringer wrote a letter in support. . I wrote about the final meeting here, which was primarily composed of a 15 minute power point by the SCA president, Lorraine Grillo, with not a single question asked by any of the members who attended.
When I finally obtained the “report” after FOILing it, I discovered it was only 1½ pages long and had no recommendations to improve the school siting process. Neither the City Council appointee nor any of the parent members of the Task Force had any input in the report, which was never publicly released. And though there are two spreadsheets listing over 30,000 publicly and privately owned vacant sites, more than 22, 000 remain unanalyzed and unexamined. Among the more than 7,000 sites the SCA did analyze, only two were identified as possible sites for schools, while hundreds were rejected for reasons that are incomprehensible.
- For example, in District 9 in the Bronx, four sites were removed for "small number of seats to site" yet the capital plan contains 952 seats funded but unsited.
- Five potential sites were eliminated in District 15 in Brooklyn, with the explanation given that there are only “small number of seats to site” and/or “no sites needed” in the district, even though there are 1,396 district seats in the capital plan that are not yet sited.
- District 11 had 17 sites removed with the explanation offered of “small number of seats to site” while there are 2,124 seats funded in the district capital plan without sites.
- District 31 on Staten Island had 216 sites removed for “all seats sited” and 345 sites removed for “small number of seats to site” yet there are 3,844 D31 seats in the capital plan without sites.
I along with other parent leaders, including Shino Tanikawa, who was a parent member appointed by the DOE, and Naila Rosario, President of NYC Kids PAC, wrote a letter to the Speaker Corey Johnson, to complain, with no response.
What lessons can be learned from these two failed Task forces?
Rather than halting the creation of all future Task Forces, the Speaker and Council Members should make sure that in future, the legislation should include the following provisions:
- Any Task Force created by the Council should include sufficient representation from stakeholder groups as well as the City Council itself, and be chaired or co-chaired by a Council Member or staffer.
- From the outset, all there should be a vigorous public outreach and participation built into the law – so that as many good ideas as possible are gathered, and most importantly, so that the Taskforce can help expand the constituency for change.
- Finally, if the Mayor’s office denies critical information to a Task Force necessary for it to do its work, the Council should consider using their subpoena power.
