DOE's irresponsibility in employing AI products regardless of whether they protect student privacy

A week ago, the NY Post featured an article about a new AI program call Yourai sold by a company called LINC, or The Learning Innovation Catalyst, that the DOE is piloting in some Brooklyn schools.  The product is supposed to help teachers develop their lesson plans.  On Twitter last week, I pointed out the idiocy of the DOE administrator who claimed this would help teachers "think creatively."

What? “Teachers can enter students’ needs and the standards they want into the app and a lesson plan will be generated, Ross explained..She called it a “game changer” that will give teachers more time to “think creatively”. Ridiculous! https://t.co/aOFbQh3qp6

— leonie haimson (@leoniehaimson) March 16, 2024

I went on to point out that two of the three testimonials on the website from NYC teachers appeared to be fake, as I couldn't find their names in a list of DOE employees.

No evidence that 2 of 3 NYC people quoted with testimonials on Yourwai website were employed by DOE through 2023 after checking payroll records on SeethroughNY. There are NyC teachers named Michael Davis, but no admins w/ below names. If anyone knows differently pl let me know! pic.twitter.com/xkVI7YkqhY

— leonie haimson (@leoniehaimson) March 17, 2024

Today, the NY Post followed up with another article, pointing out that there were apparently eight fake testimonials from NYC teachers on the website, and that after being asked about this, the company said their names "were anonymized for compliance purposes," and have now been taken down..

Apparently, the co-CEO of the company, Jason Green, is a close pal of the Chancellor, and he and his family vacationed with the Chancellor's family on Martha Vineyard last summer.  The article added that LINC has received $4.3 million from DOE since 2018 for "professional development and curriculum," including $2.3 million so far  this school year.

What they did not mention is that, aside from the likely shoddiness of the product and the fake hype surrounding it, there are real concerns about these sorts of products including the risk to student privacy, as I pointed out on twitter.  

AI products are  well known for gobbling up huge amounts of personal student data, and then using it to improve their products and create new ones.  Yet this is specifically prohibited by the regulations of NY State's student privacy law, Ed Law § 2-d.

These regulations clearly state that "Third-party contractors shall not sell personally identifiable information nor use or disclose it for any marketing or commercial purpose" and that "Commercial or Marketing Purpose means the sale of student data; or its use or disclosure for purposes of receiving remuneration, whether directly or indirectly; the use of student data for advertising purposes, or to develop, improve or market products or services to students [emphasis added]."

I also pointed out that any district vendor or other third party with access to personal student data by law is supposed to have a specific privacy addendum to its contract.  This addendum is supposed to be posted on the DOE website here, but none can be found for LINC or YourAi.  Sadly, DOE continues to flout the law when it  comes to protecting student data and the transparency required by Ed Law § 2-d, as we have noted in the past.

On twitter, I highlighted specific weaknesses in LINC's online privacy policy, including that they allow other companies to track user behavior, including “3rd parties that deliver content or offers” meaning marketing.

Correction: I finally found privacy policy on company website saying they comply with 2D but unclear how as provide no details & says they allow other companies to track user behavior including “3rd parties that deliver content or offers” ie marketing. https://t.co/a8jr52LLcR

— leonie haimson (@leoniehaimson) March 17, 2024

I also noted that the Privacy Policy said that the company reserved the right to change it at any time for any reason without prior notification to users by changing wording online.  This violates FERPA, because then, districts are not in control of how student data may be used or disclosed.

 

After noting these red flags on twitter, the co-CEO Jason Green DMed me:

We are a minority company that has been partnering with NYCPS for years. Our mission is to help teachers better support learners. I am also recently married and a dog-lover. Would you be open to learning more about us? I would love to better understand your perspective as well.

I said sure, and then asked to see his contract with DOE, to ensure that it contained the required data privacy and security protections.   I didn't hear back until yesterday, when he said he was "working with his team" to get the contract, but assured me that they don't "directly" collect or use student data.  

When I asked what "directly" means, he said they don't collect student data at all.

Then, later that day, on Friday March 22, I went back to look at the company's Privacy Policy and noticed it had been updated that very day:

Low and behold, there was a bunch of new sections added, including that the company indeed "may have access to student data" or "teacher or principal data" as defined under Ed Law § 2-d: 

They had revised the section that previously said the company may change the Privacy Policy without prior notice.  It now says  "We will send advance notice of any upcoming changes to our Privacy Policy via e-mail."  The section about allowing other companies to use user data for marketing purposes was taken out, but this passage that replaced it is not much more reassuring:

Also, Third Party Companies may want access to Personal Data that we collect from our customers. As a result, we may disclose your Personal Data to a Third Party Company; however, we will not disclose your Personal Data to any Third Party Company for the Third Party Company’s own direct marketing purposes. The privacy policies of these Third-Party Companies may apply to the use and disclosure of your Personal Data that we collect and disclose to such Third-Party Companies. Because we do not control the privacy practices of our Third-Party Companies, you should read and understand their privacy policies.

So what does it say in the actual, DOE contract with LINC, that  legally binds their use and protection of student data?  Sue Edelman of the NY Post FOILed the contract from the NYC Comptroller and sent it to me on Friday.

To make a long story short, the only LINC contract the Comptroller's office had was this one from 2020, which never mentions Ed Law § 2-d, though law was passed in 2014, and doesn't contain its required provisions.  

Instead, the contract glosses over the entire issue of student privacy, and says instead that it complies with Chancellor’s Regulations A-820 "governing access to and the disclosure of information contained in student records." Yet Chancellor's Regulations A-820 has not not been updated since 2009. 

In his blog today, Peter Greene has one of his excellent take downs of the whole notion of AI producing better lesson plans than actual living teachers.  He includes this   quote from Cory Doctorow:

We’re nowhere near the point where an AI can do your job, but we’re well past the point where your boss can be suckered into firing you and replacing you with a bot that fails at doing your job.

But beyond the lamentable mechanization and degradation of education that is being promoted by NYC and other districts nationwide, in the name of mindless innovation, the DOE apparent lack of interest in protecting student privacy and following the law remains appalling.  

Scurrilous fact-free NY Post article about briefings on upcoming CEC elections; please apply to be a CEC candidate yourself!

Yesterday, the NY Post ran a sleazy and fact-free article attacking Shino Tanikawa for a workshop she gave a month ago to inform parents about the upcoming CEC and Citywide Council elections.  Shino is a friend, a long-time parent leader, and currently the well-respected Manhattan representative on the NY State Board of Regents. I refuse to provide a link to the article but it attacked Shino for giving this briefing, because it was co-sponsored by the UFT.

The reporter, Mia Walsh, claimed that the workshop was somehow unfairly biased in favor of pro-UFT positions, even though the two parents quoted in the article who attended the briefing were unable to provide any evidence that would back up this claim:

Two attendees included Deborah Kross, a representative for the Bronx on the Citywide Council on High Schools, and Steve Stowe, president of CEC 20 in Brooklyn....Kross and Stowe found the boot camp to be informative, covering the application process and education law that governs them, but questioned the UFT involvement.

But somehow, they and the NY Post reporter remained suspicious, though they were unable to cite any example of bias: 

"I  think it’s naive to say that the UFT doesn’t have their own interest in all of this,” said Stowe. “That’s something that’s hard to communicate sometimes because the message is that parents have to always support teachers,” he said. 

Parents supporting teachers?  What a radical idea.  

The article also featured a scurrilous personal attack on Shino by Deborah Kross, without bothering to quote any of the hundreds of parents and advocates who admire Shino's principled positions and hard work to improve our schools over many years,  as the former President of CEC2 and as an appointee to many NYC task forces and DOE working groups.

For those who may still harbor suspicions of this briefing, I asked Shino for a copy.  It is posted below.  Please comment if you see any sign of political bias or favoritism to the UFT.  

Parents, please consider running for a position on the Citywide and/or Community Education Councils.  In recent years, a few CECs have been taken over by right-wing zealots, including one CEC which passed a resolution urging the Gov. Hochul to veto the class size bill -- a resolution that was full of factual errors and did not represent the wishes of their constituents, as class size reduction has been the top priority of K12 parents in that same district nearly every year that DOE parent surveys have been given. 

The deadline to nominate yourself to be a candidate is Feb. 15, only ten days away, and the process is simple.  For more information, check out the DOE website here. 

Interview with Sue Edelman, intrepid reporter; and her so-far unsuccessful effort to get DOE to respond to FOILs, despite a 2018 win in court

In the latest Talk out of School podcast, Daniel Alicea interviews investigative reporter Susan Edelman about some of the many scandals she's covered and uncovered for the NY Post over more than twenty years concerning the NYC Board of Education/Department of Education, from the days of Mayor Giuliani till the present day.  It is well worth a listen.  She explains how the Special Commissioner of Investigation for Schools no longer does the sort of sweeping investigations that the former SCI Ed Stancik did prior to mayoral control.  This not altogether surprising since the SCI is also appointed by the Mayor.  They're far more quiet too; in fact, the SCI only put out two press releases in 2021, and none so far in 2022.)

Sue calls herself "semi-retired" and is a freelancer, though she is still making headlines for the NY Post on a regular basis.  The latest include her series of articles about how ADASA- NY, the Association of Dominican -American Supervisors and Administrators,  recruited teachers from the Dominican Republic on temporary visas, and then forced them to rent overpriced rooms in apartments owned by ADASA officials, including Emmanuel Polanco, the principal of the school where many of them were assigned to teach. 

One of the other issues Sue discusses on the podcast is the NY Post 2016 lawsuit against the DOE for failing to comply with their Freedom of Information Law requests in a timely fashion.  I wrote about this lawsuit at the time, along with my own terrible experiences of DOE stonewalling for months and sometimes years despite ordinary FOIL requests.  In 2017,  the Village Voice reported that it took the DOE an average of 103 days to respond to FOIL requests, a longer time than any of other city agency.

In 2018, the NY Post won its lawsuit, as reported by Sue at the time:

DOE not only turned over public records it had withheld for up to 20 months, but agreed to reform what The Post called a “pattern and practice” of endless delays and stonewalling.  After lengthy negotiations with Post lawyers, the DOE agreed to revise its FOIL rules to halt the indefinite postponements — and stick to reasonable deadlines. New guidelines were approved in November.

Here is the official stipulation by the Court, which also subsequently required the DOE to pay the Post's attorney fees. As I was quoted in the article, however,  “I’ll believe it when I see it.  I’m still waiting for documents that I requested two years ago.”

Sure enough, the DOE has continued its stonewalling, and in fact it has gotten worse.  As Sue revealed, ever since she reported on the fact that the Chancellor had promoted Mayor Adams' romantic partner to a higher position with a 23% salary boost to $221,597 a year,  the DOE has refused to answer any of her questions at all, including routine queries.  

The only change to the DOE's FOIL practices  since the court decision is that the boilerplate letter they send to those FOILing information is longer, and cites a number of reasons that the response has been and will be further delayed.  Here is their standard response:

This letter concerns the Freedom of Information Law (FOIL) request referenced above. Pursuant to Section VI.C of DOE Chancellor’s Regulation D-110 (CR D-110), please be advised that additional time is needed to respond to the remainder of your request.

Factors to be considered ...in determining whether there exist circumstances necessitating more time to respond to a request may include, but are not limited to, “[1] the volume of a request, [2] the ease or difficulty in locating, retrieving or generating records, [3] the complexity of the request, [4] the need to review records to determine the extent to which they must be disclosed, and [5] the number of requests received by the agency.

And then they add which factors are involved, usually a combination of 1, 2, 4 and 5.  The only other change from their previous behavior is that rather sending a letter each  month, now the DOE waits between three to six months to send these letters, and sometimes forgets to send any at all. 

An example:  In October 2020, I FOILed for DOE contracts with tech companies that had access to personal student data.  They responded a year later that if I narrowed my request to the privacy portions of those contracts, I could get the information sooner.  (According to state law, the privacy sections of contracts, called the Parent Bill of Rights,  are supposed to posted on the DOE website, yet only a partial list is included here, under the ambiguous title, "Supplemental Information for Parents About DOE Agreements With Outside Entities.")

In Oct. 2021, I agreed to limit my request to the privacy portions of those contracts, over a year ago, but am still waiting. 

This morning, I posted a twitter poll, asking if  people think the NY Post should reopen its lawsuit vs the DOE.   Please respond to the poll here.

Who thinks @nypost should reopen their lawsuit over @nycschools failure to comply with consent decree to respond in timely fashion to FOILs?

— leonie haimson (@leoniehaimson) December 4, 2022

After waiting two years for DOE to respond to my FOILs, I filed an appeal

More than two years have passed without a substantive response to three Freedom of Information Law requests to the NYC Department of Education, so yesterday I filed an appeal with General Counsel Howard Friedman, pointing out these are "constructive denials" and thus violate the state FOIL law. 

The subjects of my requests are quite clear and the DOE should have easy access to this information.  It should have taken no more than a week at most to deliver to me data on their spending on technology, their spending on charter school leases and renovations, and to provide privacy agreements between DOE and third party organizations with whom they share personal student data.

Yet in no case have they produced any of this information, but have sent me the same boilerplate letters every four weeks or so, saying that "due to the volume and complexity of requests we receive and process, and to determine whether any records or portions thereof will be subject to redactions permitted under Public Officers Law §87(2), additional time is required to respond substantively to your request.”

In all three cases, three to six months have passed since I've heard anything at all from DOE about my requests, though all three are still listed as "Open" on the Open Records dashboard, here, here and here.

The NY Post sued the DOE in 2016 over its continual delays in responding to its FOILs, which they identified as "constructive denials."  Here is their legal petition  and here the memo of law.

The Post settled the lawsuit  in April 2018, when the DOE "agreed to reform what The Post called a 'pattern and practice' of endless delays and stonewalling" and to revise their "rules to halt the indefinite postponements — and stick to reasonable deadlines. New guidelines were approved in November."  I was quoted in the article as remaining skeptical.

The DOE revised the Chancellor's regulations on FOILs last December,  though the DOE FOIL page still posts the previous Chancellors regs from 2009, so you can see how seriously they take these new rules.   (The head of their FOIL office, Joseph Barandello, is also the DOE chief privacy officer who also keeps busy justifying DOE's violation of FERPA, including making student personal information to charter schools for recruiting purposes without parental consent.)

According to the new regs, the DOE was supposed to be more specific about its reasons for delaying responses, and to provide a more certain date when the request would be fulfilled.  As I write below, the only difference I saw was longer boilerplate  prose describing in more verbose terms exactly what they had earlier claimed, a longer gap between letters, and then no letters at all.

Analyses by Village Voice and Chalkbeat have found that the DOE is the worst record in responding to FOILs of any city agency.  The Mayor's overall record on this is particularly deplorable, since as Public Advocate he made the city's poor performance in responding to FOILs one of his signature issues and promised during his campaign to improve the process.

In response to a KidsPAC survey, he pledged the following: "I have a record of transparency and will ensure that under my administration, the DOE will be in full compliance with FOIL. As Public Advocate, I monitored and reported on the city’s compliance with the Freedom of Information Law (FOIL)."

Yet if anything, the DOE's performance on FOILs is worse now than it was under Bloomberg.

According to the Chancellor's regs, the DOE General Counsel is supposed to "issue a written decision on the appeal within ten (10) business days following receipt of the appeal and fully explain in writing the reasons for further denial, or provide access to the record sought."  I'll let you know if and when he does respond, and let you know what he says.

If he doesn't provide the information I've requested I will take it to court.  It is now easier to obtain legal representation as a new state law on FOILs  approved in December makes "attorney’s fees awards compulsory where a government agency had 'no reasonable basis' for denying access to a request."

Spending on bureaucracy climbs while Renewal schools continue to struggle with large class sizes

Through a Freedom of Information request and then a lawsuit, the NY Post finally received a list of DOE Renewal administrators and field staff who work out of Tweed and district offices.  According to the list, there are 71 of these individuals with a collective salary last year of $8.4 million.  See the attached spreadsheet and NY Post article here, in which I say that these schools are being buried in an avalanche of bureaucrats. 

Yet the DOE left off the list Renewal Superintendent Michael Alcoff,whose salary last year was $175,000, as well as the 39 Renewal “leadership coaches” and three “Ambassador APs” for Renewal schools, whose estimated salaries are at least $100,000 each.  When these educrats are added, the total cost of the Renewal bureaucracy is at least $12.7 million.

This contradicts a report released by the Independent Budget Office in May 2015, presumably based on DOE data, that said only $200,000 was projected to be spent in FY 16 and FY 17 for Renewal school “administrative field support.”  Quite a difference between $200,000 and $12.7 million.

Last year, Ernie Logan of the CSA, the Principal’s union, complained that “principals in the 94 schools were being overwhelmed with paperwork and meetings and micromanaged to the point that they could not do what they thought was best for their schools.”  Yet what I think is most tragic is that so many of these schools are still struggling with large class sizes, despite repeated promises by the city to the state to reduce them.

Again, in their latest round of Contract for Excellence proposed plans, the DOE claims to be focusing its efforts on reducing class size in the Renewal schools, without any specific goals or commitments, just as they did last year.   Yet we found that last year, nearly 40 percent of the Renewal schools raised their average class size, and only seven percent capped class sizes at the nearly acceptable C4E levels of 20 students per class in grades K-3, 23 in grades 4-8 and 25 in high school.  As many as sixty percent of these schools still had classes as large as thirty.  This is simply unacceptable.  While millions are being spent on a phalanx of bureaucrats, micromanaging teachers who are struggling to help their students, students continue to struggle in classes this large.

NY Post sues NYC Department of Education for its abuse of the FOIL process

NYC parents and advocates have been continually frustrated with the delay and lack of response of DOE to Freedom of Information Law requests.  Sometimes I have waited up to two years, to then receive the DOE response that they don't have the document that I requested.

I have almost never received a fulfillment to my request within a reasonable time period -- a few months or less.  Instead, each time I receive the same form letter which says that because of "the due to the volume and complexity of requests we receive and process, and to determine whether any records or portions thereof will be subject to redactions permitted under Public Officers Law §87(2), additional time is required to respond substantively to your request. Accordingly, a response is currently anticipated" in about twenty days.  Then twenty days later I get the same letter again.

A typical instance occurred when I FOILed for the study that DOE had widely proclaimed attested to the accuracy of their teacher evaluation system, based on the value-added test scores of their students, the so-called "teacher data reports."

On December 3, 2008, DOE released a document justifying their this teacher evaluation system, entitled: “Teacher Data Initiative: Support for Schools; Frequently Asked Questions".  Among the claims made was the following: "A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

On Feb. 23, 2009, I FOILed for the names of members of this panel and any written materials they had produced.

On May 24, 2010, more than fifteen months later, I finally received a response. When I read the letter, the panel of experts said the opposite of what the DOE had claimed -- that these measurements could not be relied upon to fully evaluate the quality of ANY teacher.  Instead they specifically refused to endorse "any particular use \[of the method\] for accountability, promotion or tenure" purposes.  And:

"Test scores capture only one dimension of teacher effectiveness, and . . . are not intended as a summary measure of teacher performance....If high stakes are attached, there will be potential to game these measures by teaching to the test, selecting students, altering difficult-to-audit student characteristics, or outright cheating...." 
Daily News columnist Juan Gonzalez also exposed the hypocrisy of the DOE here; while pointing out how the system utilized inaccurate student and test scores data in many instances, making it an even less reliable gauge for assessing teacher performance. 

In contrast, when reporters FOILed the actual evaluations of 12,000 real-life teachers based upon this unreliable methodology (apparently after being encouraged to do so by DOE), Chancellor Klein was eager to provide the data immediately.  Their release was only temporarily delayed by a lawsuit from the UFT.

When Bill de Blasio ran for office, he promised if elected Mayor he would increase the transparency at the DOE.  Not only did he check off the box on the NYC KidsPAC survey that asked whether he would "Respond to FOILs in a timely and complete fashion" but then added:

I have a record of transparency and will ensure that under my administration, the DOE will be in full compliance with FOIL. As Public Advocate, I monitored and reported on the city’s compliance with the Freedom of Information Law (FOIL). I also created a “Transparency Report Card” helps the public track which agencies have complied with their FOIL obligations—and which have not.

Unfortunately, the first thing that happened after he took office was that I was excluded from the customary OMB budget briefings, with the excuse that advocates were no longer allowed in these meetings.

Then DOE closed School Leadership Team meetings to the public, violating the law and reversing the policy that it had previously if inconsistently maintained. (We are still waiting for the Appellate Court to rule on DOE's appeal of the April 2015 Supreme Court decision that closing these meetings is illegal.)

The DOE's responses to my FOILs have also been as slow as during the Bloomberg years -- even as to presumably uncontroversial issues like the approved plans of the PROSE schools, which the DOE has touted as innovative and progressive.  One of my DOE FOILs has been delayed for six months, and five of them for nearly six months, without any apparent progress.

So I was encouraged to hear that in August,  the NY Post had filed suit against the DOE for the months of delay in responding to their FOILs -- sometimes up to two years.  Here is the NY Post Article 78 petition , which makes fascinating reading; here is the memo of law.   An excerpt from the petition:

WHEREFORE, Petitioners respectfully request that this Court grant judgment:

(1) Declaring that Chancellor’s Regulation Chancellors Regulation D-110(VIII)(A) is unlawful and/or is inconsistent with N.Y. Pub. Officers Law § 84 et seq. and is, accordingly, invalid;

(2) Declaring that the DOE’s practice of sending repeated Form Delay Letters is inherently unreasonable and that determinations in its administrative appeal decisions that it may continue to deny access to public records because it has reissued serial unilateral extensions violates its obligations under FOIL, N.Y. Pub.Officers Law § 84 et seq.;

(3) Granting Petitioners’ Article 78 Petition, finding that DOE has constructively denied Petitioners’ requests, that Petitioners have exhausted their administrative remedies, and directing DOE to produce all disclosable records responsive to Petitioners’ outstanding requests within twenty (20) days of the Court’s order;

(4) Awarding Petitioners their costs and attorneys’ fees pursuant to Public Officers Law § 89(4)(c); and

(5) Awarding Petitioners such other and further relief as the Court deems just and proper.

Included in the petition is a relevant advisory opinion from Kristin O'Neill of the NY Committee for Open Government, an division of the NY Department of State:

....[T]he Post has now waited between five and nineteen months and received between three and fourteen Form Delay Letters for each of the nine pending requests. As you are aware, FOIL provides direction concerning the time and manner in which agencies must respond to requests.

Specifically, §89(3)(a) of FOIL states in part that:

"... it has long been advised that when an agency is unable to deny or provide access to records within five business days, it must provide an acknowledgement within that time indicating an approximate date, not to exceed twenty additional business days, on which it will grant access in whole or in part. If it is determined, either within five business days of the receipt of the request, or at or near the expiration of twenty business days of its acknowledgement, that it is unable to respond within twenty additional business days, it must indicate a "date certain" that includes both the date and the reasons for requiring additional time. Although we recognize that there are occasions when an agency will require an extension of time beyond that which it initially predicted, there is no provision in the statute for an extension beyond the date certain or repeated extensions. The date certain must be reasonable based on attendant facts and circumstances.

....A requester who is neither granted nor denied access to records with in the time limits set forth above or in the acknowledgment letter or any extension letter(s) may consider the request constructively denied and may appeal such denial in accordance with the procedures set forth below." (emphasis mine) As stated earlier, there is no provision in the statute for repeated extensions. ... As such, the agency is not permitted to establish the right to repeated extensions via regulation, where such right does not exist in statute or in the Committee's own regulations.

Also included in the lawsuit are the original reporters' FOIL requests and the repeated boilerplate non-responses from DOE. One of their requests which has not been fulfilled for over a year was for the substantiated cases of test or grade tampering as determined by the OSI -- the DOE's internal Office of Special Investigations.  There is no excuse for covering up any proven instances of cheating in our schools.

I don't agree with all the reporting in the NY Post.  Some of it is overly sensational; and the editors are obviously out to make de Blasio look bad.  Yet several of the Post reporters over the years, including Yoav Gonen and Sue Edelman, have uncovered serious corruption and mismanagement at the DOE, whether under Klein, Walcott or now, Chancellor Farina.  And NYC residents and taxpayers deserve a DOE that will be responsive, transparent and provide the public with its right to know.

My response to the NY Times article about the political uses of test scores

Update: See Sunday's NY Post, which unlike the other papers, actually gave some historical context for our skepticism.

On Friday and Saturday, some reporters started walking back their earlier stories that gave undue credence to the apparent test scores increases in NY state and NYC.  In the process, they made some of the same points I made in my blog on Thursday, that any claims of improved achievement were untenable, that this year's results cannot be compared to last year's, and that too many groups were making these claims for political reasons. However, in the process of writing their stories, these reporters caricatured my position in the debate.

First , please read my blog from Thursday if you haven't  already.  Then read Saturday's NY Times. Although the reporter Liz Harris seems now to agree that any claims of improved achievement are untenable, she makes several errors in the process.

1.  The  article identified me as leading "some groups opposed to the tests."  I don't lead any "anti-testing groups".  There are several NYC groups that might be characterized as such, but I lead none of them.  I  run Class Size Matters, which is dedicated to reducing class size in the NYC public schools and the nation as a whole, and I co-chair the national group the Parent Coalition for Student Privacy.  Moreover,  as I  explained to Liz,  I support testing, if the tests are well-designed, grade appropriate, stable and consistently scored, with no stakes attached.   That's why I paid such close  attention in my blog to the NAEPs, the national exams that feature reliably scaled results.

Why? The results of such tests are among the very few ways we can objectively track trends in student achievement. The use of such tests are also one of the reasons we know for sure that smaller classes lead to more learning and a narrowing of the achievement gap between racial, ethnic and economic groups.  Because I believe in the importance of testing to help diagnose whether a student is learning or if system-wide policies are working, I get especially angry when the results are distorted for political ends. 

As I also explained, because of repeated changes in the state tests and the scoring, NY hasn't produced tests that can reliably track learning or achievement trends since at least 2002.  Since then, we have had 14 years of wild swings, with huge test score inflation between 2002 and 2009, then sudden deflation, then another apparent drop in achievement when John King decided to impose new Common Core tests and set the proficiency levels to prove that two thirds of the state's students were failing.  This year,  to assuage the opposition of teachers and parents who said the tests were too long, too confusing and too stressful, and to counter the opt-out movement,  the Commissioner shortened the exams and implemented them untimed, which meant that this year's results could not be compared to last year's.  Yet the NY Times reporter insisted on characterizing me as "anti-testing," presumably to imply that I was politically motivated as well.

After the NY Times reporter told me they were going to focus on the politics rather than the accuracy of the state's claims, I wrote, "The politics are irrelevant to me.  We and others pointed out the test score inflation under Mills/Bloomberg/Klein and we're pointing it out now under Elia/deBlasio/Farina.  The more that things change, the more that they stay the same."

2. The reporter also wrote that I had claimed "the state had manipulated the underlying data to have more children pass."  This is untrue.   In my blog and in my conversation with her, I pointed to at least five reasons why reporters and members of the public should be skeptical of any claims of improved achievement in NYC and statewide.  These include: the state's  history of test score inflation, more recent NAEP trends which are in the opposite direction to the state score trends,  the fact that the state tests were shorter and given untimed this year, the fact that 95% of the state's districts had a  5% or more opt out rate, with many districts at nearly 50% opt out, while the city's participation rate was much higher, at more than 95%.  This makes not only any accurate judgements tenuous but also comparisons between the state's level of achievement to the city's especially suspect.  I also said that there was additional evidence that the state MAY have manipulated the results, as the percent of raw scores out of the total possible aligned with proficiency dropped in 11 out of 12 exams this year, while at the same time, the Commissioner stated that the tests were equally rigorous.   As I wrote in my blog, "We won’t know if the questions were harder or easier until the state releases the P-values and provides other technical details."

The state has the data on P-values now, which stands for probability values, and refers to the probability that a student would respond with the right answer to a question.  The higher the P-value, the easier a question is assumed to be, based on the responses to embedded field test questions from the year before. This would help demonstrate whether the "adjustments" SED made in dropping the number and percent of raw scores for each level compared to last year before were soundly based on statistics or arbitrarily drawn to boost apparent performance.  Until then we simply don't know.  Instead of merely quoting the state's critique of our analysis, reporters should demand more transparency from NYSED.  Indeed, the public deserves answers.

3.  Another weakness of the NY Times article was to omit the Commissioner from the list of those individuals and groups who were claiming that the data showed real improvements in learning.  Though the NYSED presentation did include the telling phrase “because of the changes made to the 2016 exam and testing environment, the 2016 test scores are not an ‘apples-to-apples’ comparison with previous years,” the Commissioner then proceeded to do just that, by showing chart after chart demonstrating big jumps in proficiency this year.   Here  are also some of her tweets on the subject:

Statewide in ELA, the proficiency level increased 6.6 points to 37.9%, up from 31.3% in 2015.

— NYSED News (@NYSEDNews) July 29, 2016

In conclusion, overall, students scoring at the proficiency level increased statewide in ELA and math, with a bigger increase in ELA.

— NYSED News (@NYSEDNews) July 29, 2016

2016 Grades 3-8 Exams Show Consistent Increases in ELA & Math; State Will Continue to Engage Parents & Teachers: https://t.co/SaHawh4HOF

— NYSED News (@NYSEDNews) July 29, 2016

While in the presentation, she said that "We cannot pinpoint exactly why the test [scores] increased,” she then variously attributed them to students in grades 3 and 4 having "received instruction in the Common Core since Kindergarten and 1st grade," and the fact that teachers had more experience teaching the standards.  In a radio interview, she said the bump in ELA scores was due to an intensified focus on literacy.

Elia's insistence in ignoring the disclaimer in her own presentation encouraged NYC reporters to recite the stats showing gains in proficiency, and to echo the claims of great improvement announced by the Mayor and Chancellor, while minimizing any mention of the dubious nature of these claims. In the case of the the NY Times, the "apples to apples" disclaimer was relegated to the 8th paragraph of the article.  In a follow-up story on the  purported gains at the Renewal schools on July 31, the Times reported similarly big jumps in proficiency in these struggling schools, and reserved the "apples to apples" comment to the 8th paragraph of the story.

Chalkbeat NY ran a story on last Friday, purporting to show that testing experts disagreed with the NYSAPE analysis that the drop in the raw scores may have also contributed to the increase in proficiency rates, along with other changes.  After "eyeballing" the changes in the 3rd grade ELA raw scores, Jennifer Jennings said "that just looks like year-to-year variation."  I agree that a larger contributor to the large leaps in ELA, particularly in 3rd and 4th grades,  may have been the fact that the exams were shorter and untimed, especially since many teachers said kids didn't finish them in 2014 and 2015.  Which is not to say that the raw score adjustments might not be a factor as well, particularly in math and other grades.

The article also quoted Aaron Pallas, who said he couldn't tell if the changes in the raw scores mattered until the technical report is released, a year or so from now. This may be true, but there is no reason that reporters and researchers should not urge the state to release the P-values more quickly, to allay our concerns.

The Chalkbeat article concluded with Daniel Koretz, who said simply that last year's results can't be compared to this year's.  Exactly right. Then why did Chalkbeat itself run several stories recounting the surge in proficiency, and even speculating on the possible causes, including the potential effects of the city's education "reforms", the expansion of charters,  and/or the teaching of the Common Core?


4.  Then there's the lack of any historical context. The paper of record has a lamentable record of failing to report on the well-documented evidence of inflated test score gains that occurred from 2003-2009, until the state itself admitted what had happened and re-calibrated the cut scores in 2010.  Their unshakeable credulity led to a front page story on August 3, 2009  -- a little more than seven years ago to the day, recounting the big jump in student achievement and giving credit to the Bloomberg reforms.  Like now, they refused to explain the multiple sources of evidence to the contrary, including the fact that the NAEPs showed only modest gains over the same time period.  My argument with the Times editors even made the Village Voice .   As Wayne Barrett wrote,

The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true.

Two days after the article ran, the NY Senate voted to renew Mayoral control.  A few months later, Bloomberg was re-elected to a third term.   Sure enough, when the test bubble burst in 2010  all the gains were shown to be illusory. Even after that, though, in 2011 a writer for the NY Times Magazine reported that "since 2006, the city's elementary and middle schools have seen a 22-point increase in the percentage of students at or above grade level in math (to 54 percent) and a 6-pont increase in English (to 42 percent)."

These statistics were completely fabricated of course, provided by DOE to the reporter, and somehow neither the reporter nor any editors had bothered to  check them.  It turned out the DOE had made up the data by re-adjusting the cut scores to where the state had previously put them, essentially rewriting history as though test score inflation and deflation had never occurred.

All this is to say: If reporters at the NY Times and other media outlets are prepared to point out the unjustified claims promoted by public officials and some advocacy groups last week, that is good; but they might try also provide some context to explain the larger reasons for skepticism.