DOE's irresponsibility in employing AI products regardless of whether they protect student privacy

A week ago, the NY Post featured an article about a new AI program call Yourai sold by a company called LINC, or The Learning Innovation Catalyst, that the DOE is piloting in some Brooklyn schools.  The product is supposed to help teachers develop their lesson plans.  On Twitter last week, I pointed out the idiocy of the DOE administrator who claimed this would help teachers "think creatively."

What? “Teachers can enter students’ needs and the standards they want into the app and a lesson plan will be generated, Ross explained..She called it a “game changer” that will give teachers more time to “think creatively”. Ridiculous! https://t.co/aOFbQh3qp6

— leonie haimson (@leoniehaimson) March 16, 2024

I went on to point out that two of the three testimonials on the website from NYC teachers appeared to be fake, as I couldn't find their names in a list of DOE employees.

No evidence that 2 of 3 NYC people quoted with testimonials on Yourwai website were employed by DOE through 2023 after checking payroll records on SeethroughNY. There are NyC teachers named Michael Davis, but no admins w/ below names. If anyone knows differently pl let me know! pic.twitter.com/xkVI7YkqhY

— leonie haimson (@leoniehaimson) March 17, 2024

Today, the NY Post followed up with another article, pointing out that there were apparently eight fake testimonials from NYC teachers on the website, and that after being asked about this, the company said their names "were anonymized for compliance purposes," and have now been taken down..

Apparently, the co-CEO of the company, Jason Green, is a close pal of the Chancellor, and he and his family vacationed with the Chancellor's family on Martha Vineyard last summer.  The article added that LINC has received $4.3 million from DOE since 2018 for "professional development and curriculum," including $2.3 million so far  this school year.

What they did not mention is that, aside from the likely shoddiness of the product and the fake hype surrounding it, there are real concerns about these sorts of products including the risk to student privacy, as I pointed out on twitter.  

AI products are  well known for gobbling up huge amounts of personal student data, and then using it to improve their products and create new ones.  Yet this is specifically prohibited by the regulations of NY State's student privacy law, Ed Law § 2-d.

These regulations clearly state that "Third-party contractors shall not sell personally identifiable information nor use or disclose it for any marketing or commercial purpose" and that "Commercial or Marketing Purpose means the sale of student data; or its use or disclosure for purposes of receiving remuneration, whether directly or indirectly; the use of student data for advertising purposes, or to develop, improve or market products or services to students [emphasis added]."

I also pointed out that any district vendor or other third party with access to personal student data by law is supposed to have a specific privacy addendum to its contract.  This addendum is supposed to be posted on the DOE website here, but none can be found for LINC or YourAi.  Sadly, DOE continues to flout the law when it  comes to protecting student data and the transparency required by Ed Law § 2-d, as we have noted in the past.

On twitter, I highlighted specific weaknesses in LINC's online privacy policy, including that they allow other companies to track user behavior, including “3rd parties that deliver content or offers” meaning marketing.

Correction: I finally found privacy policy on company website saying they comply with 2D but unclear how as provide no details & says they allow other companies to track user behavior including “3rd parties that deliver content or offers” ie marketing. https://t.co/a8jr52LLcR

— leonie haimson (@leoniehaimson) March 17, 2024

I also noted that the Privacy Policy said that the company reserved the right to change it at any time for any reason without prior notification to users by changing wording online.  This violates FERPA, because then, districts are not in control of how student data may be used or disclosed.

 

After noting these red flags on twitter, the co-CEO Jason Green DMed me:

We are a minority company that has been partnering with NYCPS for years. Our mission is to help teachers better support learners. I am also recently married and a dog-lover. Would you be open to learning more about us? I would love to better understand your perspective as well.

I said sure, and then asked to see his contract with DOE, to ensure that it contained the required data privacy and security protections.   I didn't hear back until yesterday, when he said he was "working with his team" to get the contract, but assured me that they don't "directly" collect or use student data.  

When I asked what "directly" means, he said they don't collect student data at all.

Then, later that day, on Friday March 22, I went back to look at the company's Privacy Policy and noticed it had been updated that very day:

Low and behold, there was a bunch of new sections added, including that the company indeed "may have access to student data" or "teacher or principal data" as defined under Ed Law § 2-d: 

They had revised the section that previously said the company may change the Privacy Policy without prior notice.  It now says  "We will send advance notice of any upcoming changes to our Privacy Policy via e-mail."  The section about allowing other companies to use user data for marketing purposes was taken out, but this passage that replaced it is not much more reassuring:

Also, Third Party Companies may want access to Personal Data that we collect from our customers. As a result, we may disclose your Personal Data to a Third Party Company; however, we will not disclose your Personal Data to any Third Party Company for the Third Party Company’s own direct marketing purposes. The privacy policies of these Third-Party Companies may apply to the use and disclosure of your Personal Data that we collect and disclose to such Third-Party Companies. Because we do not control the privacy practices of our Third-Party Companies, you should read and understand their privacy policies.

So what does it say in the actual, DOE contract with LINC, that  legally binds their use and protection of student data?  Sue Edelman of the NY Post FOILed the contract from the NYC Comptroller and sent it to me on Friday.

To make a long story short, the only LINC contract the Comptroller's office had was this one from 2020, which never mentions Ed Law § 2-d, though law was passed in 2014, and doesn't contain its required provisions.  

Instead, the contract glosses over the entire issue of student privacy, and says instead that it complies with Chancellor’s Regulations A-820 "governing access to and the disclosure of information contained in student records." Yet Chancellor's Regulations A-820 has not not been updated since 2009. 

In his blog today, Peter Greene has one of his excellent take downs of the whole notion of AI producing better lesson plans than actual living teachers.  He includes this   quote from Cory Doctorow:

We’re nowhere near the point where an AI can do your job, but we’re well past the point where your boss can be suckered into firing you and replacing you with a bot that fails at doing your job.

But beyond the lamentable mechanization and degradation of education that is being promoted by NYC and other districts nationwide, in the name of mindless innovation, the DOE apparent lack of interest in protecting student privacy and following the law remains appalling.