California admits to using high-risk AI — including systems it failed to report last year

A year ago, California officials were required under a new state law to report how they used automated systems to make important decisions about people’s lives.

They said they never did — a startling answer for a number of reasons, sources said told CalMatters at the timeincluding that there were several prominent examples to the contrary.

Now the state has given a more comprehensive answer: It currently uses six automated systems to make consistent decisions about the lives of Californians.

Systems are used to do things like:

• Predict whether prisoners will reoffend
• Assess whether unemployment claims are fraudulent
• Remote administration of exams for California State University students
• Discover when students use generative AI for writing assignments.

This is according to a report released Friday from the state technology department. The report is mandatory under a 2023 law requiring it that government agencies annually disclose the use of “high-risk automated decision-making systems,” which the law defines as systems “used to assist or replace human discretionary decisions that have a legal or similar significant effect, including decisions that materially affect access to or approval of housing or housing, education, employment, credit, health care, and criminal justice.”

The law was pushed by civil rights, privacy and civil liberties groups concerned about harm from AI-like systems. Numerous such systems have been shown to produce results that are biased against marginalized groups, including those used for high stakes testing, predicting recidivismand AI-generated text detection.

CalMatters flagged last year’s report as surprising, noting that the state Department of Corrections reported using software to predict post-release behavior and that the Department of Labor used a fraud detection system that suspended benefits for 600,000 Californians between Christmas and New Year’s in 2020, according to a report by the Office of the Legislative Analyst.

Although the report names six high-risk systems in use today, government agencies have been using some of them for several years. These include COMPAS, which has been used by the California Department of Corrections and Rehabilitation to determine inmate recidivism scores for at least a decade.

The technology department said in the report that it found more systems for its report this year because it evaluated state agency responses more thoroughly, including by meeting with agencies and asking them about their systems.

In addition to the six high-risk systems, the department’s report revealed an additional six systems initially flagged as high-risk but later determined not to be. One was AI used to analyze legislative bills by the California Department of the Treasury.

The report also notes two high-risk systems that are not currently in use: The Department of Cannabis Control is developing artificial intelligence to analyze whether marijuana packaging violates a child-appeal law, and California State University has stopped using a language model to review job applications.

The results of the second annual survey come after cities such as San Jose and San Francisco released their first AI inventories in recent months. They also come at a time when

Big California-based AI companies like Anthropic and OpenAI are going public and seeking government contracts. Most Americans are divided on whether they trust AI and studies last year by TechEquity and Carnegie California discovered that the majority of Californians want safety over innovation. A Gallup poll of American opinion showed similar results.

Senate Bill 1248A bill that would have prohibited state employees from using automated decision-making systems as the sole basis for decision-making was killed last month in the state fast appropriations process.

What’s missing

While the newly released report shares more information than last year’s, several questions remain about the state’s use of artificial intelligence and other automated systems.

The report does not include generative AI pilots that are being run with the support of the governor’s office to do things like help businesses file taxes, support state employees who work for the homeless and an AI assistant named Poppy that uses language models like Anthropic’s Claude to do things like draft documents, research policy or build custom AI tools, according to government website. The website says 67 government departments provided information during the pilot phase and Poppy’s nationwide rollout begins next month.

A Contract with California State University with OpenAI to provide a version of ChatGPT is also not mentioned, although studies on the use of AI in educational settings have found that technology can do more harm than good.

do you know more?

Contact reporter Carrie Johnson directly on Signal at kharijohnson.67 .

The 2023 law, mandating the annual report for high-risk systems, excludes reporting from a number of state agencies, including the judiciary and the University of California College System. CalMatters reporting last month found that the majority of the approximately 60 courts operating across the country have adopted generative policies for the use of AI. Courts in Los Angeles and Riverside counties have testing has begun An AI tool to act as a clerk, drafting orders and creating research notes.

CalMatters compiles a list of automated decision-making systems used by state and local agencies across California to provide transparency on how governments use decision-making systems and AI. Know of an AI system used by a state or local agency? Email khari@calmatters.org.